JP7024533B2 - Detection pattern evaluation model generation system, method and program - Google Patents

Description

The present invention relates to a detection pattern evaluation model generation system for generating a model for evaluating a detection pattern for extracting a unique expression from a document, a detection pattern evaluation model generation method, and a detection pattern evaluation model generation program.

Personal information and confidential information contained in a document, or information indicating that a document is a confidential document, is called a named entity.

As a general method for detecting only a named entity from a document, for example, the first general method and the second general method shown below can be considered.

The first general method is to use a pattern rule or a dictionary for detecting named entity.

The second general method is a method of using morphological analysis and attributes assigned to the analysis dictionary.

Further, as a general method for estimating the presence / absence of specific data in a document, a method of estimating the presence / absence of specific data in a document using a learning device can be considered.

Further, Patent Document 1 describes a named entity extraction rule generation system. The named entity extraction rule generation system described in Patent Document 1 extracts each word from a training document prepared in advance, and based on each word and a list of correct answers prepared in advance corresponding to the training document, the named entity is expressed. Create a rule for extraction. Then, the named entity extraction rule generation system independently applies these rules to the training document, obtains the named entity extraction result for each rule, compares the named entity obtained by the application result with the correct answer list, and then compares them. Delete and modify rules.

Further, in Patent Document 2, input of target information indicating a set of character strings to be extracted is accepted, and a target expression which is a character string matching any of the character strings included in the target information and a predetermined target expression are specified. Neighboring words, which are words placed within a distance, are extracted from the target document, and the frequency of appearance of each of the neighboring words in the target document or the coordinates in the target document of each target expression is used for unsupervised learning. , It is described that a filter for obtaining a set of words to be extracted is generated.

Japanese Unexamined Patent Publication No. 2001-318792 International Publication No. WO2016 / 194054

In recent years, with the expansion of the IT (Information Technology) society, the number of companies and organizations that handle confidential documents including personal information and confidential information on computer systems is increasing. Confidential documents are often managed on the same computer system as general documents that do not contain personal or confidential information. In this case, it is difficult to thoroughly manage confidential documents for the following reasons.

The first reason is that a secret document is copied or moved to a storage area that should not be stored (for example, a storage area accessible to anyone, an unencrypted area, etc.) by the operation of a certain user. If so, the reason is that it is difficult for the administrator to detect it.

The second reason is that when the administrator periodically checks whether or not confidential documents are stored in a specific area, it is necessary to manually check the contents of each document, resulting in a large number of documents. The reason is that it is not practical to perform such confirmation work if it exists in. Moreover, even if such confirmation work can be performed, the worker may overlook the confidential document.

In order to determine whether or not the confidential document is stored in the specific area, it is conceivable to use the above-mentioned first general method or the second general method. However, when the above-mentioned first general method or second general method is used, there is a problem that all the information that matches the rules is detected, and even the information that is not originally unique information is detected. be. That is, the unique information is erroneously detected (mainly over-detected), and it is difficult to accurately confirm whether or not the confidential document is stored.

In addition, a general technique of estimating the presence or absence of specific data in a document using a learning device cannot grasp what kind of unique information is contained in the document.

Given a document, it is preferable to be able to select a detection pattern for extracting named entities from the document in order to prevent over-detection of the unique information. However, the technique described in Patent Document 1 does not provide a named entity extraction rule according to a given document.

Therefore, the present invention generates a model capable of evaluating how suitable each of a plurality of detection patterns for extracting an eigenexpression from a document is suitable for extracting an eigenexpression from a given document. It is an object of the present invention to provide a detection pattern evaluation model generation system, a detection pattern evaluation model generation method, and a detection pattern evaluation model generation program.

The detection pattern evaluation model generation system according to the present invention uses a plurality of detection patterns for extracting a unique expression from a document, and an extraction means for extracting a character string corresponding to the detection pattern from the document for each document, and a document. , The unique expression is extracted from the given document using the set of the set of the detection pattern and the judgment result of whether or not the character string extracted from the document based on the detection pattern corresponds to the proper expression as the teacher data. It is characterized by comprising a model generation means for generating a model for deriving the possibility of performing as a score for each detection pattern by machine learning.

Further, in the detection pattern evaluation model generation method according to the present invention, a computer uses a plurality of detection patterns for extracting a unique expression from a document, and extracts a character string corresponding to the detection pattern from the document for each document. A unique expression from a given document, with a set of sets of a document, a detection pattern, and a judgment result of whether or not a character string extracted from the document based on the detection pattern corresponds to a unique expression as teacher data. It is characterized in that a model for deriving the possibility of extracting the above as a score for each detection pattern is generated by machine learning.

Further, the detection pattern evaluation model generation program according to the present invention uses a plurality of detection patterns for extracting a unique expression from a document on a computer, and extracts a character string corresponding to the detection pattern from the document for each document. The set of the set of the processing, the document, the detection pattern, and the judgment result of whether or not the character string extracted from the document based on the detection pattern corresponds to the proper expression is given as the teacher data. It is characterized in that a model generation process generated by machine learning is executed to derive a model for deriving the possibility of extracting a unique expression from a document as a score for each detection pattern.

According to the present invention, a model is generated that can evaluate how suitable each of a plurality of detection patterns for extracting a named entity from a document is suitable for extracting a named entity from a given document. can.

It is a block diagram which shows the structural example of the detection pattern evaluation model generation system of embodiment of this invention. It is a schematic diagram which shows the example of a plurality of detection patterns and a plurality of named entity types stored in a detection pattern storage unit. It is a schematic diagram which shows the example of the information which the extraction result storage part stores. It is a schematic diagram which shows the example of the teacher data. It is a schematic diagram which shows the example of the vector defined for each document ID of a training document. It is a schematic diagram which shows the machine learning using a neural network. It is a schematic diagram which shows the example of the information which associated the document ID and the pattern ID of the selected detection pattern for each document. It is a flowchart which shows the example of the processing progress of the model learning step of the embodiment of this invention. It is a flowchart which shows the example of the processing progress of the actual operation step of the embodiment of this invention. It is a schematic block diagram which shows the structural example of the computer which concerns on embodiment of this invention. It is a block diagram which shows the outline of the detection pattern evaluation model generation system of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

The detection pattern evaluation model generation system of the embodiment shown below generates a model for evaluating a detection pattern for extracting a named entity from a document by machine learning. The process up to the generation of this model is referred to as a model learning step. Further, the detection pattern evaluation model generation system of the embodiment shown below derives a score indicating the possibility of extracting unique information from a given document for each of a plurality of detection patterns after the model learning step, and the score is obtained. Based on, the detection pattern according to the given document is selected from the above-mentioned plurality of detection patterns. In addition, the detection pattern evaluation model generation system extracts a character string from a given document using the selected detection pattern. After the model learning step, the process of selecting from the detection patterns and extracting the character string from the given document using the selected detection pattern is referred to as an actual operation step.

FIG. 1 is a block diagram showing a configuration example of a detection pattern evaluation model generation system according to an embodiment of the present invention. The detection pattern evaluation model generation system 1 of the present embodiment includes a document storage unit 11, a detection pattern storage unit 12, an extraction result storage unit 13, a teacher data storage unit 14, a model storage unit 15, and an extraction unit 16. , A model learning unit 17 and a detection pattern selection unit 18.

Further, the extraction unit 16 includes a document acquisition unit 161 and an information extraction unit 162. The model learning unit 17 includes a first preprocessing unit 171 and a learning unit 172. The detection pattern selection unit 18 includes a second preprocessing unit 181 and a determination unit 182.

The detection pattern evaluation model generation system 1 is realized by, for example, a computer such as a personal computer or a server. However, the detection pattern evaluation model generation system 1 may be realized by a plurality of devices. For example, the extraction unit 16, the model learning unit 17, and the detection pattern selection unit 18 may be realized by different computers. Then, the document storage unit 11, the detection pattern storage unit 12, the extraction result storage unit 13, the teacher data storage unit 14, and the model storage unit 15 may be realized by separate external storages, respectively. In the following description, a case where the detection pattern evaluation model generation system 1 is realized by one computer will be described as an example.

The document storage unit 11 is a storage device that stores a plurality of documents. The document storage unit 11 stores a plurality of training documents used for generating a model for evaluating the detection pattern, and a document for which unique information is to be extracted in the actual operation step. The number of documents for which unique information is to be extracted in the actual operation step may be one or a plurality. An ID (Identification) is defined in advance for each document. Hereinafter, this ID is referred to as a document ID.

The storage area in which the document storage unit 11 stores a document is, for example, a shared area.

The detection pattern storage unit 12 is a storage device that stores a detection pattern for extracting a named entity from a document. The detection pattern storage unit 12 stores a plurality of predetermined detection patterns. The detection pattern may be represented in a regular expression format or may be represented by the named entity itself to be extracted. Further, each detection pattern may be manually created by a user of the detection pattern evaluation model generation system 1 (hereinafter, simply referred to as a user). Alternatively, for each detection pattern, words before and after the named entity extracted from a plurality of documents used to create the detection pattern using a dictionary or the like are extracted using a general technique such as morphological analysis, and those words are extracted. May generate a detection pattern represented by a regular expression. At this time, duplicate detection patterns are deleted for each named entity type.

The named entity type represents the type of named entity extracted from the document using the detection pattern. Examples of unique expression types include "name", "address", "telephone number", "email address", "credit card number", "my number", "trade secret" and the like. In addition, "my number" is an individual number assigned individually from the country to the people in Japan. The detection pattern storage unit 12 also stores each named entity type.

An ID is predetermined for each detection pattern. Hereinafter, this ID will be referred to as a pattern ID. In addition, IDs are also defined in advance for each named entity type. Hereinafter, this ID will be referred to as a named entity type ID.

FIG. 2 is a schematic diagram showing an example of a plurality of detection patterns and a plurality of named entity types stored in the detection pattern storage unit 12. FIG. 2A shows an example of a plurality of detection patterns stored in the detection pattern storage unit 12. Each detection pattern is associated with a pattern ID and a named entity ID corresponding to the detection pattern. FIG. 2B shows an example of a plurality of named entity types stored in the detection pattern storage unit 12. A named entity type ID is associated with each named entity type.

The detection pattern is defined so that the same detection pattern having the same named entity ID does not exist in duplicate. If the named entity IDs are different, the same detection pattern may exist, and even if the detection patterns themselves are the same, if the named entity IDs are different, they are treated as different detection patterns.

The extraction result storage unit 13 is a storage device that stores a character string (extraction information) extracted from a document by the extraction unit 16 using a detection pattern. FIG. 3 is a schematic diagram showing an example of information stored in the extraction result storage unit 13. As will be described later, the extraction unit 16 (more specifically, the information extraction unit 162) extracts a character string from the document using the detection pattern. The extraction unit 16 assigns a detection ID to each extracted character string, and extracts the extracted character string (extraction information), the detection ID, the document ID of the document from which the character string is extracted, and the character string. The pattern ID of the detection pattern used, the unique expression type ID corresponding to the detection pattern, and the data flag are associated with each other and stored in the extraction result storage unit 13. FIG. 3 shows an example of each record stored in the extraction result storage unit 13 in this way.

The data flag is a record (a record including a detection ID, a data flag, a document ID, a pattern ID, a unique expression type ID, and an extracted character string as shown in FIG. 3) as teacher data described later. It is a flag indicating whether it is a record used for creating the data or a record created by extracting a character string in the actual operation step. Hereinafter, in the record used for creating the teacher data, the data flag is set to "0", and in the record created by extracting the character string in the actual operation step, the data flag is set to "1". It shall be. However, it is assumed that the initial value of the data flag is blank (represented by the symbol "-"). Therefore, in the model learning step, when the extraction unit 16 detects a character string from the document using the detection pattern and stores the record illustrated in FIG. 3 in the extraction result storage unit 13, the value of the data flag of the record is stored. Set to the initial value "-".

The user creates the teacher data. Therefore, the operation of setting the value of the data flag of the record used for creating the teacher data to "0" is performed by the user.

The teacher data storage unit 14 is a storage device that stores teacher data used when a model for evaluating each detection pattern stored in the detection pattern storage unit 12 is generated by machine learning. FIG. 4 is a schematic diagram showing an example of teacher data stored in the teacher data storage unit 14. As shown in FIG. 4, the teacher data is a set of a set of a document ID, a pattern ID, and a correct answer flag.

The correct answer flag included in the teacher data will be described. The character string extracted from the document by the extraction unit 16 using the detection pattern is not always unique information. The user determines a set of a document ID, a pattern ID, and a correct answer flag for each combination of the training document and the detection pattern stored in the detection pattern storage unit 12. Then, the user confirms the record stored in the extraction result storage unit 13 including the document ID and the pattern ID included in the set for each set, and the character string included in the record is the unique information. Judge whether or not. The correct answer flag indicates this judgment result. Further, the user sets the value of the data flag (see FIG. 3) included in the record to "0". In the present embodiment, when the user determines that the extracted character string is not unique information, the correct answer flag is set to "0", and the value of the correct answer flag of the set in which the correct answer flag is not set to "0" is set. An example of setting all to "1" will be described. That is, except when the user determines that the extracted character string is not unique information, the case where the character string is extracted and the character string is regarded as unique information will be described as an example.

Since the set of the document ID, the pattern ID, and the correct answer flag is determined for each combination of the training document and the detection pattern stored in the detection pattern storage unit 12, the line including the common document ID in the teacher data. There are as many numbers of as there are detection patterns.

The model storage unit 15 is a storage device that stores a model for evaluating each detection pattern stored in the detection pattern storage unit 12. The model learning unit 17 uses the teacher data stored in the teacher data storage unit 14 to generate a model for evaluating each detection pattern by machine learning, and stores the model in the model storage unit 15. More specifically, the model learning unit 17 generates a model for deriving a score indicating the possibility of extracting unique information from a given document for each detection pattern, and stores the model in the model storage unit 15.

The extraction unit 16 acquires each document from the document storage unit 11, and uses the detection pattern to extract a character string corresponding to the detection pattern from the document for each document. The extraction unit 16 will be described in more detail.

As described above, the extraction unit 16 includes a document acquisition unit 161 and an information extraction unit 162.

In the model learning step, the document acquisition unit 161 sequentially acquires the training documents used for generating the model from the document storage unit 11. Then, in the model learning step, the information extraction unit 162 extracts the character string corresponding to the detection pattern from the document by using each detection pattern stored in the detection pattern storage unit 12 for each training document. Then, the information extraction unit 162 stores the record illustrated in FIG. 3 in the extraction result storage unit 13.

Further, in the actual operation step, the documents to be extracted of the unique information are sequentially acquired from the document storage unit 11. Then, in the actual operation step, the information extraction unit 162 extracts the character string corresponding to the detection pattern from the document by using the detection pattern selected by the detection pattern selection unit 18 for each document. Then, the record corresponding to the extraction result is stored in the extraction result storage unit 13. In the actual operation step, the information extraction unit 162 sets the value of the data flag included in the record to “1”.

In the model learning step, the information extraction unit 162 extracts a character string from the document by using each detection pattern stored in the detection pattern storage unit 12 without depending on the document. On the other hand, in the actual operation step, the detection pattern used when extracting the character string from the document is selected by the detection pattern selection unit 18 according to the document.

The model learning unit 17 derives a score indicating the possibility of extracting unique information from a given document for each detection pattern based on the teacher data (see FIG. 4) stored in the teacher data storage unit 14. Is generated by machine learning. The model learning unit 17 will be described in more detail.

As described above, the model learning unit 17 includes a first preprocessing unit 171 and a learning unit 172. The model learning unit 17 (first preprocessing unit 171 and learning unit 172) operates in the model learning step.

As described above, when creating teacher data, a set of a document ID, a pattern ID, and a correct answer flag is determined for each combination of the training document and the detection pattern stored in the detection pattern storage unit 12. Be done. Therefore, in the teacher data, the number of lines containing the common document ID is the same as the number of detection patterns. The number of detection patterns stored in the detection pattern storage unit 12 is k, and the order from the first detection pattern to the kth detection pattern is determined by the pattern ID (see FIG. 2A). It shall be.

The first preprocessing unit 171 generates a vector corresponding to the k lines of data included in the teacher data for each document ID of the training document. This vector is a vector having k elements corresponding to k detection patterns from the first detection pattern to the kth detection pattern on a one-to-one basis. Then, the first preprocessing unit 171 refers to the data for the above k lines, respectively, and determines that the character string extracted by the i-th detection pattern corresponds to the named entity (that is, i). (When the correct answer flag is "1" in the row corresponding to the th-th detection pattern), the value of the element of the vector corresponding to the i-th detection pattern is set to 1. Further, the first preprocessing unit 171 determines that the character string extracted by the i-th detection pattern does not correspond to the named entity (that is, the correct answer flag in the line corresponding to the i-th detection pattern). Is "0"), the value of the element of the vector corresponding to the i-th detection pattern is set to 0. Here, i is each integer from 1 to k.

The first preprocessing unit 171 defines the above vector for each document ID of the training document. The number of the elements of the above vector defined for each document ID of the training document is k (the number of detection patterns stored in the detection pattern storage unit 12).

FIG. 5 is a schematic diagram showing an example of a vector defined for each document ID of the training document by the first preprocessing unit 171.

Further, the first preprocessing unit 171 generates a vector corresponding to the content of the training document for each document ID of the training document. For example, the first preprocessing unit 171 may use a vector including an element having a one-to-one correspondence with the type of the character string including a predetermined number of characters as a vector according to the content of the training document. Then, the first preprocessing unit 171 applies N-gram to the training document in the above-mentioned predetermined number, and sets the number of appearances of the character string as the value of the element corresponding to the character string. A vector corresponding to one training document may be defined. The first preprocessing unit 171 defines the above vector for each document ID of the training document.

Further, for example, the first preprocessing unit 171 may use a vector including elements having a one-to-one correspondence with various words as a vector according to the content of the training document. Then, the first preprocessing unit 171 performs morphological analysis on the training document, sets the number of appearances of the word as the value of the element corresponding to the word, and thereby responds to one training document. A vector may be defined. Also in this case, the first preprocessing unit 171 defines the above vector for each document ID of the training document.

As described above, the first preprocessing unit 171 generates a vector corresponding to the teacher data for each document ID of the training document (see FIG. 5), and for each document ID of the training document for training. Generate a vector according to the content of the document. Therefore, the first preprocessing unit 171 will generate two types of vectors for each document ID of the training document.

The learning unit 172 performs supervised machine learning using each set of vectors generated by the first preprocessing unit 171 for each document ID of the training document, thereby performing score (unique information from a given document). Generate a model that derives the score) indicating the possibility of extraction for each detection pattern. The learning unit 172 may perform machine learning using a neural network as supervised machine learning. FIG. 6 is a schematic diagram showing machine learning using a neural network. The higher the score, the higher the possibility that unique information will be extracted from the document.

The learning unit 172 stores the generated model in the model storage unit 15.

The detection pattern selection unit 18 operates in the actual operation step. In the actual operation step, the detection pattern selection unit 18 selects a detection pattern according to the document for each document for which the unique information is to be extracted. More specifically, the detection pattern selection unit 18 has a high possibility of extracting unique information from a document for each document (in other words, it is highly useful when extracting unique information from a document). , Select from each detection pattern stored in the detection pattern storage unit 12. The detection pattern selection unit 18 will be described in more detail.

As described above, the detection pattern selection unit 18 includes a second preprocessing unit 181 and a determination unit 182.

The second preprocessing unit 181 generates a vector according to the content of the document for each document for which the unique information is to be extracted. This operation is the same as the operation in which the first preprocessing unit 171 generates a vector according to the content of the document for each document.

For example, the second preprocessing unit 181 may use a vector including an element having a one-to-one correspondence with the type of the character string including a predetermined number of characters as a vector according to the content of the document. Then, the second preprocessing unit 181 applies N-gram to the document in the above-mentioned predetermined number, and sets the number of appearances of the character string as the value of the element corresponding to the character string. A vector corresponding to one document may be defined. The second preprocessing unit 181 defines the above vector for each given document.

Further, for example, the second preprocessing unit 181 may use a vector including elements having a one-to-one correspondence with various words as a vector according to the content of the document. Then, the second preprocessing unit 181 performs morphological analysis on the document, sets the number of appearances of the word as the value of the element corresponding to the word, and determines the vector corresponding to one document. May be good. Also in this case, the second preprocessing unit 181 determines the above vector for each given document.

The determination unit 182 reads the model from the model storage unit 15. Then, the determination unit 182 stores the vector (vector corresponding to the content of the document) generated by the second preprocessing unit 181 in the detection pattern storage unit 12 for each given document by applying the vector to the model. The score of each detection pattern is derived. The higher the score, the more likely it is that unique information will be extracted from the document. The determination unit 182 selects a detection pattern that satisfies the condition that the score is equal to or higher than the threshold value from the detection patterns stored in the detection pattern storage unit 12 for each given document. The threshold value may be set in advance as a constant.

As a result, a detection pattern that is likely to extract unique information is selected for each document.

The determination unit 182 stores, for example, information in which the document ID and the pattern ID of the selected detection pattern are associated with each document in the detection pattern storage unit 12. FIG. 7 is a schematic diagram showing an example of information in which the document ID and the pattern ID of the selected detection pattern are associated with each document. The number of detection patterns selected by the determination unit 182 for one document is not limited to one, and may be two or more.

When a detection pattern is selected for each given document in the actual operation step, the information extraction unit 162 extracts a character string from the document using the detection pattern selected according to the document.

Extraction unit 16 (document acquisition unit 161 and information extraction unit 162), model learning unit 17 (first preprocessing unit 171 and learning unit 172), and detection pattern selection unit 18 (second preprocessing unit 181 and determination). The unit 182) is realized by, for example, a CPU (Central Processing Unit) of a computer that operates according to a detection pattern evaluation model generation program. In this case, the CPU reads the detection pattern evaluation model generation program from a program recording medium such as a computer program storage device, and according to the program, the extraction unit 16 (document acquisition unit 161 and information extraction unit 162) and the model learning unit 17 ( It may operate as a first preprocessing unit 171 and a learning unit 172), and a detection pattern selection unit 18 (second preprocessing unit 181 and a determination unit 182).

Next, the processing process of the embodiment of the present invention will be described. FIG. 8 is a flowchart showing an example of the processing progress of the model learning step of the embodiment of the present invention. It is assumed that a plurality of training documents are stored in advance in the document storage unit 11. In addition, the matters already described will be omitted as appropriate.

The extraction unit 16 performs the processes of steps S101 to S103 described below for each training document. That is, the extraction unit 16 repeats the processes of steps S101 to S103 for the number of training documents.

In step S101, the document acquisition unit 161 acquires one training document that has not yet been acquired from the document storage unit 11.

In the next step S102, the information extraction unit 162 extracts the character string corresponding to the detection pattern from the training document acquired in step S101 by using each detection pattern stored in the detection pattern storage unit 12. .. In step S102, the information extraction unit 162 extracts a character string corresponding to the detection pattern in word units by using a general technique such as morphological analysis. As described above, even if the detection patterns themselves are the same, if the named entity IDs are different, they are treated as different detection patterns.

Next, in step S103, the information extraction unit 162 records (FIG. 3) each of the detection patterns stored in the detection pattern storage unit 12 regarding the detection pattern for which the character string could be extracted in step S102. As shown in the above, a record including a detection ID, a data flag, a document ID, a pattern ID, a unique expression type ID, and an extracted character string) is created and stored in the extraction result storage unit 13. At this time, the information extraction unit 162 sets the value of the data flag to the initial value “−”. The information extraction unit 162 assigns the value of the detection ID to each newly extracted character string (in other words, assigns it to each new record). The value of the document ID is the value of the document ID of the training document acquired in step S101. The value of the pattern ID is the value of the pattern ID of the detection pattern from which the character string is extracted, and the value of the named entity type ID is the value of the named entity type ID corresponding to the detection pattern.

After executing the processes of steps S101 to S103 for all the training documents, the process proceeds to step S104.

In step S104, the user sets the value of the correct answer flag and creates the teacher data. The user determines a set of a document ID, a pattern ID, and a correct answer flag for each combination of the training document and the detection pattern stored in the detection pattern storage unit 12. At this time, the user confirms the record stored in the extraction result storage unit 13 including the document ID and the pattern ID included in the set for each set, and the character string included in the record is the unique information. Determine if it exists. Then, when the user determines that the character string is not unique information, the correct answer flag is set to "0", and all the values of the correct answer flags of the set in which the correct answer flag is not set to "0" are set to "1". do. Further, the user updates the value of the data flag (see FIG. 3) of the record confirmed in step S104 among the records stored in the extraction result storage unit 13 to “0”.

The user stores the set of the document ID, the pattern ID, and the correct answer flag defined in step S104 in the teacher data storage unit 14.

Next, the first preprocessing unit 171 executes preprocessing for processing by the learning unit 172 (step S105). Specifically, a vector corresponding to the teacher data is generated for each document ID of the training document (see FIG. 5), and a vector corresponding to the content of the training document is generated for each document ID of the training document. do. Since the method of generating the vector according to the teacher data has already been described, the description thereof will be omitted here. Further, since the method of generating the vector according to the contents of the training document has already been described, the description thereof will be omitted here.

In step S105, after the first preprocessing unit 171 generates two types of vectors for each document ID of the training document, the learning unit 172 generates the vector for each document ID of the training document in step S105. By performing supervised machine learning using each set, a model for deriving a score (a score indicating the possibility of extracting unique information from a given document) for each detection pattern is generated (step S106). For example, when a neural network model is used for machine learning, the learning unit 172 uses an error back propagation method or the like to obtain a model so that the result of correct answer data (teacher data) can be obtained as shown in FIG. Generate.

The learning unit 172 stores the model generated in step S106 in the model storage unit 15 (step S107). In step S107, the model learning step ends.

Next, the processing progress of the actual operation step will be described. FIG. 9 is a flowchart showing an example of the processing progress of the actual operation step of the embodiment of the present invention. It is assumed that the document storage unit 11 stores in advance a document to be extracted from the unique information in the actual operation step. The fact that the document to be extracted of the unique information is stored in the document storage unit 11 means that the document to be extracted of the unique information is given. In the following description, the matters already described will be omitted as appropriate.

Documents for which unique information is to be extracted in the actual operation step are hereinafter referred to as extraction target documents.

The detection pattern evaluation model generation system 1 performs the processes of steps S201 to S207 described below for each extraction target document. That is, the detection pattern evaluation model generation system 1 repeats the processes of steps S201 to S207 for the number of documents to be extracted.

In step S201, the document acquisition unit 161 acquires one extraction target document that has not yet been acquired from the document storage unit 11.

In the next step S202, the second preprocessing unit 181 executes the preprocessing of the processing of the determination unit 182. Specifically, the second preprocessing unit 181 generates a vector according to the content of the extraction target document acquired in step S201. Since the method by which the second preprocessing unit 181 generates the vector according to the content of the document has already been described, the description thereof will be omitted here.

In the next step S203, the determination unit 182 reads the model (the model generated in the model learning step) from the model storage unit 15, and applies the vector generated in step S202 to the model to obtain the detection pattern storage unit. The score of each detection pattern stored in 12 is derived. Therefore, in step S203, the determination unit 182 obtains a score indicating the possibility of extracting unique information from the extraction target document for each detection pattern. The higher the score of the detection pattern, the more likely it is that unique information will be extracted from the document to be extracted.

In the next step S204, a detection pattern satisfying the condition that the score obtained in step S203 is equal to or higher than the threshold value is selected. The number of detection patterns selected in step S204 is not limited to one, and may be two or more. It can be said that a detection pattern that satisfies the condition that the score is equal to or higher than the threshold value has a high possibility of extracting unique information from the document to be extracted. That is, it can be said that it is a useful detection pattern when extracting unique information from the document to be extracted.

In the next step S205, the determination unit 182 associates the document ID of the extraction target document acquired in step S201 with the pattern ID of each detection pattern selected in step S204 and stores it in the detection pattern storage unit 12. ..

In the next step S206, the information extraction unit 162 indicates each detection pattern indicated by the pattern ID stored in the detection pattern storage unit 12 in association with the document ID of the document to be extracted acquired in step S201 (that is, in step S204). Each detection pattern selected by the determination unit 182) is read from the detection pattern storage unit 12. Then, the information extraction unit 162 extracts the character string corresponding to each detection pattern from the extraction target document acquired in step S201.

Each detection pattern selected by the determination unit 182 is a detection pattern that satisfies the condition that the score is equal to or higher than the threshold value, and is a detection pattern that is likely to extract unique information from the extraction target document acquired in step S201. be. Therefore, it can be said that the character string extracted in step S206 is likely to be unique information.

In the next step S207, the information extraction unit 162 has a record (as shown in FIG. 3, a detection ID) with respect to the detection pattern for which the character string could be extracted from each detection pattern used in step S206. , A record including a data flag, a document ID, a pattern ID, a unique expression type ID, and an extracted character string) is created and stored in the extraction result storage unit 13. When the information extraction unit 162 creates a record in step S207, the information extraction unit 162 sets the value of the data flag to "1". Further, the information extraction unit 162 assigns the value of the detection ID to each newly extracted character string. The value of the document ID is the document ID of the extraction target document acquired in step S201. The value of the pattern ID is the value of the pattern ID of the detection pattern from which the character string is extracted in step S206, and the value of the named entity type ID is the value of the named entity type ID corresponding to the detection pattern.

The detection pattern evaluation model generation system 1 ends the actual operation step when the processes of steps S201 to S207 are executed for all the documents to be extracted.

After the end of the actual operation step, the user uses the character string included in the record whose data flag value is "1" among the records stored in the extraction result storage unit 13 as unique information. It may be judged that it is a character string with a high possibility.

According to the present embodiment, the model learning unit 17 sets a score indicating the possibility of extracting unique information from a given document by using a set of a set of a document ID, a pattern ID, and a correct answer flag as teacher data. A model for deriving each detection pattern stored in the detection pattern storage unit 12 is generated by machine learning. Therefore, according to the present embodiment, it is possible to generate a model capable of evaluating how suitable each of the detection patterns is for extracting a named entity from a given document.

Further, according to the present embodiment, when the document acquisition unit 161 acquires the extraction target document in step S201, the detection pattern selection unit 18 generates a vector corresponding to the content of the extraction target document in the model learning step. By applying it to the above model, the score is derived for each detection pattern. Therefore, the score can be used to evaluate how suitable each detection pattern is for extracting unique information from the extraction target document.

Further, the detection pattern selection unit 18 selects a detection pattern that satisfies the condition that the score is equal to or higher than the threshold value from the detection patterns stored in the detection pattern storage unit 12. Then, the information extraction unit 162 extracts a character string from the extraction target document acquired in step S201 using the selection pattern. Therefore, this character string is likely to be unique information. Therefore, it is possible to suppress over-detection that many character strings that are not originally unique information are detected, and mainly extract character strings that are highly likely to be unique information.

Further, the detection pattern selection unit 18 selects a detection pattern according to the extraction target document acquired by the document acquisition unit 161 in step S201. Therefore, the effect of suppressing over-detection can be enhanced.

Further, in step S205, the detection pattern selection unit 18 (determination unit 182) detects the document ID of the extraction target document acquired in step S201 in association with the pattern ID of each detection pattern selected in step S204. It is stored in the pattern storage unit 12. Therefore, for example, the information illustrated in FIG. 7 is stored in the detection pattern storage unit 12, and the user can confirm the logic for extracting the character string by the selected detection pattern by confirming the information. ..

FIG. 10 is a schematic block diagram showing a configuration example of a computer according to an embodiment of the present invention. The computer 1000 includes a CPU 1001, a main storage device 1002, an auxiliary storage device 1003, and an interface 1004.

The detection pattern evaluation model generation system 1 according to the embodiment of the present invention is mounted on the computer 1000. The operation of the detection pattern evaluation model generation system 1 is stored in the auxiliary storage device 1003 in the form of a detection pattern evaluation model generation program. The CPU 1001 reads the detection pattern evaluation model generation program from the auxiliary storage device 1003, deploys it to the main storage device 1002, and executes the process described in the above embodiment according to the detection pattern evaluation model generation program.

Auxiliary storage 1003 is an example of a non-temporary tangible medium. Other examples of non-temporary tangible media include magnetic disks, optical magnetic disks, CD-ROMs (Compact Disk Read Only Memory), DVD-ROMs (Digital Versatile Disk Read Only Memory), which are connected via interface 1004. Examples include semiconductor memory. Further, when this program is distributed to the computer 1000 by a communication line, the distributed computer 1000 may expand the program to the main storage device 1002 and execute the above processing.

Next, the outline of the present invention will be described. FIG. 11 is a block diagram showing an outline of the detection pattern evaluation model generation system of the present invention. The detection pattern evaluation model generation system of the present invention includes an extraction means 91 and a model generation means 92.

The extraction means 91 (for example, the extraction unit 16) extracts a character string corresponding to the detection pattern from the document for each document by using a plurality of detection patterns for extracting the unique expression from the document.

The model generation means 92 (for example, the model learning unit 17) determines whether or not the document, the detection pattern, and the character string extracted from the document based on the detection pattern correspond to the named entity (for example,). Using the set of pairs with the correct answer flag) as teacher data, a model for deriving the possibility of extracting named entities from a given document as a score for each detection pattern is generated by machine learning.

Such a configuration produces a model that can evaluate how well each of the multiple detection patterns for extracting named entities from a document is suitable for extracting named entity from a given document. can.

Further, when a document to be extracted of the eigenexpression is given, a score indicating the possibility of extracting the eigenexpression from the document for each detection pattern using the model generated by the model generation means 92 is used. Is provided, and a detection pattern selection means (for example, a detection pattern selection unit 18) for selecting a detection pattern to be used for extracting a unique expression from the document based on the score is provided, and the extraction means 91 extracts the unique expression. When a target document is given, a character string corresponding to the detection pattern may be extracted from the document using the detection pattern selected by the detection pattern selection means.

Further, it is determined that the model generation means 92 is a vector having an element having a one-to-one correspondence with the detection pattern based on the teacher data, and the character string extracted by one detection pattern corresponds to the named entity. In that case, the value of the element corresponding to the one detection pattern is set to 1, and when it is determined that the character string extracted by the one detection pattern does not correspond to the named entity, it corresponds to the one detection pattern. A vector in which the value of the element is 0 and a vector corresponding to the content of the document are created for each document, and a model is generated by machine learning using a set of vectors created for each document. May be good.

INDUSTRIAL APPLICABILITY The present invention is suitably applicable to a detection pattern evaluation model generation system that generates a model for evaluating a detection pattern for extracting a named entity from a document.

1 Detection pattern evaluation model generation system 11 Document storage unit 12 Detection pattern storage unit 13 Extraction result storage unit 14 Teacher data storage unit 15 Model storage unit 16 Extraction unit 17 Model learning unit 18 Detection pattern selection unit 161 Document acquisition unit 162 Information extraction unit 171 First pre-processing unit 172 Learning unit 181 Second pre-processing unit 182 Judgment unit

Claims (9)

An extraction means for extracting a character string corresponding to a detection pattern from a document for each document using a plurality of detection patterns for extracting a named entity from a document.
A unique expression from a given document, with a set of sets of a document, a detection pattern, and a judgment result of whether or not a character string extracted from the document based on the detection pattern corresponds to a unique expression as teacher data. A detection pattern evaluation model generation system characterized by having a model generation means for generating a model for deriving the possibility of extracting the data as a score for each detection pattern by machine learning. Given a document to extract named entities, a score indicating the possibility of extracting named entity from the document is derived for each detection pattern using the model generated by the model generation means. , A detection pattern selection means for selecting a detection pattern used to extract a named entity from the document based on the score.
The extraction method is
The first aspect of claim 1 is to extract a character string corresponding to the detection pattern from the document by using the detection pattern selected by the detection pattern selection means when the document to be extracted of the named entity is given. Detection pattern evaluation model generation system. Model generation means
If it is a vector having an element having a one-to-one correspondence with the detection pattern based on the teacher data and it is determined that the character string extracted by one detection pattern corresponds to the unique expression, the one detection pattern is described above. A vector in which the value of the element corresponding to 1 is set to 1, and the value of the element corresponding to the 1 detection pattern is set to 0 when it is determined that the character string extracted by one detection pattern does not correspond to the unique expression. And a vector according to the contents of the document are created for each document.
The detection pattern evaluation model generation system according to claim 1 or 2, wherein a model is generated by machine learning using a set of vectors created for each document. The computer
Using multiple detection patterns for extracting named entities from a document, the character string corresponding to the detection pattern is extracted from the document for each document.
A set of sets of a document, a detection pattern, and a judgment result of whether or not a character string extracted from the document based on the detection pattern corresponds to a unique expression is used as teacher data, and the named entity is expressed from a given document. A detection pattern evaluation model generation method characterized in that a model for deriving the possibility of extracting is generated as a score for each detection pattern by machine learning. The computer
Given the document to which the named entity is to be extracted,
Using the model, for each detection pattern, a score indicating the possibility of extracting the named entity from the document is derived, and based on the score, the detection pattern used to extract the named entity from the document is selected. death,
The detection pattern evaluation model generation method according to claim 4, wherein a character string corresponding to the detection pattern is extracted from the document using the selected detection pattern. The computer
When generating a model,
If it is a vector having an element having a one-to-one correspondence with the detection pattern based on the teacher data and it is determined that the character string extracted by one detection pattern corresponds to the unique expression, the one detection pattern is described above. A vector in which the value of the element corresponding to 1 is set to 1, and the value of the element corresponding to the 1 detection pattern is set to 0 when it is determined that the character string extracted by one detection pattern does not correspond to the unique expression. And a vector according to the contents of the document are created for each document.
The detection pattern evaluation model generation method according to claim 4 or 5, wherein a model is generated by machine learning using a set of vectors created for each document. On the computer
Extraction processing that extracts the character string corresponding to the detection pattern from the document for each document using multiple detection patterns for extracting the named entity from the document, and
A set of pairs of a document, a detection pattern, and a judgment result of whether or not a character string extracted from the document based on the detection pattern corresponds to a named entity corresponds to a named entity as teacher data, and the named entity is represented from a given document. A detection pattern evaluation model generation program for executing a model generation process that generates a model by machine learning to derive the possibility of extracting the data as a score for each detection pattern. On the computer
Given a document to be extracted with named entity, the model generated by the model generation process is used to derive a score indicating the possibility of extracting named entity from the document for each detection pattern. , A detection pattern selection process that selects a detection pattern used to extract a named entity from the document based on the score, and
A claim to execute a process of extracting a character string corresponding to the detection pattern from the document using the detection pattern selected in the detection pattern selection process when the document to be extracted of the unique expression is given. Item 7. The detection pattern evaluation model generation program according to Item 7. On the computer
In the model generation process
If it is a vector having an element having a one-to-one correspondence with the detection pattern based on the teacher data and it is determined that the character string extracted by one detection pattern corresponds to the unique expression, the one detection pattern is described above. A vector in which the value of the element corresponding to 1 is set to 1, and the value of the element corresponding to the 1 detection pattern is set to 0 when it is determined that the character string extracted by one detection pattern does not correspond to the unique expression. And a vector according to the contents of the document are created for each document.
The detection pattern evaluation model generation program according to claim 7 or 8, wherein a model is generated by machine learning using a set of vectors created for each document.

Images