JP7415495B2 - Document processing program, document processing device, and document processing method - Google Patents

Description

The present invention relates to document processing technology.

In recent years, natural language processing using AI (Artificial Intelligence) technology has been increasing in industry. Natural language processing includes morphological analysis, syntactic analysis, semantic analysis, anaphora analysis, etc. In natural language processing using AI technology, for example, document analysis is performed in the following steps.
(P1) The user manually creates learning data for machine learning from a document collection.
(P2) The user causes the learning model to learn learning data using machine learning, adjusts the parameters of the learning model, and generates an analytical model that is a trained model.
(P3) The user analyzes the unknown document using the analysis model and generates an analysis result.

The analysis result includes additional information for the text of the unknown document. For example, in morphological analysis, boundary positions between morphemes, parts of speech of each morpheme, etc. are generated as additional information. A set of input data and output data is used as learning data for machine learning. The input data is a document to be analyzed, and the output data is a document to which additional information is added.

Creating training data requires a large workload, so to reduce the workload, it is recommended to reuse the training data by partially changing the existing training data and reusing it to generate another analytical model. It has increased. Examples of reuse of learning data include the following usage patterns.
(a) In order to improve the performance of machine learning, the user reviews and changes existing learning data to create more accurate learning data.
(b) In order to improve the performance of machine learning, the user creates large-scale learning data by changing existing learning data and adding new learning data.
(c) Users modify existing learning data to customize the learning data based on their own criteria.

Related to natural language processing in the biomedical field, a technique for annotating named entities included in a document is known (see, for example, Non-Patent Document 1).

P. Stenetorp et al., “BioNLP Shared Task 2011: Supporting Resources”, Proceedings of BioNLP Shared Task 2011 Workshop, pages 112-120, 2011

Machine learning uses a considerable amount of learning data, so if only a portion of the learning data is changed, the change is treated as noise in the learning process and is unlikely to be reflected in the learning results. Therefore, in order to ensure that the changes are reflected in the learning results, it is desirable to make a large number of similar changes over the entire learning data, which increases the user's workload.

Note that this problem occurs not only when changing learning data used in machine learning for natural language processing, but also when changing information included in various document sets.

In one aspect, the present invention is directed to a computer extracting change candidates from a document collection. As a result, the modification work for modifying the information of the document collection becomes more efficient.

In one proposal, the document processing program is based on a change history that indicates that a user has changed additional information of text included in a document set and includes a change example that indicates a change operation performed by the user . Estimates the change target of changes made by the user to the document set, extracts text corresponding to the change target from the document set, and converts the text extracted from the document set to the text that exists before and after the text extracted from the document set. By clustering based on text, multiple clusters are generated, and when the user changes the additional information of the text that belongs to a specific cluster among the multiple clusters, the change to the additional information of the text that belongs to the specific cluster is A computer is caused to perform a process of reflecting the additional information of other texts belonging to a specific cluster and outputting change candidate information indicating the text extracted from the document collection.

According to one aspect, it is possible to streamline the modification work of modifying information on a document set.

It is a figure which shows the text in curation of a medical field. FIG. 1 is a functional configuration diagram of a document processing device. It is a flowchart of change support processing. FIG. 2 is a functional configuration diagram showing a specific example of a document processing device. It is a figure which shows the classification result of a change example. It is a figure which shows the evaluation value of a paragraph. FIG. 3 is a diagram showing a highlighted partial document. FIG. 3 is a diagram showing word vectors. 3 is a flowchart illustrating a specific example of change support processing. It is a flowchart of edit screen generation processing. It is a flowchart of estimation processing. FIG. 2 is a hardware configuration diagram of an information processing device.

Hereinafter, embodiments will be described in detail with reference to the drawings.

In the medical field, there are cases in which humans read papers related to diseases and the like while registering the knowledge contained in the papers in a database. This kind of work is called curation, and the worker is called a curator.

The technique of Non-Patent Document 1 is used to make curation more efficient. This technology uses natural language processing to highlight parts of a paper where knowledge is described, and adds annotations to the highlighted parts. Namely expressions representing genes, gene mutations, medicines, diseases, etc. are used as places where knowledge is described, and tags indicating the type of named entity, relationships between named entities, etc. are used as annotations. Curators can maintain their database by checking the text, focusing on the highlighted parts.

FIG. 1 shows an example of text displayed on the screen of a curator's terminal device in curation in the medical field. The text in Figure 1 represents a paper in the medical field written in English, and “p.R122W”, “FNMTC” (Familial Non-Medullary Thyroid Carcinoma), and “thyroid cancer” in the paper represent genetic mutations or diseases. It has been extracted as a unique expression related to The extracted text is highlighted with a marker.

“p.R122W” is a phrase that represents a specific genetic mutation, and “FNMTC” and “thyroid cancer” are phrases that represent a specific disease. “p.R122W” is tagged with “Mutation,” and “FNMTC” and “thyroid cancer” are tagged with “Disease.” Furthermore, the “Pathogenic” arrow is displayed as the relationship between “p.R122W” and “FNMTC,” and the “Pathogenic” arrow is displayed as the relationship between “p.R122W” and “thyroid cancer.” Displayed.

Based on these tags and relationships, the curator recognizes that the genetic mutation "p.R122W" is the cause of the diseases "FNMTC" and "thyroid cancer" and registers this knowledge in the database. Further, the curator can check whether or not the processing result of natural language processing contains an error based on the tag and relationship given to each named entity. Errors in the processing results include failure to extract a named entity, an error in extracting a named entity, and an error in tags or relationships.

Here, assume that a natural language processing analysis model is generated that extracts named entities from papers in the medical field and annotates the extracted named entities. A set of input data and output data is used as learning data for such an analytical model. The input data is a paper to be analyzed, and the output data is a paper with additional information added. The additional information includes the text range of each named entity, the tag for each named entity, and the relationship between the named entities.

In curation using an analytical model, when a curator registers knowledge in a database, it is also possible to correct errors in natural language processing and use the correction results as training data for the analytical model. As a result, the accuracy of natural language processing using analytical models will gradually improve, and curators' confirmation work will gradually become more efficient.

For example, in the processing results of an analysis model, if "advanced gastric cancer" is tagged as "disease," the curator can change the range of text corresponding to the tag from "advanced gastric cancer" to "gastric cancer." Can be changed. In this case, learning data is generated in which "gastric cancer" is tagged with "disease" instead of "advanced gastric cancer."

However, there is a problem in that when a curator makes at most a few changes, the changes are difficult to be reflected in the learning results. The reason for this is that machine learning uses a considerable amount of learning data, so even if you add at most a few changes, the added changes will be treated as noise in the learning process and will not be reflected in the learning results. Therefore, in order to reflect the changes in the learning results, it is desirable to make a large number of similar changes, which increases the curator's workload.

FIG. 2 shows an example of the functional configuration of the document processing device according to the embodiment. The document processing device 201 in FIG. 2 includes a storage section 211, an estimation section 212, an extraction section 213, and an output section 214. The storage unit 211 stores a change history 221 indicating that the user has changed information included in the document collection. The estimation unit 212, the extraction unit 213, and the output unit 214 perform change support processing using the change history 221.

FIG. 3 is a flowchart illustrating an example of change support processing performed by the document processing apparatus 201 of FIG. First, the estimating unit 212 estimates the change target of the change made by the user to the document collection based on the change history 221 (step 301). Next, the extraction unit 213 extracts the text corresponding to the change target from the document collection (step 302). Then, the output unit 214 outputs change candidate information indicating the text extracted from the document collection (step 303).

According to the document processing device 201 of FIG. 2, it is possible to make the change work of changing information of a document collection more efficient.

FIG. 4 shows a specific example of the document processing device 201 shown in FIG. The document processing device 401 in FIG. 4 includes a storage section 411, an analysis section 412, an estimation section 413, an extraction section 414, a classification section 415, a modification section 416, a generation section 417, and an output section 418. The storage unit 411, the estimation unit 413, the extraction unit 414, and the output unit 418 correspond to the storage unit 211, the estimation unit 212, the extraction unit 213, and the output unit 214 in FIG. 2, respectively.

The storage unit 411 stores an analysis model 421 and a pre-analysis document set 422. The analysis model 421 is a trained model of natural language processing generated by machine learning, and analyzes a document to be analyzed to generate an analysis result including additional information. The pre-analysis document set 422 includes multiple documents to be analyzed, and each document includes multiple partial documents. A partial document corresponds to a chapter, paragraph, sentence, etc.

The analysis unit 412 generates a document with additional information added by analyzing each document included in the pre-analysis document set 422 using the analysis model 421. Then, the analysis unit 412 generates an analyzed document set 423 including the generated documents, and stores it in the storage unit 411.

In the case of curation in the medical field, the pre-analysis document set 422 is, for example, a set of papers in the medical field, and the post-analysis document set 423 is, for example, a set of papers to which additional information has been added. In this case, the additional information includes the text range of each named entity, the tag for each named entity, and the relationship between the named entities.

A user can edit documents included in the post-analysis document set 423 via a user interface included in the document processing device 401 or a terminal device that can communicate with the document processing device 401.

The output unit 418 is a display device or a communication device, and outputs the document selected by the user from the analyzed document set 423. When the output unit 418 is a display device, the output unit 418 displays the document selected by the user on the screen. When the output unit 418 is a communication device, the output unit 418 transmits the document selected by the user to the user's terminal device via the communication network. The terminal device displays the document received from the document processing device 401 on the screen.

The user refers to the additional information added to the document displayed on the screen and performs a desired change operation. At this time, the user performs an operation to change the additional information by inputting a change instruction to change the additional information into the document processing device 401. The changing unit 416 receives the input change instruction and changes the additional information according to the received change instruction. Then, the change unit 416 generates a change history 424 that includes the change contents as a change example, and stores it in the storage unit 411. As a result, a plurality of change examples indicating change operations performed by the user are accumulated in the change history 424. Change history 424 corresponds to change history 221 in FIG. 2 .

The estimation unit 413 classifies each change example included in the change history 424 into one of a plurality of change types, and determines the next change operation to be performed by the user on the analyzed document set 423 based on each change example. Estimating the change target for each change type. The extraction unit 414 searches the analyzed document set 423 for text corresponding to the change target for each change type, and extracts it as a change candidate for each change type.

Types of changes include shortening the scope of a named entity, extending the scope of a named entity, changing the type of named entity, adding a named entity, deleting a named entity, adding a relationship, deleting a relationship, changing the type of relationship, etc. can be used. The change target is information that specifies the text before change indicated by the change example, and also specifies a text that includes some words or synonyms of the text before change and is different from the text before change. Specific examples of change cases of each change type and change targets estimated from the change cases are as follows.

(C1) Shortening the range of named entity The text range of named entity is shortened. Examples of changes include, for example, changing "advanced gastric cancer: medicine" to "gastric cancer: medicine." “advanced gastric cancer: drug” indicates that the scope of the unique expression before the change is “advanced gastric cancer” and that the tag given to “advanced gastric cancer” is “drug.” On the other hand, "gastric cancer: medicine" means that the scope of the changed proper expression is "gastric cancer", and the tag given to "gastric cancer" is "medicine".

In the case of shortening the range of a named entity, a named entity that includes the last term of the changed named entity or a synonym thereof and has the same tag as the changed named entity can be used as the changed entity. The word at the end of the modified named entity is also part of the original named entity. Synonyms are determined by a thesaurus. In this case, the type of change candidate is a named entity.

For example, "* gastric cancer: drug" can be used as a change target estimated from a change example in which "advanced gastric cancer: drug" is changed to "gastric cancer: drug." “*” represents any character string. Therefore, unique expressions such as "advanced gastric cancer" and "progressive gastric cancer" that are tagged with "medicine" are extracted as modification candidates.

“* cancer: medicine” can also be used as the change target. In this case, in addition to specific expressions such as "advanced gastric cancer" and "progressive gastric cancer", specific expressions such as "advanced colon cancer" and "progressive colon cancer" are also extracted as modification candidates.

(C2) Extension of the scope of a named entity The text scope of a named entity is extended. Examples of changes include, for example, changing "gastric cancer: medicine" to "advanced gastric cancer: medicine." In the case of extending the range of a named entity, a named entity that includes the last term or a synonym of the named entity before change and has the same tag as the named entity before change can be used as the change target. In this case, the type of change candidate is a named entity.

For example, "* gastric cancer: medicine" or "* cancer: medicine" can be used as a change target estimated from a change example in which "gastric cancer: medicine" is changed to "advanced gastric cancer: medicine."

(C3) Changing the type of named entity The tag of the named entity is changed. Examples of changes include changing "gastric cancer: medicine" to "gastric cancer: cancer." In the case of changing the type of named entity, a named entity that includes the last words or synonyms of the named entity before and after the change and has the same tag as the named entity before the change can be used as the change target. In this case, the type of change candidate is a named entity.

For example, "* gastric cancer: medicine" or "* cancer: medicine" can be used as the change target estimated from a change example in which "gastric cancer: medicine" is changed to "gastric cancer: cancer."

(C4) Addition of named entity The text range and tag of the named entity are added to the additional information. An example of a change is, for example, changing "-" to "nivolumab: drug". “-” indicates that the range of the unique expression before change is not specified. On the other hand, "nivolumab: medicine" indicates that the scope of the changed named entity is "nivolumab" and that the tag given to "nivolumab" is "medicine."

In the case of adding a named entity, the word at the end of the changed named entity or its synonym can be used as the change target. The word at the end of the modified named entity is also part of the original named entity. In this case, the type of change candidate is a character string.

For example, "nivolumab" or "AAAAAA" can be used as a change target estimated from a change example in which "-" is changed to "nivolumab: drug." “AAAAAA” represents a synonym for “nivolumab”. In this case, "nivolumab" or "AAAAAA" is extracted as a change candidate.

(C5) Deletion of named entity The text range and tag of the named entity are deleted from the additional information. An example of a change is, for example, changing "nivolumab: drug" to "-". In the case of deletion of a named entity, a named entity that includes the last term of the named entity before change or a synonym thereof and has the same tag as the named entity before change can be used as the change target. In this case, the type of change candidate is a named entity.

For example, "* nivolumab: drug" or "* AAAAAA: drug" can be used as a change target estimated from a change example in which "nivolumab: drug" is changed to "-". In this case, unique expressions such as "nivolumab" and "AAAAAA" that are tagged as "medicine" are extracted as modification candidates.

(C6) Addition of relationship The relationship between named entities is added to the additional information. Examples of changes include, for example, adding the relationship "effective" between "gefitinib (drug)" and "lung cancer (disease)."

In the case of adding a relationship, a pair of named entity E1 and named entity E2 to which no relationship is assigned can be used as a change target. The named entity E1 represents a named entity that includes the last term or a synonym thereof of one of the named named entities before change, and is given the same tag as that named named entity. The named entity E2 represents a named entity that includes the last term of the other named entity before change or a synonym thereof, and is given the same tag as that named entity. In this case, the type of change candidate is a set of named entities to which no relationship is assigned.

For example, the change target estimated from the example of a change that adds the relationship "effective" between "gefitinib: drug" and "lung cancer: disease" is that "* gefitinib: drug" and "lung cancer: disease" that appear in the same sentence. A combination with “* cancer: disease” can be used. However, only pairs to which no relationship is assigned are designated as change targets. In this case, a set of "gefitinib: medicine" and "lung cancer: disease", a set of "gefitinib: medicine" and "gastric cancer: disease", etc. are extracted as change candidates.

(C7) Deletion of relationship The relationship between named entities is deleted from the additional information. Examples of changes include, for example, the deletion of the relationship ``effective'' between ``gefitinib (drug)'' and ``lung cancer (disease)''.

In the case of deletion of a relationship, the pair of named entity E1 and named entity E2 to which the relationship before change has been assigned can be used as a change target. The named entity E1 represents a named entity that includes the last term or a synonym thereof of one of the named named entities before change, and is given the same tag as that named named entity. The named entity E2 represents a named entity that includes the last term of the other named entity before change or a synonym thereof, and is given the same tag as that named entity. In this case, the type of change candidate is a set of named entities to which a relationship is attached.

For example, an example of a change that can be estimated from a change example that deletes the relationship "effective" between "gefitinib: drug" and "lung cancer: disease" is when "* gefitinib: drug" and "lung cancer: disease" appear in the same sentence. A combination with “* cancer: disease” can be used. However, only the groups to which the relationship of "effective" has been assigned are designated as change targets. In this case, the combination of ``gefitinib: drug'' and ``lung cancer: disease'' and the pair of ``gefitinib: drug'' and ``gastric cancer: disease,'' which are given the relationship ``effective,'' are candidates for change. Extracted.

(C8) Change of relationship type The relationship between named entities is changed. Examples of changes include, for example, changing the relationship between "gefitinib (drug)" and "lung cancer (disease)" from "effective" to "ineffective."

In the case of changing the type of relationship, the set of named entity E1 and named entity E2 to which the relationship before change has been assigned can be used as a change target. The named entity E1 represents a named entity that includes the last term or a synonym thereof of one of the named named entities before change, and is given the same tag as that named named entity. The named entity E2 represents a named entity that includes the last term of the other named entity before change or a synonym thereof, and is given the same tag as that named entity. In this case, the type of change candidate is a set of named entities to which a relationship is attached.

For example, the change target estimated from a change case in which the relationship between “gefitinib: drug” and “lung cancer: disease” is changed from “effective” to “ineffective” is “*” that appears in the same sentence. A combination of "gefitinib: drug" and "* cancer: disease" can be used. However, only the groups to which the relationship of "effective" has been assigned are designated as change targets.

By using the change targets shown in (C1) to (C8) as estimation results, it is possible to calculate not only the named entity before change indicated by the change example, but also some words or phrases of the named entity before change or their synonyms. You can include another named entity in the change target. This makes it possible to predict in advance the change operation that the user will perform next, and to extract change candidates from the analyzed document set 423 based on the prediction results.

After the change candidates for each change type are extracted, the estimation unit 413 calculates the appearance frequency of the change candidates for each change type in the post-analysis document set 423, and based on the appearance frequency, selects a specific change type from among the plurality of change types. Select the change type.

Furthermore, the estimation unit 413 calculates the evaluation value of each partial document based on the appearance frequency of change candidates for each change type and the number of change candidates for each change type included in each partial document in each document. , select a specific partial document based on the calculated evaluation value.

FIG. 5 shows an example of classification results of change cases included in the change history 424. The change ID is identification information of the change type, the pre-change additional information represents additional information before the change operation indicated by the change example, and the post-change additional information indicates when the change operation indicated by the change example was performed. Represents later additional information. The change target represents the change target estimated from the change example, the change candidate type represents the type of text corresponding to the change target, and the case score 1 and case score 2 represent evaluation values of the change type.

Change ID "1" indicates a shortening of the range of the named entity, and a change example in which "progressive gastric cancer: disease" is changed to "gastric cancer: disease" is classified as change ID "1". In this example, "* gastric cancer: disease" is determined to be changed.

Change ID “2” indicates a change in the type of named entity, and a change example in which “AAAAAA: product” is changed to “AAAAAA: drug” is classified as change ID “2”. In this example, “*AAAAAA:Product” or “nivolumab:Product” is determined to be changed.

Change ID “3” indicates addition of a named entity, and a change example in which “-” is changed to “nivolumab: drug” is classified as change ID “3”. In this example, "nivolumab" or "AAAAAA" is determined to be changed.

Change ID “4” indicates deletion of a named entity, and a change example in which “tumor: disease” is changed to “-” is classified as change ID “4”. In this example, "*tumor: disease" is determined to be the change target.

Change ID “5” indicates deletion of a relationship, and a change example that deletes the relationship “=>” between “gefitinib: drug” and “lung cancer: disease” is classified as change ID “5”. ing. In this example, the pair of "* gefitinib: drug" and "* lung cancer: disease" to which the relationship "=>" is attached is determined to be the change target.

FIG. 6 shows an example of evaluation values for a paragraph, which is an example of a partial document. The paragraph ID is identification information of each paragraph in each document included in the post-analysis document set 423, and the paragraph score represents the evaluation value of the paragraph. The change ID represents the change ID of the change type corresponding to the change candidate extracted from the paragraph. In the example of FIG. 6, only paragraphs “1” to “4” are shown for simplicity, but the post-analysis document set 423 may include more paragraphs.

For each change candidate extracted from each paragraph, the estimation unit 413 identifies a change target corresponding to the change candidate, and obtains a change ID of the change type to which the identified change target belongs. For example, paragraph “1” includes one change candidate corresponding to each of change ID “1”, change ID “2”, and change ID “4”. Furthermore, paragraph “2” includes one change candidate corresponding to each of change ID “3” and change ID “4”.

The estimation unit 413 calculates the appearance frequency of change candidates of each change type based on the change IDs of all change candidates extracted from all paragraphs, and records the calculated appearance frequency as case score 1 in FIG. . For example, since the change ID "1" is included in each of paragraphs "1", "3", and "4", the case score 1 of the change ID "1" is "3". Further, since only one change ID “2” is included in the paragraph “1”, the case score 1 of the change ID “2” is “1”.

Next, the estimating unit 413 records the sum of case scores 1 of change IDs included in each paragraph as the paragraph score in FIG. 6 . For example, the paragraph score of paragraph “1” is the sum of case scores 1 of change ID “1”, change ID “2”, and change ID “4”, and the paragraph score is “8”. Further, the paragraph score of paragraph “2” is the sum of case score 1 of change ID “3” and change ID “4”, and the paragraph score is “7”.

Next, the estimating unit 413 calculates the total paragraph score of the paragraphs including the change candidates of each change ID, and calculates the case score 2 by multiplying the sum of the paragraph scores by the case score 1 of the change ID.

For example, since change ID "1" is included in paragraph "1", paragraph "3", and paragraph "4", the total paragraph score is 8+11+10=29. Since the case score 1 of the change ID "1" is "3", the case score 2 of the change ID "1" is 3*29=87.

Furthermore, since the change ID "2" is included only in the paragraph "1", the total paragraph score is "8". Since the case score 1 of the change ID "2" is "1", the case score 2 of the change ID "2" is 1*8=8.

Next, the estimating unit 413 selects change types in descending order of case score 2, and selects M paragraphs (M is an integer greater than or equal to 1) in descending order of paragraph scores as specific partial documents.

The generation unit 417 generates change candidate information including information for highlighting the selected specific partial document, and the output unit 418 outputs the generated change candidate information.

When the output unit 418 is a display device, the output unit 418 displays a document including a specific partial document on the screen, and highlights the text of the specific partial document. As a form of highlighting, changing the display color of the text, adding a marker or a surrounding frame to the text, etc. can be used. When the output unit 418 is a communication device, the output unit 418 transmits the document including the specific partial document and the change candidate information to the user's terminal device via the communication network. The terminal device displays the document received from the document processing device 401 on the screen and highlights the text of a specific partial document.

FIG. 7 shows an example of a highlighted partial document. “BBB mutation”, “gastric cancer”, “celecoxib”, “advanced gastric cancer”, “gefitinib”, “bladder cancer”, and “lung cancer” represent candidate changes, and paragraph 701 is the highlighted partial document. represents. The paragraph 701 is highlighted by changing the display color, using a marker, a surrounding frame, or the like.

For example, "gastric cancer" and "advanced gastric cancer" are change candidates corresponding to the change target "* gastric cancer: disease." This change target is estimated from, for example, a change example in which "progressive gastric cancer: disease" is changed to "gastric cancer: disease."

The tag "disease" attached to "gastric cancer" and "advanced gastric cancer" can be displayed on the screen by clicking on the text of each candidate change. Furthermore, the relationship between the change candidates is displayed on the screen by clicking the arrows between the change candidates.

The user checks the change candidates in the highlighted paragraph, the tags attached to the change candidates, and the relationships between the change candidates, and performs a desired change operation on the additional information. The changing unit 416 receives a change instruction input by the user, and changes the additional information according to the received change instruction. Then, the changing unit 416 adds a change example indicating the content of the change to the change history 424.

In this way, by providing the estimation unit 413 and the extraction unit 414, when the user changes the additional information included in the post-analysis document set 423, change candidates related to the change example are automatically extracted and the user can Presented. The presented change candidate may be the same text as the change example that appears in a different position from the change example in the post-analysis document set 423, or may be text similar to the change example but slightly different. There is also.

The user can perform the same modification operation as the modification example on the analyzed document set 423 by simply checking and modifying the presented modification candidates. Therefore, there is no need to manually search for new change candidates, making the change work more efficient. In this case, the possibility that the change candidates in the highlighted partial document will be changed increases, and the possibility that the change candidates in other partial documents will be changed becomes low.

For example, in the case of curation in the medical field, a curator who is a user is likely to change additional information similar to additional information changed in the past.

As an example, a curator changes the scope of a named entity tagged “disease” from “advanced gastric cancer” to “gastric cancer,” and also adds modified expressions to tags for some diseases. Assume the case where it is excluded from the target. In this case, the curator may repeat the change operation to exclude the modified expression of disease from being tagged.

As another example, assume that the curator excludes unique expressions such as "melancholia" (depression) and "postpartum depression" (postpartum depression) that have been tagged as "illness" from being tagged. In this case, the curator may repeat the change operation of excluding expressions related to mental illness from being tagged because he or she is not interested in mental illness.

As yet another example, assume that the curator deletes some relationships between named entities related to "gefitinib" which is tagged as "drug". In this case, because the FDA (Food and Drug Administration) certification of gefitinib has been revoked, the curator may repeat the change operation to remove all relationships related to ``gefitinib'' with the tag ``drug.'' There is.

Therefore, by estimating the change target based on change examples indicating change operations performed by the user in the past, it becomes possible to extract and present change candidates that are likely to be changed by the user. By presenting change candidates to the user and adding change candidates actually changed by the user to the change history 424 as change examples, information on change operations by the user is accumulated, and the accuracy of estimating the change target is improved.

When presenting change candidates to the user, calculate the evaluation value of the partial document based on the number of change candidates included in each partial document, and select the partial document to be highlighted based on the evaluation value of the partial document. This allows partial documents containing more change candidates to be presented preferentially. Therefore, the user can intensively perform multiple modification operations on the presented partial document, and the modification work can be made more efficient.

By estimating the change target for each change type such as (C1) to (C8) described above, it becomes possible to present appropriate change candidates according to the characteristics of the change type. For example, if "advanced gastric cancer: medicine" is changed to "gastric cancer: medicine", "advanced gastric cancer: medicine", "advanced colon cancer: medicine", "progressive colon cancer: medicine", etc. are candidates for change. presented as.

Furthermore, when "gastric cancer: disease" is changed to "gastric cancer: cancer", "gastric cancer: disease", "colon cancer: disease", etc. are presented as change candidates. If the relationship between “gefitinib: drug” and “lung cancer: disease” is deleted, the pair “gefitinib: drug” and “lung cancer: disease” with the same relationship, “gefitinib: drug” and “colorectal cancer: disease” are presented as candidates for change.

Note that the change candidates included in the highlighted partial document are change candidates that are likely to be changed by the user, but are not necessarily the change candidates that the user desires. If there is no need to change the change candidate of the highlighted partial document, the user requests the document processing device 401 to present another change candidate without performing a change operation. In this case, the document processing device 401 highlights the partial document having the next highest paragraph score.

In the change example shown in FIG. 5, the additional information added to the text of the document included in the post-analysis document set 423 is changed, but the user can also change the text itself of the document included in any document set. can. Even when a user changes the text of a document, change candidates related to the change example are automatically extracted and presented to the user in the same way as when additional information is changed.

Even if change candidates are automatically extracted and presented to the user, the user's workload increases if the user reviews and changes the many presented change candidates one by one. Therefore, it is desirable that the user only need to make a few similar changes and the changes will be reflected in the entire post-analysis document set 423.

Therefore, the classification unit 415 generates a plurality of clusters by clustering the text of the change candidate extracted from the post-analysis document set 423 based on the texts that exist before and after the text.

For example, the classification unit 415 generates a plurality of clusters by clustering change candidates of the change type selected by the estimation unit 413 based on text existing before and after the change candidates. Each cluster includes one or more change candidates. The classification unit 415 then stores the generated clusters in the storage unit 411 as classification results 425.

Hierarchical clustering or non-hierarchical clustering can be used as a clustering algorithm. For example, when the k-means method, which is an example of non-hierarchical clustering, is adopted, the classification result 425 can be generated by the following procedure.
(P11) The classification unit 415 vectorizes W words (W is an integer of 1 or more) before and after the text of the change candidate using a bag of words, thereby generating a word vector representing the context around the change candidate. generate.
(P12) The classification unit 415 classifies the generated word vectors into C clusters (C is an integer of 2 or more) using the k-means method. As the distance function of the k-means method, a cosine distance, Euclidean distance, Mahalanobis distance, etc. between feature vectors can be used.

FIG. 8 shows an example of word vectors generated from change candidates. The appearance position ID is the identification information of the change candidate in the post-analysis document set 423, the immediately preceding W word represents the W words that appear immediately before the change candidate in the document, and the immediately following W word represents the W words that appear immediately before the change candidate in the document. represents the W words that appear immediately after the change candidate. In this example, W=3. The word vector represents a word vector generated by a bag of words from the immediately preceding W word and the immediately following W word.

For example, immediately before the change candidate with appearance position ID "1", three words "aa", "bb", and "cc" appear, and immediately after, "dd", "ee", and Three words "ff" appear. Immediately before the change candidate with appearance position ID "2", three words "dd", "ee", and "gg" appear, and immediately after that, "aa", "ee", and "cc" appear. ” three words appear. Immediately before the change candidate with appearance position ID "3", three words "aa", "bb", and "dd" appear, and immediately after that, "ee", "ff", and "gg" appear. ” three words appear.

Each element of the word vector represents the number of times each word appears in the order [aa, bb, cc, dd, ee, ff, gg]. For example, "aa", "bb", "cc", "dd", "ee", and "ff" appear once in the W word immediately before and immediately after the appearance position ID "1". However, since "gg" does not appear, the word vector becomes [1, 1, 1, 1, 1, 1, 0].

When the user changes the additional information of a change candidate that belongs to a specific cluster among the C clusters included in the classification result 425, the change unit 416 changes the additional information according to the change operation, and also changes the additional information of a change candidate that belongs to the same cluster. The additional information of other change candidates is also changed in the same way. Then, the changing unit 416 adds the changes to all the change candidates of the cluster to the change history 424 as change cases. Thereby, the change operation performed by the user is automatically reflected on other change candidates belonging to the same cluster.

In this way, by providing the classification unit 415, even if the change candidates belong to the same change type, each change candidate can be classified into different clusters according to the surrounding context. By simply changing a representative change candidate included in the highlighted partial document among the change candidates included in each cluster, the user can simultaneously change other change candidates belonging to the same cluster.

By the way, clustering often takes a long time, and even if it is fast, the amount of calculation for N change candidates is O(N^2). If only the unique expression before change indicated by the change example is used as the change target, each time the user performs a new change operation targeting a change candidate different from the previous one, the search and clustering of the change candidate will be performed. be done. In this case, each time a new change operation is performed, a waiting time is generated to wait for the completion of clustering, which reduces work efficiency.

For example, if the user performs the first change operation to change “advanced gastric cancer: medicine” to “gastric cancer: medicine”, “advanced gastric cancer: medicine” will be changed from other parts of the post-analysis document set 423. It is searched and extracted as a change candidate. The extracted change candidates are then clustered and presented to the user. When the user performs a second change operation to change a representative change candidate included in each cluster, all change candidates belonging to the same cluster are changed at once.

Next, when the user performs a third change operation to change “advanced colon cancer: medicine” to “colon cancer: medicine”, “advanced colon cancer: medicine” is changed from other parts of the post-analysis document set 423. is searched and extracted as a change candidate. The extracted change candidates are then clustered and presented to the user. When the user performs a fourth change operation to change a representative change candidate included in each cluster, all change candidates belonging to the same cluster are changed at once. In this case, a waiting time occurs between the third change operation and the fourth change operation, resulting in a decrease in work efficiency.

On the other hand, as mentioned above, by including another named entity that includes some of the words or synonyms of the named entity before the change into the change target, the named entity that is similar to the named entity before the change can also be changed. , it becomes possible to complete the search and clustering in advance. Therefore, work efficiency is improved compared to the case where only the unique expression before the change is used as the change target.

For example, when the user performs the first change operation, "advanced gastric cancer: medicine" and "advanced colon cancer: medicine" are also searched from other parts of the post-analysis document set 423 and extracted as modification candidates. . The extracted change candidates are then clustered and presented to the user.

In this case, the presented change candidates include "advanced gastric cancer: medicine" and "advanced colon cancer: medicine," so the user can change both of the change candidates. Thereby, the user can perform the second change operation and the fourth change operation simultaneously, and there is no waiting time between the third change operation and the fourth change operation. Therefore, work efficiency is improved compared to when only "advanced gastric cancer: medicine" is used as the change target.

The classification unit 415 can perform clustering on change candidates for each of a plurality of change types selected in descending order of case score 2, and can generate a classification result 425 for each change type. The number of clustering processes that can be executed simultaneously is determined by the performance of the document processing device 401. For example, if the document processing device 401 can simultaneously execute P clustering processes (P is an integer of 1 or more), the classification unit 415 performs the clustering process on P change types selected in descending order of case score 2. Execute.

By selecting change types in descending order of case score 2, it is possible to preferentially perform clustering of change types that include more change candidates. Therefore, clustering of change candidates that are likely to be changed by the user can be quickly completed and the change candidates can be presented to the user.

When the user finishes editing the post-analysis document set 423, the post-analysis document set 423 is used as new learning data for the analysis model 421. The document processing device 401 updates the analysis model 421 by making the analysis model 421 learn the analyzed document set 423 through machine learning, adjusting the parameters of the analysis model 421. Thereby, the editing result of the analyzed document set 423 can be reflected in the analysis model 421.

FIG. 9 is a flowchart showing a specific example of change support processing performed by the document processing apparatus 401 of FIG. First, the changing unit 416 receives a request for an edit screen from the user (step 801), and the document processing device 401 generates an edit screen for the analyzed document set 423 (step 802). Then, the output unit 418 outputs the generated editing screen (step 803).

Next, the changing unit 416 receives a change instruction from the user for a change candidate included in the editing screen as a change operation for the change candidate, and changes the additional information of the change candidate according to the received change operation (step 804). Then, the changing unit 416 similarly changes the additional information of other change candidates belonging to the same cluster as the change candidate, and adds the change contents for all the change candidates of the cluster to the change history 424 as change examples.

FIG. 10 is a flowchart showing an example of the editing screen generation process in step 802 of FIG. First, the estimation unit 413 estimates the change target of the next change operation performed by the user for each change type based on the change examples included in the change history 424 (step 901). Then, the extraction unit 414 extracts change candidates corresponding to change targets for each change type from the analyzed document set 423.

Next, the classification unit 415 clusters change candidates belonging to a specific change type to generate a classification result 425 including a plurality of clusters (step 902). Next, the estimation unit 413 selects a specific paragraph based on the paragraph score of each paragraph included in the post-analysis document set 423 (step 903). Then, the generation unit 417 generates change candidate information including information for highlighting a specific paragraph, and the output unit 418 outputs an editing screen including a document including the specific paragraph and the generated change candidate information. (Step 904).

FIG. 11 is a flowchart showing an example of the estimation process in step 901 of FIG. First, the estimation unit 413 selects K recent change cases (K is an integer of 1 or more) included in the change history 424, and classifies each selected change case into one of a plurality of change types. (Step 1001). As K, for example, an integer in the range of 10 to 100 can be used.

Next, the estimation unit 413 estimates a change target for each change type, and the extraction unit 414 extracts change candidates corresponding to the change targets for each change type from the analyzed document set 423 (step 1002).

Next, the estimation unit 413 calculates the case score 1 for each change type based on the change candidates extracted for each change type (step 1003), and calculates the paragraph score of each paragraph using the case score 1. (Step 1004). Then, the estimating unit 413 calculates a case score 2 for each change type using the case score 1 and the paragraph score (step 1005), and selects a specific change type using the case score 2 (step 1006).

The configurations of the document processing device 201 in FIG. 2 and the document processing device 401 in FIG. 4 are merely examples, and some components may be omitted or changed depending on the use or conditions of the document processing device. For example, in the document processing device 401 of FIG. 4, if the analyzed document set 423 is stored in the storage unit 411 in advance, the analysis unit 412 can be omitted. If clustering of change candidates is not performed, the classification unit 415 can be omitted.

The flowcharts in FIGS. 3 and 9 to 11 are merely examples, and some processes may be omitted or changed depending on the configuration or conditions of the document processing device. For example, in the document processing apparatus 401 of FIG. 4, if clustering of change candidates is not performed, the process of step 902 of FIG. 10 can be omitted.

The texts shown in FIGS. 1 and 8 are merely examples, and documents in various fields can be used as documents to be edited. The document to be edited is not limited to learning data used in machine learning, but may be other documents. The object of the user's modification operation may be the text of the document, or may be additional information added to the text of the document.

Case score 1 and case score 2 shown in FIG. 5 are only examples, and the evaluation value of the change type may be calculated using another calculation method. The paragraph score shown in FIG. 6 is only an example, and the evaluation value of the partial document may be determined using another calculation method. The word vectors shown in FIG. 8 are merely examples, and word vectors may be obtained using other methods.

FIG. 12 shows an example of the hardware configuration of an information processing device (computer) used as the document processing device 201 in FIG. 2 and the document processing device 401 in FIG. 4. The information processing apparatus in FIG. 12 includes a CPU (Central Processing Unit) 1101, a memory 1102, an input device 1103, an output device 1104, an auxiliary storage device 1105, a medium drive device 1106, and a network connection device 1107. These components are hardware and are connected to each other by a bus 1108.

The memory 1102 is, for example, a semiconductor memory such as a ROM (Read Only Memory), a RAM (Random Access Memory), or a flash memory, and stores programs and data used for processing. The memory 1102 can be used as the storage unit 211 in FIG. 2 or the storage unit 411 in FIG. 4.

The CPU 1101 (processor) operates as the estimating unit 212 and the extracting unit 213 in FIG. 2 by executing a program using the memory 1102, for example. The CPU 1101 also operates as the analysis unit 412, estimation unit 413, extraction unit 414, classification unit 415, modification unit 416, and generation unit 417 in FIG. 4 by executing a program using the memory 1102.

The input device 1103 is, for example, a keyboard, a pointing device, or the like, and is used to input instructions or information from an operator or user. The output device 1104 is, for example, a display device, a printer, a speaker, or the like, and is used for making inquiries or instructions to an operator or user, and outputting processing results. The instruction from the user may be a change operation, and the processing result may be a highlighted partial document. The output device 1104 can be used as the output unit 214 in FIG. 2 or the output unit 418 in FIG. 4.

The auxiliary storage device 1105 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, or the like. Auxiliary storage 1105 may be a hard disk drive or flash memory. The information processing device can store programs and data in the auxiliary storage device 1105 and load them into the memory 1102 for use. The auxiliary storage device 1105 can be used as the storage unit 211 in FIG. 2 or the storage unit 411 in FIG. 4.

A medium drive device 1106 drives a portable recording medium 1109 and accesses the recorded contents. The portable recording medium 1109 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, or the like. The portable recording medium 1109 may be a CD-ROM (Compact Disk Read Only Memory), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, or the like. An operator or user can store programs and data in this portable recording medium 1109 and load them into the memory 1102 for use.

In this way, a computer-readable recording medium that stores programs and data used for processing is a physical (non-temporary) recording medium such as the memory 1102, the auxiliary storage device 1105, or the portable recording medium 1109. It is a medium.

The network connection device 1107 is a communication interface circuit that is connected to a communication network such as a LAN (Local Area Network) or a WAN (Wide Area Network) and performs data conversion associated with communication. The information processing device can receive programs and data from an external device via the network connection device 1107, load them into the memory 1102, and use them. The network connection device 1107 can be used as the output section 214 in FIG. 2 or the output section 418 in FIG. 4.

Note that the information processing device does not need to include all the components shown in FIG. 12, and some components can be omitted depending on the application or conditions. For example, if the portable recording medium 1109 or the communication network is not used, the medium drive device 1106 or the network connection device 1107 may be omitted.

Having described the disclosed embodiments and their advantages in detail, those skilled in the art will appreciate that various modifications, additions, and omissions can be made without departing from the scope of the invention as clearly set forth in the claims. Dew.

Regarding the embodiment described with reference to FIGS. 1 to 12, the following additional notes are further disclosed.
(Additional note 1)
Estimating a change target of a change made by the user to the document set based on a change history indicating that the user has changed information included in the document set;
extracting text corresponding to the change target from the document set;
outputting change candidate information indicating text extracted from the document set;
A document processing program that allows a computer to perform processing.
(Additional note 2)
The process of estimating the change target is executed before a change instruction from the user is input to the text extracted from the document set.
The document processing program according to supplementary note 1, characterized in that:
(Additional note 3)
The change history includes a change example indicating a change operation performed by the user,
The change target specifies the text before change indicated by the change example, includes some words or phrases in the text before change, or synonyms of the some words, and is different from the text before change. information that identifies different texts,
The document processing program according to supplementary note 1 or 2, characterized in that:
(Additional note 4)
The information included in the document set is additional information of the text included in the document set,
The document processing program is
Generating a plurality of clusters by clustering the text extracted from the document set based on texts existing before and after the text extracted from the document set,
When the user changes the additional information of the text belonging to a specific cluster among the plurality of clusters, the change to the additional information of the text belonging to the specific cluster is changed to the additional information of other texts belonging to the specific cluster. To reflect,
The document processing program according to appendix 3, further causing the computer to execute processing.
(Appendix 5)
The change history includes a plurality of change examples indicating change operations performed by the user,
The process of estimating the change target is performed for each of the plurality of change types in which each of the plurality of change cases is classified,
The process of extracting the text includes a process of extracting text for the change target for each type,
The clustering process is performed on text corresponding to a specific type of change target selected based on the appearance frequency of text corresponding to the change target for each type in the document set.
The document processing program according to appendix 4, characterized in that:
(Appendix 6)
The document set includes a plurality of documents,
Each of the plurality of documents includes a plurality of partial documents,
The computer includes:
of each of the plurality of partial documents based on the appearance frequency of the text corresponding to the change target for each type and the number of texts corresponding to the change target of each type included in each of the plurality of partial documents. Calculate the evaluation value,
Selecting a specific partial document from the plurality of partial documents based on the evaluation value of each of the plurality of partial documents,
The change candidate information includes information that highlights the specific partial document.
The document processing program according to appendix 5, characterized in that:
(Appendix 7)
The document set is learning data for machine learning that generates an analysis model, and the analysis model analyzes a document to be analyzed to generate additional information of text included in the document to be analyzed.
7. The document processing program according to any one of Supplementary Notes 4 to 6.
(Appendix 8)
a storage unit that stores a change history indicating changes made by the user to information included in the document collection;
an estimation unit that estimates a change target of a change made by the user to the document set based on the change history;
an extraction unit that extracts text corresponding to the change target from the document set;
an output unit that outputs change candidate information indicating text extracted from the document set;
A document processing device comprising:
(Appendix 9)
The estimating unit estimates the change target before a change instruction from the user is input to the text extracted from the document set.
The document processing device according to appendix 8, characterized in that:
(Appendix 10)
The change history includes a change example indicating a change operation performed by the user,
The change target specifies the text before change indicated by the change example, includes some words or phrases in the text before change, or synonyms of the some words, and is different from the text before change. information that identifies different texts,
The document processing device according to appendix 8 or 9, characterized in that:
(Appendix 11)
The information included in the document set is additional information of the text included in the document set,
The document processing device includes:
a classification unit that generates a plurality of clusters by clustering the text extracted from the document set based on texts that exist before and after the text extracted from the document set;
When the user changes the additional information of the text belonging to a specific cluster among the plurality of clusters, the change to the additional information of the text belonging to the specific cluster is changed to the additional information of other texts belonging to the specific cluster. A change part to be reflected,
The document processing device according to appendix 10, further comprising:
(Appendix 12)
The change history includes a plurality of change examples indicating change operations performed by the user,
The estimating unit estimates the change target for each of the plurality of change types classified into each of the plurality of change cases,
The extraction unit extracts text for the change target for each type,
The classification unit performs clustering on text corresponding to a specific type of change target selected based on the appearance frequency of text corresponding to the change target for each type in the document set.
The document processing device according to appendix 11, characterized in that:
(Appendix 13)
The document set includes a plurality of documents,
Each of the plurality of documents includes a plurality of partial documents,
The estimating unit calculates the number of texts corresponding to the change target for each type based on the appearance frequency of texts corresponding to the change target for each type and the number of texts corresponding to the change target for each type included in each of the plurality of partial documents. calculating an evaluation value of each partial document, and selecting a specific partial document from the plurality of partial documents based on the evaluation value of each of the plurality of partial documents;
The change candidate information includes information that highlights the specific partial document.
The document processing device according to appendix 12, characterized in that:
(Appendix 14)
The document set is learning data for machine learning that generates an analysis model, and the analysis model analyzes a document to be analyzed to generate additional information of text included in the document to be analyzed.
14. The document processing device according to any one of Supplementary Notes 11 to 13.
(Appendix 15)
Estimating a change target of a change made by the user to the document set based on a change history indicating that the user has changed information included in the document set;
extracting text corresponding to the change target from the document set;
outputting change candidate information indicating text extracted from the document set;
A document processing method characterized in that processing is performed by a computer.
(Appendix 16)
The process of estimating the change target is executed before a change instruction from the user is input to the text extracted from the document set.
The document processing method according to appendix 15, characterized in that:
(Appendix 17)
The change history includes a change example indicating a change operation performed by the user,
The change target specifies the text before change indicated by the change example, includes some words or phrases in the text before change, or synonyms of the some words, and is different from the text before change. information that identifies different texts,
The document processing method according to appendix 15 or 16, characterized in that:
(Appendix 18)
The information included in the document set is additional information of the text included in the document set,
The computer further includes:
Generating a plurality of clusters by clustering the text extracted from the document set based on texts existing before and after the text extracted from the document set,
When the user changes the additional information of the text belonging to a specific cluster among the plurality of clusters, the change to the additional information of the text belonging to the specific cluster is changed to the additional information of other texts belonging to the specific cluster. To reflect,
The document processing method according to appendix 17, characterized in that:
(Appendix 19)
The change history includes a plurality of change examples indicating change operations performed by the user,
The process of estimating the change target is performed for each of the plurality of change types in which each of the plurality of change cases is classified,
The process of extracting the text includes a process of extracting text for the change target for each type,
The clustering process is performed on text corresponding to a specific type of change target selected based on the appearance frequency of text corresponding to the change target for each type in the document set.
The document processing method according to appendix 18, characterized in that:
(Additional note 20)
The document set includes a plurality of documents,
Each of the plurality of documents includes a plurality of partial documents,
The computer includes:
of each of the plurality of partial documents based on the appearance frequency of the text corresponding to the change target for each type and the number of texts corresponding to the change target of each type included in each of the plurality of partial documents. Calculate the evaluation value,
Selecting a specific partial document from the plurality of partial documents based on the evaluation value of each of the plurality of partial documents,
The change candidate information includes information that highlights the specific partial document.
The document processing method according to appendix 19, characterized in that:
(Additional note 21)
The document set is learning data for machine learning that generates an analysis model, and the analysis model analyzes a document to be analyzed to generate additional information of text included in the document to be analyzed.
21. The document processing method according to any one of appendices 18 to 20.

201, 401 Document processing device 211, 411 Storage unit 212, 413 Estimation unit 213, 414 Extraction unit 214, 418 Output unit 221, 424 Change history 412 Analysis unit 415 Classification unit 416 Modification unit 417 Generation unit 421 Analysis model 422 Document before analysis Set 423 Document set after analysis 425 Classification results 701 Paragraph 1101 CPU
1102 Memory 1103 Input device 1104 Output device 1105 Auxiliary storage device 1106 Media drive device 1107 Network connection device 1108 Bus 1109 Portable recording medium

Claims (8)

Based on the change history, which is a change history indicating that the user has changed additional information of text included in the document set , and includes a change example indicating a change operation performed by the user, the user Estimate the target of changes made by
extracting text corresponding to the change target from the document set;
Generating a plurality of clusters by clustering the text extracted from the document set based on texts existing before and after the text extracted from the document set,
When the user changes the additional information of the text belonging to a specific cluster among the plurality of clusters, the change to the additional information of the text belonging to the specific cluster is changed to the additional information of other texts belonging to the specific cluster. reflect,
outputting change candidate information indicating text extracted from the document set;
A document processing program that allows a computer to perform processing. The process of estimating the change target is executed before a change instruction from the user is input to the text extracted from the document set.
The document processing program according to claim 1, characterized in that: The change target specifies the text before change indicated by the change example, and includes some words or phrases in the text before change or synonyms of the some words, and is different from the text before change. is information that identifies different texts,
The document processing program according to claim 1 or 2, characterized in that: The change history includes a plurality of change examples indicating change operations performed by the user,
The process of estimating the change target is performed for each of the plurality of change types in which each of the plurality of change cases is classified,
The process of extracting the text includes a process of extracting text for the change target for each type,
The clustering process is performed on text corresponding to a specific type of change target selected based on the appearance frequency of text corresponding to the change target for each type in the document set.
The document processing program according to any one of claims 1 to 3, characterized in that: The document set includes a plurality of documents,
Each of the plurality of documents includes a plurality of partial documents,
The computer includes:
The total appearance frequency of the text corresponding to the change target for each type, which is included in each of the plurality of partial documents, is calculated as an evaluation value for each of the plurality of partial documents. ,
Selecting a specific partial document from the plurality of partial documents based on the evaluation value of each of the plurality of partial documents,
The change candidate information includes information that highlights the specific partial document.
5. The document processing program according to claim 4 . The document set is learning data for machine learning that generates an analysis model, and the analysis model analyzes a document to be analyzed to generate additional information of text included in the document to be analyzed.
The document processing program according to any one of claims 1 to 5 , characterized in that: a storage unit that stores a change history indicating that a user has changed additional information of a text included in a document collection , including a change example indicating a change operation performed by the user ;
an estimation unit that estimates a change target of a change made by the user to the document set based on the change history;
an extraction unit that extracts text corresponding to the change target from the document set;
a classification unit that generates a plurality of clusters by clustering the text extracted from the document set based on texts existing before and after the text extracted from the document set;
When the user changes the additional information of the text belonging to a specific cluster among the plurality of clusters, the change to the additional information of the text belonging to the specific cluster is changed to the additional information of other texts belonging to the specific cluster. A change part to be reflected,
an output unit that outputs change candidate information indicating text extracted from the document set;
A document processing device comprising: Based on the change history, which is a change history indicating that the user has changed additional information of text included in the document set , and includes a change example indicating a change operation performed by the user, the user Estimate the target of changes made by
extracting text corresponding to the change target from the document set;
Generating a plurality of clusters by clustering the text extracted from the document set based on texts existing before and after the text extracted from the document set,
When the user changes the additional information of the text belonging to a specific cluster among the plurality of clusters, the change to the additional information of the text belonging to the specific cluster is changed to the additional information of other texts belonging to the specific cluster. reflect,
outputting change candidate information indicating text extracted from the document set;
A document processing method characterized in that processing is performed by a computer.

Images