JP7268220B2 - Text processing device and text processing method - Google Patents

Description

The present invention relates to a text processing apparatus and a text processing method, and more particularly to text processing suitable for quickly classifying a large amount of text according to a purpose and efficiently obtaining a summary.

With the explosive spread of the Internet, the most important issue for each company is how to use Web systems to deal with customers. Under such circumstances, companies not only transmit information using web systems, but also work in conjunction with call centers and SNS (Social Networking Services) to collect information from customer requests and comments on products and services. Identifying points for improvement and promptly responding to such requests are important points in corporate activities.

In order for companies to make effective use of documents collected from the Internet, technology for classifying and summarizing documents by topic using an information processing device is required. For example, Patent Literature 1 discloses a method of selecting a question category as a candidate based on extracted personal attributes and topics, and classifying user utterances by answering questions belonging to the question category. Further, Patent Literature 2 discloses a technique of outputting a plurality of abstract sentences from an input text, evaluating the abstract sentences based on a plurality of evaluation scales, and extracting the most appropriate abstract sentence.

JP 2016-206894 A JP 2016-161967 A

For example, in a large diversified company such as a railroad operator, such documents collected from the Internet amount to hundreds of thousands of documents per year. It is desirable to appropriately and timely process such document information and utilize it in corporate activities.

In particular, it may be necessary to extract sentences related to keywords that are trending on the Internet, process them in a timely manner, and provide opinions and corresponding information. For example, a railway operator needs to obtain information from sentences related to keywords related to accidents on specific routes and take appropriate measures.

The technique described in Patent Literature 1 does not take into consideration such a technique of specifying and classifying specific keywords, and cannot classify sentences in response to news and trends. Moreover, Patent Document 2 does not disclose a technique for creating a summary with respect to a specific viewpoint such as a user's request. For example, no consideration is given to improving customer satisfaction by extracting a summary sentence based on the viewpoint of the user's request, identifying points for improvement according to the user's request, and dealing with them.

An object of the present invention is to classify a huge amount of target sentences by keywords corresponding to news and trends, to summarize the sentences according to the sentence structure, and to make the sentences more appropriate from a specific point of view than the summarized sentences. An object of the present invention is to provide a sentence processing device and a sentence processing method capable of acquiring information.

The text processing device of the present invention is preferably a text processing device that summarizes a text and outputs a summary of the text, and is summary performance data that is a pair of the text to which the classification tag is assigned and the summary. , perform structural analysis on simple sentences of the input text, extract meanings related to structural analysis, extract meaningful simple sentences related to structural analysis for each classification tag of summary performance data, When the degree of similarity between the simple sentence of the input sentence and the sentence of the summarized performance data is equal to or higher than a specified threshold, the summary sentence of the summary performance data is used as the summary sentence of the input sentence.
The present invention is also configured as a text processing method for the text processing device.

According to the present invention, it is possible to classify a huge amount of target sentences by keywords corresponding to news and trends, summarize the sentences according to the sentence structure, and appropriately classify the sentences from the summarized sentences from a specific viewpoint. information can be obtained.

It is a figure which shows the functional structure of a sentence processing apparatus. It is a figure which shows the hardware software configuration of a sentence processing apparatus. It is a figure which shows the outline|summary of the characteristic of the text processing of this invention. It is a figure which shows an example of classification performance data. It is a figure which shows an example of summary performance data. It is a figure which shows an example of classified and summarized data. It is a figure which shows an example of sentence-analysis result data. It is a figure which shows an example of a report preparation start screen. It is a figure which shows an example of a sentence classification / summarization result screen. FIG. 11 is a diagram showing an example of a text classification/summary details list screen; It is a figure which shows an example of a classification change screen. It is a figure which shows an example of a summary change screen. It is a figure which shows an example of a report display screen. It is a flow chart which shows sentence information accumulation processing. It is a flow chart which shows sentence analysis processing. It is a flow chart which shows sentence classification processing. It is a flow chart which shows text summarization processing. 4 is a flowchart showing classification learning processing; 4 is a flowchart showing summary learning processing; 4 is a flow chart showing report creation processing. 4 is a flowchart showing semi-automatic classification processing;

An embodiment according to the present invention will be described below with reference to FIGS. 1 to 21. FIG.
First, the configuration of a text processing apparatus according to an embodiment will be described with reference to FIGS. 1 and 2. FIG.

As shown in FIG. 1, the text processing apparatus 10 according to the present embodiment includes a text analysis unit 110, a text classification unit 120, a classification change unit 130, a text summary unit 140, a summary change unit 150, and a report generator. It is composed of a part 160 .

The text analysis unit 110 is a part that analyzes the structure of a text received from the outside and outputs text analysis result data (details will be described later). The sentence classification unit 120 is a part that classifies (categorizes) sentences based on words appearing in the sentences and assigns classification tags. The classification change section 130 is a section that changes the classification classified by the text classification section 120 and reattaches the classification tag. The sentence summarization section 140 is a part that summarizes the sentence and creates a summary sentence. The summary changing section 150 is a section that changes the summary created by the text summarizing section 140 . The report creation section 160 is a section that outputs a result to the user based on the information that classifies the text and the information that summarizes it.

The text analysis unit 110 is composed of a word analysis unit 111 , a structure analysis unit 112 and a text analysis result processing unit 113 . The word analysis unit 111 is a part that analyzes sentences at the word level and divides the sentences into words. The structural analysis unit 112 is a part that analyzes sentences at the structural level and extracts their meanings. The sentence analysis result processing unit 113 is a part that processes the results of the word analysis unit 111 and the structure analysis unit 112 .

The sentence classification section 120 is composed of a classification processing section 121 , a semi-automatic classification section 122 and a classification learning section 123 . The classification processing unit 121 is a part that performs calculations and judgments regarding the classification of sentences. The semi-automatic classification unit 122 is a part that performs semi-automatic classification by inputting a keyword for classification and classifying. The classification learning unit 123 is a part that stores information that associates sentences and classification tags as learning data.

The text summarizing section 140 includes a summary processing section 141 and a summary learning section 142 . The summary processor 141 is a part that creates a summary from the text. The summary learning unit 142 is a part that stores information that associates sentences and abstracts.

As for the hardware configuration of the text processing device, for example, it is realized by a general information processing device such as a personal computer shown in FIG.

The text processing device 10 has a form in which a CPU (Central Processing Unit) 202, a main memory device 204, a network I/F 206, a display I/F 208, an input/output I/F 210, and an auxiliary memory I/F 212 are connected by a bus. ing. The text processing device 10 is also connected to the text server 400 via the network 60 . The text server 400 is a server that stores text and distributes the text to the text processing device 10 .

The CPU 202 controls each part of the text processing device 10, loads necessary programs into the main memory 204, and executes them. The main memory device 204 is normally composed of a volatile memory such as a RAM, and stores programs executed by the CPU 202 and data to be referred to. A network I/F 206 is an interface for connecting with the network 60 . A display I/F 208 is an interface for connecting a display device 220 such as an LCD (Liquid Crystal Display). The input/output I/F 210 is an interface for connecting input/output devices. In the example of FIG. 2, a keyboard 230 and a mouse 232 as a pointing device are connected as input/output devices.

Auxiliary storage I/F 212 is an interface for connecting auxiliary storage devices such as HDD (Hard Disk Drive) 250 and SSD (Solid State Drive).

The HDD 250 has a large storage capacity and stores programs for executing the present embodiment. A text analysis program 310, a text classification program 320, a classification change program 330, a text summary program 340, a summary change program 350, and a report creation program 360 are installed in the text processing device 10. FIG.

The text analysis program 310, the text classification program 320, the classification change program 330, the text summary program 340, the summary change program 350, and the report creation program 360 are the text analysis unit 110, the text classification unit 120, the classification change unit 130, and the text summary, respectively. It is a program for realizing each function of the section 140 , the summary changing section 150 and the report creating section 160 .

The HDD 250 also stores classified performance data 510 , summary performance data 520 , classified/summarized data 530 , and sentence analysis result data 540 . The significance and structure of each data will be explained in detail later.

Next, with reference to FIG. 3, an outline of features of text processing according to the present invention will be described.
Conventionally, when classifying sentences, the likelihood/certainty that applies to the classification is calculated based on the past classified performance data from the sentences, and the classification is performed.

In the present invention, the user inputs a designated classification together with a sentence, and semi-automatic classification is performed (phase I in FIG. 3). Here, the designated classification means a classification tag name and a keyword for classification newly input by the user. Inputting a keyword for classification in such a manner and causing the information processing apparatus to classify is called semi-automatic classification. This makes it possible to semi-automatically extract the classification result that the user wants to extract.

In addition, conventionally, summary results are generated based on parameters from sentences. Also, parameters are updated based on the evaluation results. Here, the parameter is a quantity expressing the feature of a sentence or a set of sentences as a weight, as described in Patent Document 2, for example.

In the present invention, when summarizing a sentence, the structure and meaning of the sentence are analyzed, and the classification information in phase I in FIG. 3 is added to the information to create a summary sentence (phase II in FIG. 3). As a result, it becomes possible to create a summary that matches the classification.

Next, the data structure used in the text processing apparatus of this embodiment will be described with reference to FIGS. 4 to 7. FIG.
The classification performance data 510 is data used as a standard for classifying sentences, and has a data format consisting of columns of classification tags 511 and sentences 512, as shown in FIG. This indicates that the text pattern stored in the sentence 512 is classified by the tag indicated by the classification tag 511 . This classification performance data 510 is generated by being learned by a classification learning process (described later).

The summary performance data 520 is standard data for creating a summary of sentences, and as shown in FIG. have This indicates that a summary sentence 524 is generated for the text pattern of the sentence 522 classified by the classification tag 521 and having the sentence structure described in the sentence structure 523 . This classification performance data 510 is generated by learning through a summary learning process (described later).

The classified/summarized data 530 is data saved as a result of classifying sentences and creating a summary by the sentence processing apparatus of the present embodiment. As shown in FIG. 532, classification tag 1 (533), classification tag 2 (534), and summary text 535.

The text stored in the text 532 is given the tags described in the classification tag 1 (533) and the classification tag 2 (534) as tags for classification on the registration date described in the registration date 531, This indicates that the abstract written in the abstract 535 has been generated. In the description of this embodiment, the number of classification tags is two, but the number of classification tags may be one, or three or more classification tags may be provided.

The sentence analysis result data 540 is data that holds the analysis results of sentences included in sentences to be processed. As shown in FIG. It has a data format consisting of 545 columns.

A sentence described in a simple sentence 541 is described in each word 542, decomposed into parts of speech described in a part of speech 543, and has a sentence structure described in a sentence structure 544. It indicates that it has the meaning described in the meaning 545 from the structure. For example, in the example of FIG. 7, it is shown that the simple sentence "I want you to broadcast promptly" has the meaning of a (customer's) request from the sentence structure of "I want you to do".

Next, the user interface of the text processing apparatus of this embodiment will be described with reference to FIGS. 8 to 12. FIG.
The report creation start screen 700 is a screen that is first displayed to generate and display a report of text classification and summarization processing by the text processing device 10, and the classification tag name ( Hereafter, this is a screen for inputting a classification keyword) and a classification keyword. As shown in FIG. 8, the report creation start screen 700 has display elements such as a sentence target range column 701, a specified classification input column 702, and a report creation start button 705. FIG.

The sentence target range column 701 is a column for inputting the period of the sentence to be handled. For example, the registration date and time of the text can be the time when the text is created or the time when the text is stored in the text server 400 . A designated classification input field 702 is a field for inputting information for a designated classification, and consists of a classification tag name input field 703 and a classification keyword input field 704 . A classification tag name input field 703 is a field for inputting a classification tag name used for classification. A classification keyword input field 704 is a field for inputting a keyword used for classification. In the example shown in FIG. 8, "equipment failure" is entered as the classification tag name, and "signal" and "failure" are entered as the classification keywords. A report creation start button 705 is a button that is selected when a specified classification is entered and a report is created and displayed.

The text classification/summarization result screen 710 is a screen for displaying the result of the text classification/summarization processing, and as shown in FIG. It has repeated display elements.

The classification tag display field 711 is a field for displaying the input tag name of the designated classification. The number display column 712 is a column for displaying the number of sentences belonging to the specified classification. A detailed list display button 713 is a button that is selected to display a text classification/summary detailed list screen 720 (described later) belonging to the specified classification.

The sentence classification/summary details list screen 720 is a screen for displaying detailed information of the sentences subjected to the sentence classification/summary processing and for designating changes. As shown in FIG. It consists of display elements such as a summary sentence display field 722 , a classification change button 723 and a summary change button 724 .

The original sentence display column 721 is a column for displaying the original sentence. The summary display field 722 is a field for displaying the summary of the original text. A change classification button 723 and a change summary button 724 are buttons selected to display a change classification screen 730 (described later) and a change summary screen 740 (described later), respectively.

The classification change screen 730 is a screen for changing the specified classification of sentences, and as shown in FIG. have elements.

The original sentence display column 731 is a column for displaying the original sentence of the sentence. The summary display field 732 is a field for displaying the summary of the text. A classification designation combo box 733 is an input display element for changing the classification tag of the sentence. A change reflection button 734 is a button that is selected to add the selected classification tag to the specified sentence.

In this embodiment, up to three types of classification tags can be specified. can be

The summary change screen 740 is a screen for editing and changing the text summary, and as shown in FIG. display elements.

The original sentence display column 741 is a column for displaying the original sentence of the sentence. The classification display field 742 is a field for displaying the classification tag of the text. A summary text editing area 743 is a column for displaying a summary of the text. The abstract editing area 743 displays the contents of the current abstract, and the user can change the abstract using an input device such as a keyboard. A change reflection button 744 is a display element selected to reflect the summary edited in the summary editing area 743 .

The report display screen 750 is a screen for displaying a report created after inputting the designated classification, and as shown in FIG. 754 , has a display element for the close button 755 .

The text target range display column 751 is a column for displaying the period of the target text input from the report creation start screen 700 . The designated classification display field 752 is a field for displaying the classification tag name and keyword of the designated classification input from the report creation start screen 700 . The abstract display column 753 is a column for displaying the retrieved abstract. A print button 754 is a button that is selected when printing the information displayed on the report display screen 750 . A close button 755 is a button selected when closing the report display screen 750 .

Next, processing performed by the text processing apparatus will be described with reference to FIGS. 14 to 21. FIG.
The sentence information accumulation process shown in the flowchart of FIG. 14 is a process of accumulating sentences and the results of classification and summarization of the sentences in the sentence processing device.

In the text information accumulation process, text analysis processing is first performed (S101). This sentence analysis processing will be described in detail later.
Next, sentence classification processing is performed (S102). This sentence classification processing will be described in detail later.
Next, text summary processing is performed (S103). This text summarization process will be described in detail later.
Next, the results of text classification and text summarization in S102 and S103 are accumulated in the classified/summarized data 530 shown in FIG. 6 (S104).

Next, sentence analysis processing will be described with reference to FIG.
The sentence analysis process shown in FIG. 15 is a process of analyzing a sentence at the word level/structure level and storing the result, which corresponds to S101 in FIG.
In the text analysis process, the text processing device 10 receives text from the text server 400 via the network 60 (S201).

Next, the word analysis unit 111 of the sentence analysis unit 110 shown in FIG. 1 analyzes the received sentence and stores it in the sentence analysis result data 540 shown in FIG. 7 (S202).
Next, the structure analysis unit 112 analyzes the related structure of the divided words, extracts the meaning, and stores it in the sentence analysis result data 540 (S203).

The sentence classification process shown in FIG. 16 is a process of adding classification tags for classification to sentences, and corresponds to the process of S102 in FIG.
In the sentence classification process, first, the information of appearing words and sentence structure stored in the sentence analysis result data 540 shown in FIG. 7 is input to the sentence classification unit 120 (S301).
Next, the processing of S303 to S305 is performed for each classification tag (not shown) held by the text processing device (S302 to S306).

The classification processing unit 121 uses the appearing words to calculate the degree of similarity between the sentence and each classification tag (S303). The degree of similarity between a sentence and each classification tag can be calculated, for example, by summing the number of times the keyword of the classification tag appears in the sentence. For example, a group of words {equipment failure, device failure, device output} is associated with a classification tag "equipment failure" as a keyword. The similarity is obtained by dividing by the number of words.

When the degree of similarity is equal to or greater than the predetermined threshold (S304: YES), the process proceeds to S305, and when the degree of similarity is less than the predetermined threshold (S304: NO), the process proceeds to evaluation with the next classification tag (S302). .

When the degree of similarity is equal to or greater than a predetermined threshold (S304: YES), the sentence is given a target classification tag (S305).

The text summarization process shown in FIG. 17 is a process of creating a summary from the target text, and corresponds to the process of S103 in FIG.
In the text summarization process, first, the words and sentence structure information stored in the text analysis result data 540 shown in FIG. Input (S401).

Next, the summary processing unit 141 acquires the classification tags added to the target sentence (S402), and repeats the processes of S404 to S410 for all classification tags (S403 to S411).

First, the summary processing unit 141 extracts meaningful simple sentences from the sentence analysis result data 540 (S404). A meaningful simple sentence is a simple sentence stored in the simple sentence 541 corresponding to the record in which the column of the meaning 545 of the sentence analysis result data 540 is not blank. For example, in the example shown in FIG. 7, the simple sentence 541 of the record stored in the meaning 545 "request" stores the simple sentence "Please broadcast immediately."

Next, the summary processing unit 141 acquires the summary performance data 520 having the target classification tag (S405).
Next, S407 to S409 are repeated for each summary performance data 520 having the acquired target classification tag (S406 to S410).

First, the summary processing unit 141 calculates the degree of similarity between the simple sentence of the target sentence and the sentence of the summary performance data 520 (S407). The degree of similarity between the simple sentence of the target sentence and the sentence of the summary performance data 520 is, for example, the number of times the word of the simple sentence of the target sentence appears in the word of the sentence of the summary performance data 520, or the total number of words of the sentence of the summary performance data 520. A value obtained by dividing by is evaluated as a degree of similarity.

When the degree of similarity is equal to or greater than the threshold (S408: YES), go to S409, and when the degree of similarity is less than the threshold (S408: NO), go to the next summary performance data 520. FIG.

When the degree of similarity is equal to or greater than the threshold (S408: YES), the summary sentence 524 of the summary performance data 520 is calculated as the summary sentence of the target sentence (S409).
Here, a plurality of abstract sentences may be extracted from the target sentence, or the one when the similarity in S407 is high may be used as the abstract sentence of the target sentence.

The classification learning process shown in FIG. 18 is a process of adding new information to the classification record data 510, which is learning data for associating sentences with classification tags.
In the classification learning process, first, a new classification tag for the target sentence is designated from the classification change screen 730 shown in FIG. 11 (S501).
Next, the information of the target sentence and the new classification tag is input to the classification changing section 130 (S502).
Next, information on the target sentence and the new classification tag is stored in the classification result data 510 (S503).

The summary learning process shown in FIG. 19 is a process of adding new information to the summary performance data 520, which is learning data for association between sentences and abstracts.
In the summary learning process, first, a summary sentence for the target sentence is entered from the summary change screen 740 shown in FIG. 12 (S601).
Next, the information of the target sentence and the new abstract is input to the summary changing unit 150 (S602).
Next, the information of the target sentence and the new abstract is stored in the summary performance data 520 (S603).

The report creation process shown in FIG. 20 is a process of creating and displaying a report after inputting a specified classification.
In the report creation process, first, a target sentence range (period), a classification tag name, and a classification keyword are described on the report creation start screen 700 shown in FIG. 8 (S701).
Next, the report creation unit 160 acquires data having a registration date within the input target sentence range (period) from the classified/summarized data 530 (S702).

Next, the report creation unit 160 aggregates the classified/summarized data 530 using the information in the classification column and calculates the number of cases (S703).
Next, the process of S705 is repeated for each designated classification tag that is input (S704 to S706).

In the loop, semi-automatic classification/summarization processing is performed (S705). The details of the semi-automatic classification/summarization process will be described later.
Finally, the report creation unit 160 displays the result on the report display screen 750 shown in FIG. 13 (S707).

The semi-automatic classification/summarization process shown in FIG. 21 performs classification and summary processing based on the designated classification after inputting the designated classification, and corresponds to S705 in FIG.
In the semi-automatic classification/summarization process, first, the report creation unit 160 searches for the classified/summarized data 530 having the keyword described in the specified classification information (S801).
Next, the classification learning unit 123 accumulates the searched classified/summarized data 530 in the classification performance data 510 with the classification tag name of the designated classification tag name (S802).

Next, the classified/summarized data 530 having the registration date of the sentence target range (period) is input to the classification processing unit 121 (S803).
Then, the processing of S805 to S808 is repeated for each classified/summarized data 530 having the registration date of the text target range (period) (S804 to S809).

First, the classification processing unit 121 calculates the degree of matching between the target sentence and the designated classification tag (S805). The degree of applicability between the target sentence and the specified classification tag is an index indicating the validity when the specified classification tag is used as the classification tag of the sentence. For example, keywords associated with the tag name of the specified classification tag are defined, A value obtained by dividing the number of times the keyword appears in the target sentence by the total number of words in the target sentence can be defined.

If the degree of applicability between the target text data and the designated classification tag is equal to or greater than the prescribed threshold (S805: YES), go to S807; ), go to S804 to process the next classified/summarized data 530 .

When the degree of fit between the target text data and the designated classification tag is equal to or greater than the prescribed threshold (S805: YES), the classification processing unit 121 assigns the classification tag of the designated classification to the text of the classified/summarized data 530. (S807).
Next, the summary processor 141 acquires a summary of the classified/summarized data 530 (S808).

As described above, the text processing apparatus according to the present embodiment selects sentences of up-to-date topics by inputting a designated classification for data that has been classified and summarized, and reclassifying the data. In addition, since the summary processing is performed based on the sentence structure, it is possible to obtain a valuable summary report for the enterprise, such as reflecting the user's request.

10 sentence processing device 110 sentence analysis unit 111 word analysis unit 112 structure analysis unit 113 sentence analysis result processing unit 120 sentence classification unit 121 classification processing unit 122 semi-automatic classification unit 123 classification learning unit 130 classification Modifying unit 140 Sentence summarizing unit 141 Summarizing processing unit 142 Summarizing learning unit 150 Summarizing changing unit 160 Report creating unit

Claims (2)

A text processing device that summarizes a text and outputs a summary of the text,
Holds summary performance data, which is a pair of sentences and summary sentences given classification tags,
Structural analysis is performed on the simple sentence of the input sentence, the meaning related to the structural analysis is extracted,
For each classification tag of the summary performance data, a meaningful simple sentence related to the structural analysis is extracted, and when the similarity between the simple sentence of the input sentence and the sentence of the summary performance data is equal to or higher than a prescribed threshold value A text processing apparatus, wherein the summary of the summary result data is the summary of the input text. A text processing method for summarizing a text and outputting a summary of the text,
Holds summary performance data, which is a pair of sentences and summary sentences given classification tags,
Structural analysis is performed on the simple sentence of the input sentence, the meaning related to the structural analysis is extracted,
For each classification tag of the summary performance data, a meaningful simple sentence related to the structural analysis is extracted, and when the similarity between the simple sentence of the input sentence and the sentence of the summary performance data is equal to or higher than a prescribed threshold 7. A text processing method, wherein said summary of said summary performance data is set as said summary of said input text.

Images