JPH0619962A - Text dividing device - Google Patents

Abstract

PURPOSE:To accurately divide a text and to efficiently execute language processing by detecting the divided position of the text in terms of a dividing segmenting character and a format, segmenting the text on the position and outputting the segmented text. CONSTITUTION:An original sentence to be translated is inputted S1 and one sentence segmenting processing is executed S2. The one sentence segmenting processing extracs format information such as the layout/character sorts of a text in each sentence by using segmenting characters included in the text and their format information and stores extracted layout information also correspondingly to respective sentences included in the text. The text is successively translated S3 by setting up each segmented sentence as an input unit and the formating of translated sentences is executed S4 by applying stored format information and outputted S5. Since the text is divided by using format information other than normal segmenting characters, the segmentation of a sentence which can not be expressed only by segmenting characters can be correctly segmented and the text can be segmented in each sentence.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a text segmentation device used in language processing such as machine translation, text summarization, and keyword extraction. The present invention relates to a text dividing device for cutting out and outputting to subsequent processing.

[0002]

2. Description of the Related Art In language processing such as machine translation, it is general to perform a predetermined processing after inputting original texts collectively by an OCR (optical character reader). In this case, processing such as machine translation, text summarization, and keyword extraction is performed in units of one text. Therefore, it is necessary to divide the original texts that have been input in batches into individual sentences.

Conventionally, the processing for cutting out one sentence is performed by recognizing a delimiter character such as a period (.), A colon (:), or a semicolon (;) which represents a break of a sentence, and a text is recognized at the delimiter portion. Was done by dividing.

[0004]

In such a conventional text dividing device, if the delimiter does not exist, the text cannot be divided at that portion. Therefore, the title part and the body part of the text are not divided and output as a batch, or many items listed over several lines in the list are output as one sentence as a whole. Processing was sometimes performed. If such an erroneous one-sentence cut-out process is performed, an operation of correcting the erroneous part is required in order to correctly perform the subsequent process. Therefore, if the conventional text segmentation device is used, there is a problem in that the efficiency of the entire language processing becomes poor.

Therefore, an object of the present invention is to provide a text segmentation device that can perform text segmentation with higher accuracy than the conventional text segmentation device, and as a result can efficiently perform subsequent language processing. Is.

[0006]

A text division device according to the present invention is a division position detection means for detecting a division position of a text based on a delimiter included in the text and a text format, and a division position detection means. Output means for dividing and outputting the text at the division position detected by.

[0007]

In the text dividing device according to the present invention, not only the delimiter character as the division position but also the text format is used for the detection processing, and the text is divided at the division position thus detected.

[0008]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail below with reference to the drawings. In the present specification, the “format” of text indicates all information indicating the arrangement of characters, such as the layout indicating the arrangement of sentences and the character type in which each character forming the sentence is used. To do.

FIG. 1 is a schematic diagram showing a flowchart of a process performed by a machine translation device using a text segmentation device according to an embodiment of the present invention and a part of hardware. First, in step S1, an original sentence to be translated is input by an OCR (not shown) or the like.

Then, in step S2, a one-sentence cutting process is performed using the text segmentation device according to the present invention. At this time, the one-sentence cut-out process is performed for each sentence by taking out format information such as text layout / character type and using not only the delimiter characters included in the text but also the format information. At this time, the extracted layout information is also stored in comparison with each sentence included in the sentence as shown in FIG.

In step S3, translation processing is sequentially performed with the one sentence cut out in step S2 as an input unit.

Subsequently, in step S4, the translated sentence obtained in step S3 is formatted by applying the format information such as the layout / character type temporarily stored in the process of step S2.

Then, in step S5, the resulting sentence formatted in step S4 is output and the process ends.

FIG. 2 is a flow chart showing a more detailed procedure of the one-sentence cutout process performed in step S2 of FIG. FIG. 3 is a schematic diagram showing an example of the input text, and FIG. 4 shows the correspondence between the layout / character type information obtained when the one-sentence cutout process is performed on FIG. 3 and the cutout sentence. It is a schematic diagram of the buffer shown.

Referring to FIG. 2, one sentence cutout is performed as follows. First, in step S11, a process of setting a line pointer that points to a line to be processed in the text to the first line of the text is performed.

In step S12, a process for determining whether the line pointed to by the pointer is a blank line is performed. A blank line is a line that does not contain any characters. If the line to be processed is a blank line, the process proceeds to step S15, and if it is not a blank line, the process proceeds to step S13.

In step S13, it is determined whether the beginning of the line to be processed is a combination of numbers and symbols. If the beginning of a line is a combination of numbers and symbols, this line is likely to be a title. Therefore, the process proceeds to step S14. If the beginning of the line is not a combination of numbers and symbols, the process proceeds to step S18.

In step S14, the line currently being processed is set to 1
A process of storing in the buffer as a sentence (one unit) is performed. The post-processing of step S14 proceeds to step S15.

When the process proceeds from step S13 to step S18, it is determined in step S18 whether or not the line head of the line to be processed is a space. If the line head is a space, the process proceeds to step S19, and if not, the process proceeds to step S21.

In step S19, a determination is made as to whether the first character after the space is a symbol. If it is a symbol, the process proceeds to step S20; otherwise, the process proceeds to step S21.

In step S20, a determination is made as to whether the line next to the current line or the line preceding the current line has the same shape as the current line. Whether or not they have the same shape means that the beginning of a line is a space and the first character is a symbol, or that such a condition is not satisfied. If the next line or the previous line has the same shape as the current line, the process proceeds to step S14, and if not, the process proceeds to step S21. When the process proceeds to step S14, the process of storing the current line as one sentence in the buffer is performed as in the case where the beginning of the line is a combination of a number and a symbol, and the process further proceeds to step S15 and thereafter.

On the other hand, if the determination is NO in any of the three determinations of step S18, step S19, and step S20, the process proceeds to step S21. In step S21, the normal one-sentence cutting process is performed from the current line to the position before the next blank line. That is, the text is divided by delimiters such as periods and colons included in the text, and each is processed as one sentence. The process proceeds to step S22.

In step S22, processing for advancing the row pointer to a portion where one sentence is not yet cut out is performed. The post-processing of step S22 proceeds to step S17.

On the other hand, in step S15, the layout /
A process of storing text format information such as a character type in a buffer is performed. Here, the format information includes the number of spaces at the beginning of a sentence, the type of type used (for example, bold type,
Italics), including information such as whether there is a line break at the end of the sentence. Details of this will be described later with reference to FIGS.

The post-processing of step S15 is step S16.
Then, processing for advancing the row pointer by one is performed. As a result, the line to be processed is moved forward by one. The post-processing of step S16 proceeds to step S17.

In step S17, it is determined whether or not there is a line to be processed at the position indicated by the line pointer newly set in steps S16 and S22. If it exists, the process returns to step S12 and the processes of step S12 and thereafter are repeatedly executed. If the line does not exist, the process ends.

By performing the one-sentence cutout process as shown in FIG. 2, the following results can be obtained.
FIG. 3 is an example of input text. In the case of the text shown in FIG. 3, the title and the body are separated by a blank line. Also, the text further includes a portion that represents the text of the ground and a number of example portions introduced by the text of the ground, the two parts being separated by blank lines.

In the case of the text shown in FIG. 3, if the text is not divided at a part other than the normal delimiter,
For example, the title and the ground sentence may be connected to each other, or a plurality of exemplified sentences may be connected to each other, so that the correct one-sent-line cutting process is not performed.

On the other hand, when this sentence is divided using the text dividing device of the present invention, the result is as shown in FIG. Referring to FIG. 4, the title of sentence number 1 and the ground sentence of sentence number 3 are separated by the blank line of sentence number 2. In addition, sentence number 3 and sentence number 4 are separated by a normal delimiter (period),
The sentence number 4 and the sentence number 6 are separated by a normal delimiter (colon) and a blank line of the sentence number 5. In addition, since the line breaks are present at the end of the sentences in the example portions of sentence numbers 6 and 7, the layout information
It is divided into two sentences. Other example sentences are separated as well. Further, the sentence number 1 and the sentence number 3 can be divided by using the different character types used.

As described above, in the text dividing device according to the present invention, the text is divided using the format information other than the normal delimiter. Therefore, it is possible to correctly detect sentence delimiters that cannot be expressed only by delimiters and to cut out the text one sentence at a time. Since a sentence that cannot be divided only by the delimiter can be correctly divided, it is not necessary to correct the result of the one-sentence cutout process prior to the subsequent language processing, and the processing efficiency can be improved.

[0031]

As described above, in the text segmentation device according to the present invention, the segmentation position of the text that cannot be represented only by the normal delimiter is detected using the text format information, and the segmentation thus detected is detected. You can split text by position. Therefore, the accuracy of the text division is further improved as compared with the case where the text division is performed using only the delimiter, and the necessity of correcting the text division processing result prior to the subsequent processing is reduced.

As a result, it is possible to provide a text division device which can further improve the accuracy of text division and can improve the efficiency of the subsequent language processing.

[Brief description of drawings]

FIG. 1 is a schematic diagram showing a flowchart of a process performed by a machine translation device using a text segmentation device according to an embodiment of the present invention and a part of the device.

FIG. 2 is a flowchart of a one-sentence cutout process.

FIG. 3 is a schematic diagram showing an example of input text.

FIG. 4 is a schematic diagram of a buffer showing a processing result when the text shown in FIG. 3 is divided by the text division device according to the present invention.

Claims (1)

[Claims] 1. A division position detection unit for detecting a division position of the text based on a delimiter included in the text and a format of the text, and the text at the division position detected by the division position detection unit. A text segmentation device including an output means for dividing and outputting.