KR102498839B1 - Apparatus and method for detection of sentence boundary - Google Patents

Abstract

The present invention relates to an apparatus and method for recognizing sentence boundaries for text without sentence boundary markings. The sentence boundary recognition method of the present invention for this purpose includes extracting feature information including engram information of input text; extracting candidate probabilities for tentative sentence boundary candidates based on the learning corpus including sentence boundary information; calculating candidate scores for sentence boundary tentative candidates based on the candidate probabilities; selecting sentence boundary tentative candidates whose candidate scores are greater than or equal to a preset threshold score as sentence boundary candidates; and finally classifying whether or not a sentence boundary is present using feature information including engram information of sentence boundary candidates and candidate score information.

Description

Sentence boundary recognition device and method {APPARATUS AND METHOD FOR DETECTION OF SENTENCE BOUNDARY}

The present invention relates to an apparatus and method for recognizing sentence boundaries for text without sentence boundary markings.

Sentence boundary recognition technology refers to a technology of dividing text included in a document into sentence units by taking a document or voice input text as an input. However, in the existing speech recognition system, there is no recognition of a sentence boundary with respect to voice input text. Accordingly, in the case of application systems that require language understanding, such as automatic interpretation and translation systems, conversation systems, conversation-based education systems, and human-computer interfaces that require accurate understanding of input sentences, Otherwise, there is a problem of having to endure the inconvenience of uttering in sentence units. In addition, in the above systems, there is a problem in that the service error and performance degradation of the system due to the lack of sentence boundary information have to be endured.

In addition to voice recognition systems, many users of portable computing terminals often input text without sentence boundary marks when inputting text such as text transmission or memo writing. Recognizing the sentence boundary of the input text is also a task to be solved in application systems that require language understanding in the future.

In a study on finding actual sentence boundary marks in large texts with punctuation marks such as news texts for information retrieval, specific punctuation marks such as ".", "?", and "!" A technique for classifying whether or not it is a sentence boundary based on learning based on the left and right n-gram context information of the candidate is widely used. It is possible to secure a large-capacity learning corpus, it is easy to select sentence boundary candidates, and when all sentence boundary candidates are classified as sentence boundaries, that is, the baseline classification accuracy is high, the accuracy of sentence boundary candidates in this field varies depending on the corpus. However, it is generally 97% to 99.8%.

Instead, the accuracy of sentence boundary recognition for voice input text is still low at about 40% to 60% depending on the corpus, so it is not particularly used in real life environments because it is difficult to secure a large-capacity learning corpus. In this case, a rule-based or learning-based sentence boundary recognition technology is used. The rule-based boundary recognition technology has disadvantages such as low performance and the need to build a new rule by a person whenever the field or language changes. The method has problems in that it is difficult to secure a large amount of speech transcription corpus as a learning corpus, and the final accuracy of sentence boundary recognition is low because there are many sentence boundary candidates and their accuracy is low.

Prior art related to the present invention includes Korean Patent Registration No. 1259558 (Sentence Boundary Recognition Apparatus and Method) and US Patent No. 8364485 (Method for automatically identifying sentence boundaries in noisy conversational data).

An object of the present invention is to provide a sentence boundary recognizing apparatus and method capable of automatically recognizing sentence boundaries even in text without sentence boundaries.

A sentence boundary recognition method of the present invention for solving the above problems includes extracting feature information including engram information of input text; extracting candidate probabilities for tentative sentence boundary candidates based on the learning corpus including sentence boundary information; calculating candidate scores for sentence boundary tentative candidates based on the candidate probabilities; selecting sentence boundary tentative candidates whose candidate scores are greater than or equal to a preset threshold score as sentence boundary candidates; and finally classifying whether or not a sentence boundary is present using feature information including engram information of sentence boundary candidates and candidate score information.

In addition, the step of calculating the candidate scores may include setting left engrams and right engrams for each tentative sentence boundary candidate; and calculating engram candidate scores for each sentence boundary provisional candidate based on candidate probabilities including the probability that the set left engram is on the left side of the sentence boundary provisional candidate and the right engram is on the right side of the sentence boundary provisional candidate. steps may be included.

In addition, when there is voice information for the input text, a step of extracting static sound and static sound length information included in the voice information may be further included.

Also, calculating the candidate scores may include calculating weighted candidate scores based on information about the silence and the length of the silence when there is voice information about the input text.

In addition, the step of calculating the candidate scores may be performed by summing the engram candidate score and the weighted candidate score for each provisional sentence boundary candidate.

In addition, by analyzing the sentence boundary candidates, the step of selecting the sentence boundary code includes the probability that the set left engram comes to the left of the provisional sentence boundary candidate, the probability that the right engram comes to the right of the provisional sentence boundary candidate, and the probability that the set left engram comes to the right of the provisional sentence boundary candidate This can be achieved by using feature information including at least one of a star engram candidate score and a weighted candidate score calculated based on voice information.

In addition, the step of selecting sentence boundary codes by analyzing sentence boundary candidates may be performed through a classification model including a support vector machine (SVM) and a maximum entropy model (MEM).

Also, engrams may include unigrams, bigrams, and trigrams.

According to the apparatus and method for recognizing sentence boundaries of the present invention, since it is easy to secure a learning corpus by using plain text, statistical information that can be used for selecting sentence boundary candidates in recognizing sentence boundaries for text without sentence boundary information such as voice input It has the advantage of being easy to obtain.

In addition, the present invention primarily recognizes sentence boundary candidates using the sentence boundary probability score of the engram extracted and calculated from the learning corpus and the sentence boundary probability score of the jeongeum, and all vocabulary, voice characteristics, and probability scores are characterized by , it has the advantage of high sentence boundary recognition accuracy by using a method for recognizing sentence boundaries as a learning-based classification method.

In addition, according to the apparatus and method for recognizing sentence boundaries according to an embodiment of the present invention, it is possible to improve an inconvenient user interface in which one sentence must be uttered each time in voice input, and the accuracy of sentence boundaries for natural speech is improved, so that automatic interpretation and translation can be performed. It can be used in fields that require a lot of language understanding, such as systems, conversation systems, and human-computer interfaces.

1 is a block diagram of an apparatus for recognizing sentence boundaries according to an embodiment of the present invention.
2 is a block diagram of a processing unit according to an embodiment of the present invention.
3 is a flowchart of a sentence boundary recognition method according to an embodiment of the present invention.
4 is a flowchart of a method for building a learning corpus according to an embodiment of the present invention.

The present invention will be described in detail with reference to the accompanying drawings. Here, repeated descriptions, well-known functions that may unnecessarily obscure the subject matter of the present invention, and detailed descriptions of configurations are omitted. Embodiments of the present invention are provided to more completely explain the present invention to those skilled in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clarity.

Hereinafter, an apparatus and method for recognizing sentence boundaries according to an embodiment of the present invention will be described. An apparatus and method for recognizing sentence boundaries according to an embodiment of the present invention is a portable computing terminal such as a smart phone or a voice recognition result text without a sentence boundary display through a microphone of a PC, and a user using a keyboard of a portable or other computing device. It is characterized in that sentence boundaries can be automatically recognized for text input without sentence boundary marks through the device.

To this end, the sentence boundary recognition apparatus and method according to an embodiment of the present invention utilizes a large-capacity plain text corpus as a learning corpus and calculates engram candidate scores and weighted candidate scores for each sentence boundary tentative candidate sentence boundary candidate It is possible to provide a sentence boundary recognition method and apparatus for selecting and checking whether a corresponding position matches a sentence boundary using a feature-based classification technique.

1 is a block diagram of an apparatus 100 for recognizing sentence boundaries according to an embodiment of the present invention. The apparatus 100 for recognizing sentence boundaries according to an embodiment of the present invention may include a control unit 110 and a processing unit 120.

The control unit 110 functions to transfer text input from the user 10 through the input unit 20 to the processing unit 120 and controls the processing unit 120 described below. Here, the text input through the input unit 20 may include not only text input as text but also text recognized by voice recognition of the user's utterance. In addition, when the user's utterance is voice recognized, the controller 110 may transmit voice information such as silence and silence length to the processor 120 together. In addition, the text input through the user 10 (hereinafter referred to as an input test) may not have sentence boundary marks or may not be case-sensitive in the case of English.

The processing unit 120 performs a function of recognizing sentence boundaries of the input text based on the input text transmitted through the control unit 110 and the learning corpus stored in the storage unit 30 . A sentence boundary recognition method through the processing unit 120 is as follows.

First, the processor 120 extracts feature information including n-gram information of the input text. Also, as mentioned above, when the input text is generated through voice recognition, voice information including silence and silence length may also be included in the feature information.

Then, the processing unit 120 extracts sentence boundary tentative candidates for the input text and candidate probabilities for these sentence boundary tentative candidates based on the learning corpus stored in the storage unit 30 . Here, the learning corpus is defined as a corpus generated in advance based on text in a specific field as well as general text that can be secured in large quantities and stored in the storage unit 30 . In addition, the learning corpus may include engram data with a high probability of existing before and after sentence boundary marks, and probability information that may exist before and after sentence boundary marks for each engram data. Accordingly, the processing unit 120 may extract sentence boundary temporary candidates for the input text and candidate probabilities for the temporary candidates using the learning corpus stored in the storage unit 30 . Here, since the method of generating the learning corpus will be described again below, the description thereof will be omitted.

Then, the processing unit 120 calculates candidate scores for sentence boundary tentative candidates based on the candidate probabilities derived based on the learning corpus. Specifically, the processing unit 120 sets left engrams and right engrams for each tentative sentence boundary candidate in order to identify left and right contexts, and determines the probability that the set left engram is on the left side of the tentative sentence boundary candidate, and the right engram Based on the candidate probabilities including the probabilities of coming to the right of the provisional sentence boundary candidates, engram candidate scores can be calculated for each sentence boundary provisional candidate. In addition, the processing unit 120 may calculate a weighted candidate score based on the static sound and the static sound length information when there is voice information about the input text. Then, the processing unit 120 may calculate the candidate score by adding the engram candidate score and the weighted candidate score for each tentative sentence boundary candidate. Here, the calculation method for the candidate scores can be expressed as Equation 1 below.

In Equation 1, score is the candidate score for the boundary code, a is the weight, Score _{i -gram} is the probability score of the boundary code to the left and right of the engram, and Score _pause is the voice information if there is voice information. It represents a weighted candidate score calculated based on the information (probability score for Jeongeum in the corresponding location). And, in Equation 1, i in i-gram represents 1, ..., n, and may include, for example, unigram, bigram, and trigram. Here, the calculation method for score _{i -gram} can be expressed as in Equation 2 below, and the calculation method for score _pause can be expressed as in Equation 3 below.

In Equation 2, / is the sentence boundary, Pr(n-gram _left ) is the probability that the sentence boundary comes to the left of a specific engram, and Pr(n-gram _right ) is the sentence boundary that comes to the right of the specific engram. represents the probability.

And, in Equation 3, c and d represent weights, and Pr(pause) and Pr(silence_length) represent the probability of silence. Pr(pause) corresponds to, for example, spacing between sentences silence, and Pr(silence_length) represents silence between utterances between one sentence and the next sentence.

An example of calculating candidate scores through the aforementioned processing unit 120 is as follows. Here, the input text is assumed to be "i like it very much how about you". Also, in the following description, it is assumed that the input text is input without voice information. When the corresponding input text is input, the processing unit 120 may select "i like it / very much / how about you /" as a tentative sentence boundary candidate (/) based on the learning corpus stored in the storage unit 30. there is. As mentioned above, the learning corpus stored in the storage unit 30 includes engram data with a high probability of existing before and after sentence boundary marks, and probabilities of existence before and after sentence boundary marks for each engram data. information may be included. Accordingly, provisional sentence boundary candidates are set between it and very, between much and how, and after you, rather than between i and like, and between very and much, which have a relatively low probability of existing before and after sentence boundary marks. It can be.

Accordingly, in the example above, the left engram (the engram located to the left of the sentence boundary /) is "it", "like it", "i like it", "very much", and the right engram (the sentence boundary / is located on the left). Engrams located to the right of /) are more likely to be set to "how", "how about", or "how about you". Since engrams are assumed to include unigrams, bigrams, and trigrams, each tentative sentence boundary candidate may appear as described above. In addition, in the above example, very much has a very low probability of being a right engram, so whether to allow a sentence boundary temporary candidate to the left of very much depends on the specific candidate selection score calculation method and candidate selection score threshold ), depending on how you set it.

In addition, although English is used in the example presented for the description of the present invention, it should be understood that the method of the present invention can be implemented regardless of language.

After that, the processing unit 120 performs a function of comparing the candidate score derived based on Equations 1 to 3 above with a preset threshold score. Here, provisional sentence boundary candidates whose candidate scores are preset critical scores may be selected as sentence boundary candidates.

Then, processing unit 120 may select a sentence boundary code by analyzing sentence boundary candidates. Specifically, the processing unit 120 may add corresponding analysis information to feature information by analyzing sentence boundary tentative candidates. Here, the analysis information may include a probability that a left engram comes to the left of a provisional sentence boundary candidate, a probability that a right engram comes to the right of a sentence boundary provisional candidate, and an engram candidate score for each sentence boundary provisional candidate. In addition, the feature information may include a weighted candidate score calculated based on voice information when voice information exists.

That is, although sentence boundary candidates have already been selected through the above-described process, the processing unit 120 analyzes the left and right contexts and the entire context of the above-mentioned tentative sentence boundary candidates so that sentence boundary codes can be selected with higher accuracy. Add analysis information to information. Here, the feature information may also include vocabulary or stem engrams and part-of-speech engrams in order to grasp the left and right contexts of the tentative sentence boundary candidates. In addition, when a part of the learning corpus is a corpus with voice information and the input is text without voice information, 0 or a preset default value may be used for the static feature-related parameter.

After adding the above-described analysis information to the feature information, the processing unit 120 selects sentence boundary codes by applying the above-described feature information to a classification model including a support vector machine (SVM) and a maximum entropy model (MEM). Here, the reason why the classification models are described as the above two types is that the above two models can be operated lightly compared to other classification models. That is, the present invention is not limited to the above two models and various classification models may be applied.

The processing unit 120 applies the feature information to the above-described classification model to classify sentence boundary candidates as actual sentence boundary codes. The classification category corresponds to binary classification, with yes if it is a sentence boundary and no if it is not. Also, as a result of the classification, among sentence boundary candidates classified as yes, sentence boundary candidates are left in the corresponding text, and candidate positions classified as no are deleted from the corresponding text.

For example, in the sentence “i like it / very much / how about you /” mentioned above, the classification result through the processing unit 120 is that the first candidate is classified as no, and the second and third candidates are classified as yes. And as a result of processing the sentence boundary candidates according to the classification result, "i like it very much / how about you /" can be output as text with sentence boundary information.

In addition, in the above process, for the readability of the output text, when outputting the text with information on the sentence boundary, post-processing such as uppercase and lowercase processing and boundary generation in the case of English, "I like it very much. How about you?" It can be output as highly readable text such as

In addition, the processing unit 120 may construct a learning corpus used for sentence boundary recognition. As mentioned above, the learning corpus is characterized in that it is generated based on texts in a general field as well as texts in a specific field. Specifically, the learning corpus may be a voice transcription corpus or a voice recognition result corpus to which sentence boundary codes are added by a human, but may also be plain text without voice information at all, or a part of text in a specific field that is the same as the text to be classified. It may also be a text corpus combining large-capacity text in the general field. Accordingly, the processing unit 120 of the present invention may construct a learning corpus based on various texts. A method of constructing a learning corpus through the processing unit 120 is as follows.

First, the processing unit 120 extracts the engram frequency from the text of the learning corpus for constructing the learning corpus. After that, the processing unit 120 calculates the probability Pr(/|n-gram left) of these engrams to the left of the sentence boundary code and the probability Pr(/|n-gram right) of the engram to be located to the right of the sentence boundary code. do. In addition, if there is voice information in the learning corpus, silence can also be included in the engram, and the probability Pr(/|pause) and Pr(/|silence_length) of the sentence boundary code appearing at the corresponding position for each silence or silence length are calculated. The processing unit 120 may store the calculated scores in the storage unit 30 .

Then, processing unit 120 uses the candidate probabilities stored in storage unit 30 to recognize all possible sentence boundary candidates for the training corpus. Since the method for recognizing sentence boundary candidates is the same as the method for deriving sentence boundary candidates for the input text mentioned above with reference to Equations 1 to 3, an additional description thereof is omitted. In addition, as mentioned above, the sentence boundary candidate recognized in this way may actually be a sentence boundary, but may not exist.

Then, the processing unit 120 may select a sentence boundary code by analyzing the sentence boundary candidates derived through the above process. Specifically, the processing unit 120 adds analysis information to feature information by analyzing sentence boundary candidates, and selects sentence boundary codes by performing a classification process based on the feature information. Since the method for selecting such a sentence boundary code has been described in detail above, an additional description thereof will be omitted. However, if a sentence boundary candidate is actually a sentence boundary, it corresponds to an event or example sentence that is yes in the classification category, and if it is not an actual sentence boundary, it corresponds to an event or example that is no in the classification category. If part of the training corpus is a corpus with voice information and part is text, parameters related to the quiet characteristics of plain text may be set to 0 or a default value. These examples may be stored in the storage unit 30 through the learning process of the classifier.

In addition, it should be understood that the method of constructing the learning corpus described above is only an example and may be constructed in various ways.

2 is a block diagram of a processing unit 120 according to an embodiment of the present invention. As mentioned above, the processing unit 120 according to an embodiment of the present invention functions to recognize sentence boundaries of the input text based on the input text input from the user and the learning corpus stored in the storage unit. To this end, the processing unit 120 includes a feature information extraction module 121, a candidate probability extraction module 122, a sentence boundary candidate selection module 123, an analysis module 124, and a sentence boundary code selection module 125. can be configured. Here, since each component included in the processing unit 120 is a list of features mentioned with reference to FIG. 1 by function to help understanding of the specification, it should not be understood that the processing unit 120 includes all of these components. In addition, in the following, details overlapping with the detailed description of the processing unit 120 mentioned above are omitted and described.

The feature information extraction module 121 functions to extract feature information including n-gram information of input text. Also, as mentioned above, when the input text is generated through voice recognition, voice information including silence and silence length may also be included in the feature information.

The candidate probability extraction module 122 functions to extract provisional sentence boundary candidates for the input text and candidate probabilities for these sentence boundary provisional candidates on the basis of the learning corpus stored in the storage unit 30 .

The sentence boundary candidate selection module 123 functions to select a sentence boundary candidate from the sentence boundary temporary candidates extracted through the candidate probability extraction module 122 through a preset algorithm. Specifically, the sentence boundary candidate selection module 123 calculates candidate scores for sentence boundary tentative candidates based on the candidate probabilities stored in the candidate probability extraction module 122 . Here, the candidate scores may include probability scores of boundary codes coming to the left and right sides of the engram, and weighted candidate scores calculated based on the voice information when there is voice information. Here, since the calculation method for the candidate score has been described in detail with reference to Equations 1 to 3 above, additional description thereof will be omitted.

The analysis module 124 functions to analyze the sentence boundary candidates selected through the sentence boundary candidate selection module 123 . Specifically, the analysis module 124 functions to add analysis information to feature information by analyzing sentence boundary candidates. Accordingly, the feature information may further include a probability that the left engram comes to the left of the temporary candidate sentence boundary, a probability that the right engram comes to the right of the temporary candidate sentence boundary, and engram candidate scores for each temporary candidate sentence boundary. In addition, the feature information may include a weighted candidate score calculated based on voice information when voice information exists.

The sentence boundary code selection module 125 functions to select sentence boundary codes by applying feature information to a classification model. Here, the classification model may include SVM and MEM.

Also, although not shown in FIG. 2 , the processing unit 120 may perform a function of constructing a learning corpus. Since the description of the method of constructing the learning corpus has been described in detail with reference to FIG. 1, further description thereof will be omitted.

3 is a flowchart of a sentence boundary recognition method according to an embodiment of the present invention. Hereinafter, descriptions are made by omitting items overlapping with those described above.

First, a step of extracting feature information including engram information of the input text (S110) is performed. Here, the input text may include not only text input as text but also text recognized by voice recognition of the user's utterance. In step S110, when the user's utterance is voice-recognized, voice information such as silence and silence length may also be extracted.

Thereafter, a step of extracting sentence boundary tentative candidates for the input text and candidate probabilities thereof is performed (S120). Specifically, step S120 serves to extract provisional sentence boundary candidates for the input text and candidate probabilities for the sentence boundary provisional candidates for the input text, based on the learning corpus including sentence boundary information.

Thereafter, a step S130 of calculating candidate scores for provisional sentence boundary candidates based on the candidate probabilities extracted in step S120 is performed. As mentioned above, the candidate scores may include probability scores for border codes coming to the left and right of the engram, and weighted candidate scores calculated based on the voice information when there is voice information. To this end, step S130 is a step of setting left engrams and right engrams for each sentence boundary temporary candidate, and the probability that the set left engram is on the left side of the sentence boundary temporary candidate, and the right engram is on the right side of the sentence boundary temporary candidate. A step of calculating engram candidate scores for each tentative sentence boundary candidate based on the candidate probability including the probability of coming may be included. Further, step S130 may include calculating a weighted candidate score based on static sound and static sound length information when there is voice information about the input text. Here, since the calculation method for the candidate score has been described in detail with reference to Equations 1 to 3 above, additional description thereof will be omitted.

After that, a step of comparing the candidate score with a predetermined threshold score is performed. As a result of the comparison, provisional sentence boundary candidates whose candidate scores exceed a predetermined threshold score are selected as sentence boundary candidates (S140).

Thereafter, a step of analyzing sentence boundary candidates (S150) and selecting a sentence boundary code (S160) is performed. Specifically, step S160 is a step of adding analysis information to feature information by analyzing sentence boundary candidates, and selecting a sentence boundary code based on the feature information. Here, the feature information is based on the probability that the set left engram comes to the left of the provisional sentence boundary candidate, the probability that the right engram comes to the right of the sentence boundary provisional candidate, the engram candidate score for each sentence boundary provisional candidate, and voice information. At least one of the calculated weighted candidate scores may be further included.

In step S160, a sentence boundary code is selected by applying the generated feature information to a classification model. Here, the classification model may include, for example, SVM and MEM.

4 is a flowchart of a method for building a learning corpus according to an embodiment of the present invention.

First, an engram frequency is extracted from the text of the learning corpus for construction of the learning corpus (S210).

After that, a step (S220) of calculating engram candidate scores is performed. Specifically, in step S220, the probability Pr(/|n-gram left) of engrams to come to the left of the sentence boundary code and the probability Pr(/|n-gram right) to be located to the right of the sentence boundary code are calculated. include In addition, if there is voice information in the learning corpus, in step S220, silence may also be included in the engram, and the probability that a sentence boundary code comes at the corresponding position for each silence or silence length Pr(/|pause), Pr(/|silence_length) It may include the step of calculating .

Step S230 is a step of storing the engram candidate scores derived through step S220 in a storage unit.

Thereafter, based on the candidate probabilities stored in the storage unit, a step (S240) of extracting all possible sentence boundary candidates for the learning corpus is performed. Here, step S240 is performed for the input text described with reference to Equations 1 to 3 above. Since it is the same as the method of deriving sentence boundary candidates, an additional description thereof is omitted.

Thereafter, the sentence boundary candidates derived through step S240 are analyzed (S250), and analysis information is added to feature information. Then, by applying the feature information to a classification model, a sentence boundary code is selected, A step (S260) of storing this in the storage unit is performed.

As described above, the optimal embodiment has been disclosed in the drawings and specifications. Although specific terms have been used herein, they are only used for the purpose of describing the present invention and are not used to limit the scope of the present invention described in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: sentence boundary recognition device 110: control unit
120: processing unit

Claims (10)

In the sentence boundary recognizing method of the sentence boundary recognizing device,
extracting, by a feature information extraction module, feature information including engram information of the input text;
extracting, by a candidate probability extraction module, sentence boundary tentative candidates for the input text and candidate probabilities for the sentence boundary tentative candidates for the input text, based on a learning corpus including sentence boundary information;
calculating, by a sentence boundary candidate selection module, candidate scores for the sentence boundary tentative candidates based on the candidate probabilities;
selecting, by the sentence boundary candidate selection module, sentence boundary tentative candidates whose candidate scores are greater than or equal to a preset threshold score as sentence boundary candidates; and
analyzing, by an analysis module, the sentence boundary candidates based on the feature information and candidate scores for the sentence boundary tentative candidates, and adding the analyzed analysis information to the feature information; and
Selecting, by a sentence boundary code selection module, a sentence boundary code by applying the feature information to a preset classification model;
The learning corpus is
Engram data with more than a predetermined probability to exist before and after the sentence boundary code, and probability information that may exist before and after the sentence boundary code for each engram data;
The analysis information
A sentence boundary recognizing method, characterized in that it includes a probability that a left engram comes to the left of a provisional sentence boundary candidate, a probability that a right engram comes to the right of a provisional sentence boundary candidate, and an engram candidate score for each sentence boundary provisional candidate. The method of claim 1,
Calculating the candidate scores
setting left engrams and right engrams for each tentative sentence boundary candidate; and
Calculating engram candidate scores for each sentence boundary provisional candidate based on candidate probabilities including the probability that the set left engram is on the left side of the sentence boundary provisional candidate and the right engram is on the right side of the sentence boundary provisional candidate step;
Characterized in that it comprises a, sentence boundary recognition method. The method of claim 2,
The sentence boundary recognition method
extracting, by the candidate probability extraction module, static sound and static sound length information included in the speech information, when the speech information of the input text exists;
Characterized in that it further comprises, sentence boundary recognition method. The method of claim 3,
Calculating the candidate scores
and calculating a weighted candidate score based on the static sound and static sound length information when there is voice information for the input text. The method of claim 4,
Calculating the candidate scores
Sentence boundary recognizing method characterized in that for each sentence boundary provisional candidate, the engram candidate score and the weighted candidate score are summed. delete The method of claim 5,
The step of selecting the sentence boundary code is
A sentence boundary recognition method, characterized in that for selecting the sentence boundary code using a classification model including a support vector machine (SVM) and a maximum entropy model (MEM). The method of claim 1,
The method of recognizing sentence boundaries, characterized in that the engram includes a unigram, a bigram, and a trigram. A feature information extraction module for extracting feature information including engram information of input text;
a candidate probability extraction module that extracts sentence boundary tentative candidates for the input text and candidate probabilities for the sentence boundary tentative candidates based on the learning corpus including sentence boundary information;
a sentence boundary candidate selection module that calculates candidate scores for the sentence boundary tentative candidates based on the candidate probabilities, and selects sentence boundary tentative candidates whose candidate scores are equal to or greater than a preset threshold score as sentence boundary candidates; and
an analysis module for adding analyzed analysis information to the feature information by analyzing the sentence boundary candidates based on the feature information and candidate scores for the sentence boundary tentative candidates; and
A sentence boundary code selection module for selecting a sentence boundary code by applying the feature information to a preset classification model;
The learning corpus is
Engram data with more than a predetermined probability to exist before and after the sentence boundary code and probability information that may exist before and after the sentence boundary code for each engram data;
The analysis information
A sentence boundary recognizing apparatus, characterized in that it includes a probability that a left engram comes to the left of a provisional sentence boundary candidate, a probability that a right engram comes to the right of a provisional sentence boundary candidate, and an engram candidate score for each sentence boundary provisional candidate. The method of claim 9,
The sentence boundary candidate selection module
Set left engrams and right engrams for each tentative sentence boundary candidate,
Calculating engram candidate scores for each sentence boundary provisional candidate based on candidate probabilities including the probability that the set left engram is on the left side of the sentence boundary provisional candidate and the right engram is on the right side of the sentence boundary provisional candidate Characterized in that, sentence boundary recognizing device.

Images