KR102146625B1 - Apparatus and method for computing incrementally infix probabilities based on automata - Google Patents

Abstract

In the present invention, a character string is obtained as a regular expression using a DFA model obtained based on a deterministic finite automata, and then converted into a clear regular expression, and the appearance of a character string expressed as a clear regular expression using a probabilistic finite automata. It is possible to provide an automata-based incremental median probability calculation apparatus and method that enables the probability to be accurately calculated and to easily obtain the occurrence probability according to the increment of the string expressed by the regular expression by the incremental method of the regular expression. have.

Description

Device and method for calculating incremental median probability based on automata {APPARATUS AND METHOD FOR COMPUTING INCREMENTALLY INFIX PROBABILITIES BASED ON AUTOMATA}

The present invention relates to a median probability calculation apparatus and method, and to an automata-based incremental median probability calculation apparatus and method.

One of the main tasks in natural language processing is to calculate the probability of occurrence of a given phrase or of all phrases that match a given pattern.

In general, probabilistic context-free grammar or Probabilistic Finite Automata (hereinafter referred to as PFA) is mainly used to model the distribution of strings matching a given pattern with a language model. . Among these, the probabilistic finite automata uses PFA for most of the current speech processing tasks due to the advantage of being able to provide many probabilistic language phenomena in a simple yet powerful and understandable expression.

An important problem in PFA is calculating the probability of affixes for a given pattern distribution. That is, given a PFA and a character string (w), the probability of the character string (w) appearing at various locations in the distribution of the character string modeled by the PFA is calculated. For example, in the string distribution modeled by the PFA, the string (w) is placed in the form of a prefix such as wx by placing it in the front of another arbitrary character or string x, or in the form of a suffix such as xw by placing it at the rear end. Can appear. In addition, arbitrary characters or strings can be placed between x and y to appear in the middle, such as xwy, and the PFA must calculate the sum of the probabilities for all these cases.

The probability of a given string (w) appearing as a prefix is relatively easy to calculate, while the probability of appearing as a suffix or infix is easy to calculate the probability due to the problem that the string (w) may appear repeatedly. Not. Nevertheless, a method for calculating the probability that the character string w appears as a prefix, as well as a suffix or an infix, has already been conducted and is known.

However, in the existing studies, while the probability of the occurrence of a given string (w) can be calculated, there is a limitation in that the probability of the increment (or expansion) of the string (w) cannot be calculated. For example, the probability of the occurrence of the character string (w) in the distribution of the character string modeled by the PFA can be calculated, while the probability that the character (a) is added to the character string (wa) must be separately recalculated. Similarly, even if the probability for the string (wa) is calculated, the probability for the string (w) must be calculated separately.

Therefore, since the probability according to the increment of the string must be calculated separately, there is a problem that not only a very long calculation time is required, but also a lot of resources are required.

Korean Patent Registration No. 10-1645890 (registered on July 29, 2016)

An object of the present invention is to provide an automata-based incremental median probability calculation apparatus and method capable of accurately and quickly calculating a probability according to an increment of a character string, thereby easily obtaining an incremental median probability.

Another object of the present invention is to provide an automata-based incremental median probability calculation apparatus and method capable of simultaneously calculating the incremental median probability of a character string and calculating the probability according to an increment of a character string at high speed.

In order to achieve the above object, the automata-based incremental median probability calculation apparatus according to an embodiment of the present invention is based on a deterministic finite automata (hereinafter referred to as DFA) for each character of the character string w for calculating the appearance probability. A DFA model acquisition unit that acquires a DFA model composed of a plurality of states and transition functions to acquire a regular language; After adding an initial state, a single final state, and a transition function corresponding to the initial state and the single final state to the DFA model, paths that may overlap among the paths between the states of the DFA model are dynamically repeatedly removed, and A regular expression conversion unit that converts a regular expression into a DFA model capable of expressing a regular expression; A PFA model intersection that calculates the probability of each state and transition path of the transformed DFA model based on a PFA model obtained by a probabilistic finite automata (hereinafter referred to as PFA) and applies it as a weight of the transformed DFA model; And the character string (F(w)\) included in the character string set (F(wa)) in which the incremented character string (wa) appears among the character string set (F(w)) including the character string (w) in the weighted DFA model. Incremental probability of obtaining a DFA model to which an incremental weight is applied by adding a state and transition function for F(wa)), and calculating the occurrence probability of a character string and an incremented character string by using the acquired incremental weighted DFA model. Calculation unit; Includes.

The automata-based incremental median probability calculation method according to another embodiment of the present invention for achieving the above object is based on a deterministic finite automata (hereinafter referred to as DFA) for each character of the character string w for calculating the appearance probability. Obtaining a DFA model composed of a plurality of state and transition functions to obtain a canonical language; After adding an initial state, a single final state, and a transition function corresponding to the initial state and a single final state to the DFA model, paths that may overlap among the paths between the states of the DFA model are dynamically and repeatedly removed, and Converting a regular expression into a DFA model capable of representing; Calculating the probability of each state and transition path of the transformed DFA model based on a PFA model obtained by a probabilistic finite automata (hereinafter referred to as PFA), and applying it as a weight of the transformed DFA model; And the character string (F(w)\) included in the character string set (F(wa)) in which the incremented character string (wa) appears among the character string set (F(w)) including the character string (w) in the weighted DFA model. Obtaining a DFA model to which an incremental weight is applied by adding a state and a transition function for F(wa)), and calculating the occurrence probability of a character string and the incremented character string by using the obtained DFA model to which the incremental weight is applied; Includes.

Therefore, the automata-based incremental median probability calculation apparatus and method according to an embodiment of the present invention obtains a character string as a regular expression using a DFA model obtained based on a deterministic finite automata, and then converts it into a clear regular expression, Using a probabilistic finite automata, the probability of occurrence of a string expressed by a clear regular expression can be accurately calculated. In addition, the probability of occurrence according to the increment of the string expressed by the regular expression can be easily obtained by obtaining the incremental method of the regular expression.

1 shows a schematic structure of an automata-based incremental median probability calculation apparatus according to an embodiment of the present invention.
FIG. 2 shows an example in which the regular expression conversion unit of FIG. 1 adds states and transitions of a DFA model.
3 shows an algorithm of the intersection of the DFA model and the PFA model.
4 shows the relationship between a character string and a regular expression for its increment.
FIG. 5 shows an algorithm in which the incremental probability calculation unit of FIG. 1 accumulates and calculates a character string and a probability that the incremental character string appears from a DFA model.
6 shows the result of measuring the time required for operation according to the application of the incremental and crossover method according to the length of a character string and the size of a state set.
7 shows an automata-based incremental median probability calculation method according to an embodiment of the present invention.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the implementation of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the accompanying drawings. However, the present invention may be implemented in a number of different forms, and is not limited to the described embodiments. In addition, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it means that other components are not excluded, but may further include other components unless otherwise stated. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. And software.

1 shows a schematic structure of an automata-based incremental median probability calculation apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the automata-based incremental median probability calculation apparatus according to the present embodiment includes a string acquisition unit 110, a DFA model acquisition unit 120, a regular expression conversion unit 130, a matrix conversion unit 140, A PFA model intersection unit 150 and an incremental probability calculation unit 160 are included.

First, the character string acquisition unit 110 acquires a character string w for calculating an appearance probability. At this time, the string acquisition unit 110 may also acquire a language (sentence) to be searched for whether the string w is included. In the following, the set of characters is referred to as Σ, the set of all strings is referred to as Σ ^* , and the string to be searched (w) is an element (w = w) of the set of strings (Σ ^* ). ₁ , w ₂ , ..., w _n ∈ Σ ^* ). And the length (n) of the string (w) is |w| = n, and the empty string is called λ.

The DFA model acquisition unit 120 acquires a DFA model D modeled based on a deterministic finite automata (hereinafter referred to as DFA) having a unique state change for each character of the character string w.

The DFA model (D) is a model for converting the character string (w) to be searched into Regular Language (L(D)). In general, the 5 of (Q, Σ, δ, q ₁ , F) It is modeled to have a 5-tuple. Where Q represents a finite set of states of multiple states (q = q ₁ , q ₂ , ..., q _n ∈ Q), and δ is a transition function, where Q ), the transition path of the character string set Σ satisfies the condition of δ: Q × Σ → Q included in at least one of the states of the state set Q. And, q ₁ represents the initial state (q ₁ ∈ Q) of the state set (Q), and F is a set of final state having at least one state, which is in the state set (Q). It is the included set (F ⊂ Q).

Since the language is a set of character strings, the DFA model (D) modeled in advance by the DFA may recognize the acquired language as a regular language (L(D)). In this case, the DFA model D may acquire and recognize the regular language L(D) in the form of an unambiguous regular expression.

A regular expression is a general expression method of a regular language, and allows a simple expression of a regular language (L(D)) using a variety of known metacharacters. For example, the regular expression allows a regular language such as a, aa, aaa, ... to be briefly expressed as a ^* by using the metacharacter "*" meaning zero or more occurrences. In addition, in a regular expression, "+" is a metacharacter that means more than one occurrence, and "ab+c" may mean "abc", "abbc", "abbbc", and the like.

However, if the DFA model (D) recognizes the regular language (L(D)) and converts it into a regular expression, it is highly likely that an ambiguous regular expression will be generated.

As an example, search for the string baa in the regular expression ((a∪b) ^* aa(a∪b) ^* ) for the case where the characters in the set of strings (Σ) are a and b (Σ = {a, b}). If so, the character string baa can be searched in more than one position. To clarify this, if a subscript is added to the above regular expression ((a∪b) ^* aa(a∪b) ^* ), the regular expression ((a ₁ ∪b ₁ ) ^* a ₂ a ₃ (a ₄ ∪b ₂ It can be expressed as) ^* ). And in the regular expression ((a ₁ ∪b ₁ ) ^* a ₂ a ₃ (a ₄ ∪b ₂ ) ^* ), the string (baa) can also be searched as b ₁ a ₁ a ₂ a ₃ , and b ₁ a ₂ a It can also be searched for ₃ a ₄ . That is, in one regular expression ((a∪b) ^* aa(a∪b) ^* ), one string (baa) may be searched for duplicate. This is an ambiguous regular expression that generates an error that allows the occurrence probability of a character string (baa) to be calculated redundantly.

On the other hand, the regular expression (b*a(bb*a)*a(a∪b)*) is a clear expression that does not cause duplicate searches when searching for a string (baa).

In order to prevent such an error from occurring, the regular expression conversion unit 130 includes a clear regular expression from the regular language (L(D)) recognized by the DFA model (D) obtained by the DFA model acquisition unit 120. To the form of.

If the DFA model (D) is D = (Q, Σ, δ, q ₁ , F), the length of the set of states (Q) (|Q|) = n, and is sequentially ordered from 1 to n, the regular expression transformation The unit 130 adds two new states (q ₀ , q _n+1 ) to the specified state set (Q) of the DFA model (D). And a transition (δ(q ₀ , λ) = q ₁ ) is added to the DFA model (D) so that the added state (q ₀ ) becomes the newly designated starting state, and the added state (q _n+1 ) is newly designated. The same transition (∀q ∈ F, δ(q, λ) = q _n+1 ) is added to the DFA model (D) to be the only final state.

FIG. 2 shows an example in which the regular expression conversion unit of FIG. 1 adds states and transitions of a DFA model.

In FIG. 2, (a) and (b) respectively show an example of the DFA model (D) obtained by the DFA model acquisition unit 120, and (c) and (d) are respectively the regular expression conversion unit 130 ( A model in which new states and transitions are added to the DFA model (D) of a) and (b) is shown.

That is, in the new starting state (q ₀ ), an empty string (λ), that is, transitioned to the previous starting state (q ₁ ) without any additional condition, and all of the previous at least one final state (F) is the new and only final state (q _n+1 ) to be transferred without any additional conditions.

And, using Equation 1, states (q = q ₁ , q ₂ , ..., q _n ∈ Q) are dynamically repeatedly removed from the DFA model (D) to which states and transitions are added.

는 상태(q_i)로부터 시작하여, k(1 ≤ k ≤ n)번째 반복 제거에서 중간 상태(q_l)(여기서 l < k)인 경로로 상태(q_j)로 전이되는 문자열의 집합에 대응하며, 유사하게

는 상태(q_i)로부터 시작하여, 중간 상태(q_k)를 통해 상태(q_j)로 전이되는 모든 문자열에 대응한다.In Equation 1

Corresponds to the set of strings starting from state (q _i ) and transitioning from the k(1 ≤ k ≤ n) th iteration elimination to the intermediate state (q _l ) (where l <k) to state (q _j ) And similarly

Corresponds to all the strings that transition to the state (q _j), starting from the state (q _i), through an intermediate state (q _k).

로서 획득될 수 있다.Equation 1 follows the general concatenation, concatenation, and Kleene star rules for regular expressions, and by removing the paths of nestable regular expressions that may appear indistinctly from Equations 1 and 2, the regular language (L(D)) The clear regular expression (R) for

Can be obtained as

는 수학식 2의 조건에 따라 결정된다.here

Is determined according to the condition of Equation 2.

)에 대한 연산은 수학식 3의 4가지 연산 방식으로 수행된다.Also, in Equation 1, the letter (c) and the empty set (

) Is performed by the four calculation methods of Equation 3.

When the DFA model (D) is modified by the regular expression conversion unit 130 and a clear regular expression (R) is obtained through repetitive state detection, the clear regular expression (R) obtained by the matrix conversion unit 140 is obtained. Transform into matrix form.

The matrix conversion unit 140 maps a clear regular expression R according to Equation 4 and converts it into a matrix.

및

라 하면, 2개의 정규 표현식(R, S) 사이의 여러 연산은 수학식 5와 같은 행렬의 연산 형태로 표현될 수 있다.Here, matrices corresponding to two random regular expressions (R, S) are each

And

In other words, several operations between two regular expressions (R, S) may be expressed in the form of matrix operations as shown in Equation 5.

)로부터 정규 표현식(R)의 가중치를 획득하여, 정규 표현식 변환부(130)에 의해 수정된 DFA 모델(D)의 각 상태 및 전이 경로에 대해 확률로 표현되는 가중치를 적용함으로써, PFA 모델(P)과 교차된 DFA 모델(D)을 획득한다.The PFA model intersection 150 is a regular expression matrix for a clear regular expression (R) based on the PFA model (P) modeled by probabilistic finite automata (PFA).

) From the regular expression (R), and by applying a weight expressed as a probability to each state and transition path of the DFA model (D) modified by the regular expression conversion unit 130, the PFA model (P ) And the crossover DFA model (D) is obtained.

The PFA model (P) is a model for obtaining a probability function having a weight corresponding to [0, 1] for all sets of strings (Σ ^* ), and the weight (P(w)) of the string (w) and Calculate the weight (P(π)) of the path (π).

Considering the string to be searched (w = w ₁ , w ₂ , ..., w _n ∈ Σ ^* ), the corresponding path (π) is π = (q ₀ , w ₁ , q ₁ in the PFA model (P)) ), (q ₁ , w ₂ , q ₂ ), ..., (q _n-1 , w _n , q _n ).

The PFA model (P) is modeled to have a 5-tuple of (Q, Σ, δ, I, F), similar to the DFA model (D), where in the PFA model (P) the transfer function ( δ) satisfies the condition of δ: Q × Σ × Q → [0, 1], and I and F are the probability of each state of the set of initial state having at least a state (I: Q → [0, 1]), and F represents the probability (F: Q → [0, 1]) of each state of a set of final states having at least one state. Here, the default value of the transfer function δ is assumed to be 0. That is, if there is no transition between states, the weight is considered to be zero.

)이고, 모든 상태에 대해 최종 상태의 확률과 경로별 확률의 합은 1(

)이고, 모든 상태는 접근 가능하거나 상태간 상호 접근이 가능하다는 조건을 만족한다.And in the PFA model (P), the sum of the probabilities of all sets of initial states is 1 (

), and for all states, the sum of the probability of the final state and the probability of each path is 1(

), and all states are accessible or satisfy the condition that states are mutually accessible.

If the above conditions are satisfied, the weight (P(w)) for the character string w is 0≦P(w)≦1, and Σ _w∈Σ* P(w) = 1.

Accordingly, in the PFA model P, the weight, that is, the probability, for the path π may be calculated by Equation 6.

로 계산된다.If the set of all paths corresponding to the string (w) is Φ _w , then the weight, that is, the probability, for the string (w) in the PFA model (P) is

Is calculated as

)로 표현한 경우, PFA 모델(P) 는

와 같이 행렬 형태의 튜플을 갖는다. 여기서

는

인 |Q| × |Q| 전이 행렬의 집합이다. 그리고

는

인 1 × |Q|이고,

는

인 |Q| × 1 벡터이다.Meanwhile, as in Equation 4, the character (c) is a matrix (

), the PFA model (P) is

It has a tuple in the form of a matrix. here

Is

Phosphorus |Q| × |Q| It is a set of transition matrices. And

Is

Is 1 × |Q|,

Is

Phosphorus |Q| It is a × 1 vector.

Accordingly, when the character string w is expressed in the form of a matrix, the probability for the character string w matrix is expressed as in Equation 7.

로 표현하고, 0과 1을 각각 0 행렬 및 항등 행렬(identity matrix)로 표현할 수 있다.For brevity

And 0 and 1 can be expressed as a 0 matrix and an identity matrix, respectively.

로 계산될 수 있다.In the PFA model (P), the weight of the matrix for the set of characters (Σ) is

Can be calculated as

Accordingly, in the PFA model P, the probability that the character string w appears as a prefix (wΣ ^* ) or a suffix (Σ ^* w) for the character string set (Σ ^* ) can be calculated by Equations 8 and 9, respectively.

If the automaton (M) is Σ _q∈Q I(q) ≤ 1 or F(q) + Σ _{q'∈Q,c∈Σ} δ(q, c, _q'as required by the PFA, excluding ∀q ∈ Q) ) ≤ 1, the automaton (M) can be referred to as a sub-PFA. And sub-PFA(M) has 0 ≤ M(w) ≤ 1 as a weight (M(w)) for a character string (w) in a set of characters (Σ), and Σ _w∈Σ* M(w) ≤ 1.

The stochastic language for a set of characters (Σ) is a set (S) with S ⊆ Σ ^* , where each string in the set (S) is the probability of association (Pr _S (w)), 0 ≤ Pr _S (w) ≤ 1, and Σ _w∈Σ* Pr _S (w) = 1. If ∀w ∈ Σ ^* for a given probabilistic language (S) and Pr _S (w) = Pr _S (w), then the probabilistic language (S) is called a regular stochastic language.

In addition, the method of cross-applying the DFA model (D) and the PFA model (P) may be performed in a form of calculating and applying the weight of the regular language (L(D)), and according to Equations 10 and 1, the sub-PFA It can be obtained by generating ([D ∩ P]).

The intersection algorithm of sub-PFA ([D ∩ P]) according to Equations 10 and 11 is a new model with a state Q _W = Q _D × Q _P given a DFA model (D) and a PFA model (P) ( W), where the model (W) is δ for two states ((x, y), (x', y')) and the letter (c ∈ Σ) if _D (x, c) = x' _W ((x, y), c, (x', y')) = δ _P (y, c, y'), otherwise it appears as zero. Similarly, if the initial state of the DFA model (D) is p, then I _W ((p, q)) = I _P (q), and if the final state of the DFA model (D) is p', then F _W ((p' , q')) = F _P (q'), otherwise 0.

The intersection algorithm of the DFA model (D) and the PFA model (P) can be consequently organized as shown in FIG. 3.

3 shows an algorithm of the intersection of the DFA model and the PFA model.

According to the algorithm shown in FIG. 3, the sum of the weights of all character strings in the regular language L(D) can be calculated by Equation 12.

)에 대한 가중치를 수학식 13으로 획득할 수 있다.The PFA model intersection 150 is a regular expression matrix of a clear regular language (L(D)) from Equations 11 and 12

) Can be obtained by Equation 13.

The DFA model (D) intersected with the PFA model (P) in the PFA model intersection unit 150 is a model that calculates the probability of the occurrence of the character string (w). Therefore, it is not possible to calculate the probability that an increment (for example, wa) of the string (w) will appear.

Accordingly, the incremental probability calculation unit 160 acquires the DFA model (D) intersected with the PFA model (P) corresponding to the increment so as to calculate the probability that the increment of the string (w) will appear, and the PFA model for the acquired increment Using the DFA model (D) intersected with (P), the probability that the character string w and the incremental character string appear is calculated.

In the present embodiment, a character string language F(w), which is a set of character strings in which the character string w occurs only once, is defined. Here, the string language (F(w)) refers to a set of strings where w appears only once as a suffix. Thus, the set of strings (F(w)ㅇΣ ^* ) is the set of all strings containing the string (w), and given a clear regular expression for the string language (F(w)), the set of strings (Σ ^* ) Combined with, you can create a clear regular expression for all strings that contain string (w).

The probability of appearance for the character string language F(w) may be calculated using the DFA model D intersected with the PFA model P obtained at the PFA model intersection 150.

를 만족하는 언어 연산자이다.On the other hand, for a character (a ∈ Σ) causing an increment in the character string w, a regular language L satisfying F(wa) = F(w)ㅇL is searched. The regular language L satisfying F(wa) = F(w)ㅇL can be obtained as F(w)\F(wa). Where ＼ is given in two languages (R, S),

Is a language operator that satisfies

Therefore, given F(w) and F(w)\F(wa), F(wa) can be obtained. That is, in the present embodiment, F(wa) is not obtained directly, and a character string language (F(w)\F(wa)) belonging to the character string language (F(wa)) among the character string languages (F(w)) is used. , F(wa) = F(w)ㅇF(w)\F(wa) is obtained.

4 shows the relationship between a character string and a regular expression for its increment.

Fig. 4 shows a case where the character string w is a character a and the incremental character is expanded to a and ab.

Referring to Fig. 4, when the characters of the set of strings (Σ) are a and b (Σ = {a, b}), the form of the regular expression for the string language (F(w)) F(a) = b ^* Expressed by a In addition, the string language F(wa) in which the string w is incremented by the increment character a may be expressed as a regular expression (F(aa) = b ^* aㅇ(bb ^* a) ^* a). Here, b ^* a is F(a), and (bb ^* a) ^* a is F(a)\F(aa). Similarly, a string language (F(wab)) with an incremental character (b) further incremented is a regular expression (F(aab) = b ^* aㅇ() by F(aa) and F(aa)\F(aab). It can be expressed as bb ^* a) ^* aㅇa ^* b).

This is a clear regular expression for F(aa)\F(aab) corresponding to additionally incremented string languages (F(wa), F(wab)) based on the previously acquired string language (F(w)). When obtaining the DFA model (D) intersected with the PFA model (P) that expresses (D), that is, the weighted DFA model (D), the weighted string language (F(wa), F(wab)) is applied. It means that the DFA model (D) can be easily obtained.

로 추출될 수 있으며, 유사하게 문자열 언어(F(w = w₁, w₂, ..., w_k))에 대한 명료한 정규 표현식은

로 추출될 수 있다.Clear from the DFA model (D) obtained to have n+1 states including the final state (q _n+1 ) for the string language (F(w = w ₁ , w ₂ , ..., w _n )) The regular expression is

Can be extracted as, and similarly clear regular expressions for string languages (F(w = w ₁ , w ₂ , ..., w _k ))

Can be extracted with

이다. 따라서 수학식 14를 이용하여, DFA 모델(D)의 상태들(q = q₁, q₂, ..., q_n ∈ Q)은 동적으로 반복하여 제거될 수 있다.In the DFA model (D), in step (k) of the state removal procedure, the initial state (q ₀ ) is connected only to states up to k-1,

to be. Therefore, using Equation 14, the states (q = q ₁ , q ₂ , ..., q _n ∈ Q) of the DFA model D can be dynamically and repeatedly removed.

Meanwhile, Equation 14 may be simply expressed as Equation 15.

이며, 따라서 수학식 15는 수학식 16과 같이 표현될 수 있다.In Equation 15

Therefore, Equation 15 can be expressed as Equation 16.

And from Equation 16, F(w = w ₁ , w ₂ , ..., w _k-1 )\F(w = w ₁ , w ₂ , ..., w _k ) can be calculated by Equation 17 have.

According to Equation 17, it is possible to obtain a representing a clear regular expression for F(aa)\F(aab), and by adding a corresponding state and transition function to the weighted DFA model (D), The DFA model (D) to which is applied can be transformed to correspond to the incremented character string set (F(wa) = F(w)ㅇF(w)\F(wa)). In addition, the DFA model (D) to which the weight transformed in correspondence with the incremented character string (wa) is applied may calculate and output the appearance probability of the incremented character string (wa).

FIG. 5 shows an algorithm in which the incremental probability calculation unit of FIG. 1 accumulates and calculates a character string and a probability that the incremental character string appears from a DFA model.

In the algorithm of Fig. 5, the generation of (n+3) × (n+3) tables (T, T') in the DFA model (D) is as described above, where the DFA model (D) is a character string language (F(w = w ₁ , w ₂ , ..., w _n )), including n+1 states including the final state (q _n+1 ).

)는 이전 문자열로부터 기록된 결과이고, 초기값은

로 정해진다. 문자열(w = w₁, w₂, ..., w_k)에 대한 확률은

로 획득되며, 문자열(w = w₁, w₂, ..., w_k)은 도5 에 도시된 바와 같이, 증분될 수 있다.And vector(

) Is the result recorded from the previous string, and the initial value is

It is determined by The probability for a string (w = w ₁ , w ₂ , ..., w _k ) is

Is obtained, and the string (w = w ₁ , w ₂ , ..., w _k ) can be incremented as shown in FIG. 5.

6 shows the result of measuring the time required for operation according to the application of the incremental and crossover method according to the length of a string and the size of a state set.

In the conventional method, the probability of occurrence of each incremental character string has to be calculated, so the calculation time increases exponentially according to the number of incremental character strings, whereas in this embodiment, as the number of incremental characters increases, The speed of calculating the probability of occurrence of the sentence sequence is further reduced.

In the example shown in FIG. 6, when the automata-based incremental median probability calculation apparatus according to the present embodiment is used, when the size (|Q|) of the set of states (Q) is 1455, and the length of the string is incremented to 9 , It was confirmed that a maximum calculation speed improvement of 560.76% can be obtained.

7 shows an automata-based incremental median probability calculation method according to an embodiment of the present invention.

Referring to FIG. 1, the automata-based incremental median probability calculation method of FIG. 7 will be described. First, a character string w for calculating an appearance probability is obtained (S10). Then, the DFA model D is modeled and acquired based on the DFA for each character of the acquired string w (S20).

The DFA model (D) is modeled to obtain a regular language (L(D)) corresponding to the acquired string (w), and the regular language (L(D)) can be obtained in the form of a regular expression.

When the DFA model (D) is obtained, a new initial state (q ₀ ) and a new single final state (q _n+1 ) are added to the state set (Q) of the DFA model (D), and a corresponding transition function (δ) And, by dynamically repeatedly removing states (q = q ₁ , q ₂ , ..., q _n ∈ Q) from the DFA model (D) to which states and transitions are added according to Equation 1, DFA The model (D) is transformed to obtain a clear regular expression (S30).

When states and transitions are added to the DFA model (D) and transformed, each character of the regular expression is mapped to a predetermined matrix and converted into a regular expression matrix in the form of a matrix (S40).

And, based on the PFA model (P) modeled by the PFA, each state (i.e., character) of the DFA model (D) transformed into a regular expression matrix and the weight of the transition path (P(w), P(π)) are calculated. Thus, by applying the converted DFA model (D) to obtain a DFA model (D) crossing the PFA model (P) (S50).

The DFA model (D) intersected with the PFA model (P) can calculate the occurrence probability for the string (w), but cannot calculate the occurrence probability for the incremented string, so the string (w) appears only once as a suffix. F(w) subtracting the probability of occurrence for the set of strings (F(w)) and the set of strings (F(wa)) in which the incremented string (wa) appears as a suffix only once in the set of strings (F(w)) )DFA model applied by accumulating and applying additional state and transition functions to the DFA model (D) intersected with the PFA model (P) to calculate the probability of occurrence of \F(wa), and accumulating additional states and transition functions Using (D), the appearance probability of the set of character strings (F(wa)) incremented according to F(wa) = F(w)ㅇF(w)\F(wa) is calculated (S60).

The method according to the present invention may be implemented as a computer program stored in a medium for execution on a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and ROM (Read Dedicated memory), RAM (random access memory), CD (compact disk)-ROM, DVD (digital video disk)-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those of ordinary skill in the art will appreciate that various modifications and other equivalent embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

110: string acquisition unit 120: DFA model acquisition unit
130: regular expression conversion unit 140: matrix conversion unit
150: PFA model intersection 160: incremental probability calculation unit

Claims (10)

A DFA model acquisition unit that acquires a DFA model composed of a plurality of state and transition functions in order to obtain a regular language in the form of a regular expression based on a deterministic finite automata (hereinafter referred to as DFA) for each character of the character string w to calculate the probability of appearance. ;
After adding an initial state, a single final state, and a transition function corresponding to the initial state and a single final state to the DFA model, paths that may overlap among the paths between the states of the DFA model are dynamically and repeatedly removed, and A regular expression conversion unit for converting a regular expression into a DFA model capable of expressing a regular expression;
A PFA model intersection that calculates the probability of each state and transition path of the transformed DFA model based on a PFA model obtained by a probabilistic finite automata (hereinafter referred to as PFA) and applies it as a weight of the transformed DFA model;
In the weighted DFA model, the character string (F(w)\F) included in the character string set (F(wa)) in which the incremented character string (wa) appears among the character string set (F(w)) including the character string (w) (wa)) by adding a state and transition function to obtain an incremental weighted DFA model, and by using the acquired incrementally weighted DFA model, an incremental probability calculation that calculates the occurrence probability of the string and the incremented string part; And
Automata-based incremental median probability calculation apparatus comprising a matrix transform unit for converting each character of the regular expression into a matrix form in the DFA model transformed to express a clear regular expression by the regular expression transform unit. The method of claim 1, wherein the regular expression conversion unit
An initial state (q ₀ ) and a single final state (q _n+1 ) in the DFA model with a set of states (Q ∋ q) containing _n states (q = q ₁ , q ₂ , ..., q _n ). ) And the corresponding transfer function, and the equation

(here,

Denotes a set of strings starting from state (q _i ) and transitioning from the k (1 ≤ k ≤ n) th iteration elimination to the intermediate state (q _l ) (where l <k) to state (q _j ) ,

The state starting from a (q _i), through an intermediate state (q _k) represents all strings that transition to the state (q _j)) automata based incrementally median probability calculation apparatus in accordance with the repeated removal path. The method of claim 2, wherein the PFA model intersection
Based on the PFA model (P) modeled based on the PFA, the weight of the converted DFA model (D) is calculated to express a clear regular expression corresponding to the string (w).

(Where L(D) is a clear regular expression for the string (w) represented by the transformed DFA model (D))
An automata-based incremental median probability calculation device that is acquired and applied according to. The method of claim 3, wherein the incremental probability calculation unit
Duplicate strings (F(w)\) included in the string set (F(wa)) in which the incremented string (wa) appears among the string sets (F(w)) including the string (w) in the weighted DFA model. Equation of state and transition function for F(wa))

An automata-based incremental median probability calculation device that acquires a DFA model to which an incremental weight is applied by acquiring according to and adding it to a weighted DFA model. delete Obtaining a DFA model composed of a plurality of state and transition functions in order to obtain a regular language in the form of a regular expression based on a deterministic finite automata (hereinafter referred to as DFA) for each character of the character string w for calculating an appearance probability;
After adding an initial state, a single final state, and a transition function corresponding to the initial state and a single final state to the DFA model, paths that may overlap among the paths between the states of the DFA model are dynamically and repeatedly removed, and Converting a regular expression into a DFA model capable of representing;
Calculating the probability of each state and transition path of the transformed DFA model based on a PFA model obtained with a probabilistic finite automata (hereinafter referred to as PFA), and applying the transformed DFA model as a weight;
In the weighted DFA model, the character string (F(w)\F) included in the character string set (F(wa)) in which the incremented character string (wa) appears among the character string set (F(w)) including the character string (w) obtaining a DFA model to which an incremental weight is applied by adding a state and a transition function for (wa)), and calculating the occurrence probability of a character string and the incremented character string using the obtained DFA model to which the incremental weight is applied; And
Automata-based incremental median probability calculation method comprising converting each character of the regular expression into a matrix form in a transformed DFA model to represent a clear regular expression. The method of claim 6, wherein converting to the DFA model
An initial state (q ₀ ) and a single final state (q _n+1 ) in the DFA model with a set of states (Q ∋ q) containing _n states (q = q ₁ , q ₂ , ..., q _n ). ) And a transfer function corresponding thereto; And
Equation

(here,

Denotes a set of strings starting from state (q _i ) and transitioning from the k (1 ≤ k ≤ n) th iteration elimination to the intermediate state (q _l ) (where l <k) to state (q _j ) ,

Starting from state (q _i ) and repeatedly removing paths according to all strings that transition to state (q _j ) through intermediate state (q _k ); Automata-based incremental median probability calculation method comprising a. The method of claim 7, wherein applying the weight of the DFA model
Based on the PFA model (P) modeled based on the PFA, the weight of the converted DFA model (D) is calculated to express a clear regular expression corresponding to the string (w).

(Where L(D) is a clear regular expression for the string (w) represented by the transformed DFA model (D))
Automata-based incremental median probability calculation method obtained and applied according to. The method of claim 8, wherein calculating the probability of occurrence
Duplicate strings (F(w)\) included in the string set (F(wa)) in which the incremented string (wa) appears among the string sets (F(w)) including the string (w) in the weighted DFA model. Equation of state and transition function for F(wa))

Acquiring according to the method and adding to the weighted DFA model;
Obtaining a DFA model to which incremental weights are applied according to the weighted DFA model to which the state and transition function are added; And
Calculating a probability of occurrence of an incremental character string using the DFA model to which the incremental weight is applied; Automata-based incremental median probability calculation method comprising a. delete

Images