KR19990016586A - Optimal Product Approximation Method of Probability Distribution by K-Kernel Dependency and Multiple Decision Combining Method by Dependency - Google Patents

Abstract

An object of the present invention is to provide an optimal product approximation method of a probability distribution due to a higher order dependence of a first order or higher order.

It is another object of the present invention to provide a method for obtaining a constitutive distribution term in an optimal product approximation of a probability distribution due to a higher order dependency.

It is still another object of the present invention to provide a method of combining multiple decisions by dependency.

In order to achieve the first object, The optimal product approximation method of the probability distribution due to the differential dependence,

K set of determiners ,

A set of L decision candidates ,

Input ,

The degree of dependency is ,

If it is,

Probability of force terrier end

= when,

= Lt; / RTI &

end Lt; / RTI >

If it is,

The And

= If the relationship is established,

The above-mentioned high-order probability distribution To In a method of optimally approximating a product of a low-order probability distribution by a differential dependence,

Obtaining a primary dependency relationship;

Obtaining a conditionally independent assumption;

Obtaining a secondary dependency relationship;

Obtaining a conditional primary dependency;

.

.

.

Obtaining a differential dependency; And

Conditional And a step of obtaining a differential dependence relationship.

In order to achieve the above-mentioned second object, The method of obtaining the constituent distribution term at the time of the optimal product approximation of the probability distribution by the differential dependence is as follows:

K set of determiners ,

A set of L decision candidates ,

Input ,

The degree of dependency is ,

,

Probability of force terrier end

= when,

= Lt; / RTI &

end Lt; / RTI >

when,

The And

= If the relationship is established,

A method for finding a constituent distribution term at an optimal product approximation of a probability distribution by a differential dependence,

Actual probability distribution , An approximate probability distribution when,

Defined as

Minimize the measure of closeness Quot;

for ㉠ do / * Primary dependency * /

/ * Here, the above is the first dependency relation If

And

Given as * /

for ㉡ do / * Secondary dependency * /

/ * Where the above is the second dependency relation If

And

Given as * /

........ ........

for ㉢ do / * ( -1) Dependency relation * /

/ * At this time, -1) As a dependency relation 0 i ( -1) (j), ..., i1 (j) j

And

Given as * /

while (㉣) do / * Car dependency relationship * /

/ * ≪ / RTI > 0 as a car dependency i (j), ..., i1 (j) j

And

Given as * /

.......

end

end

.......

end

end

, One primary dependency, one secondary dependency, ..., one ( -1) dependency, (K- )doggy And the dependency relation -1) nested for loops and one while loop.

According to another aspect of the present invention,

K set of determiners ,

A set of L decision candidates ,

Input ,

The degree of dependency is ,

When there is a relationship, A method for combining a decision method by performing a method of obtaining an optimal product approximation of a probability distribution due to a differential dependence and a constituent distribution term for this with a normal Bayesian decision method,

Referring to the training sample data, A first step of obtaining an optimal product approximate set of a probability distribution by a differential dependence; And

A second step of probabilistically combining the determinations of the plurality of determinants by applying the approximate set obtained in the above step to the combination formula with the Bayesian method

And is characterized by considering dependencies between determinants that make multiple decisions without requiring an independent assumption.

The application fields and effects of the present invention are as follows:

1) The present invention avoids the problems that can be caused by independent households by not taking an independent assumption.

2) In approximating the higher-order probability distribution by the product of the lower-order probability distributions, it is possible to obtain the optimal product approximate distribution set based on the order dependence relation while changing the dependency order.

3) The required storage complexity is larger than that of the independent family, but it is smaller than the BKS method. Because O (L ² ) O (L ^{k + 1} ) O (L ^{K + 1} ).

4) Extending existing research results on product approximation by suggesting a methodology that can deal with higher order dependency, not just the first dependency relation.

5) It has been shown that combining the decisions of multiple determinants based on higher order dependence is superior in performance.

6) In the field of pattern recognition, the present inventors have excelled in the present invention through experiments in which a plurality of recognizers are used to combine the determinations.

7) In the field of pattern recognition, the excellence of the present invention is shown through experiments combining a plurality of recognizers with the determinations. Although the present invention has been described with respect to only the embodiment of the pattern recognition field, the idea of the present invention is applicable to other fields such as group decision making and control fields.

Description

Optimal Product Approximation Method of Probability Distribution by K-Kernel Dependence and Multiple Decision Combining Method by Dependency (fdsafdsa)

The present invention relates to an optimal product approximation method of a probability distribution by a second dependence relation, an optimal product approximation method of a probability distribution by k-order dependency expansion thereof, an optimal product approximation of a probability distribution by a first- A method for obtaining the constituent distribution term, and a method for combining multiple decisions by dependency.

In the determiner for performing the discrete decision, there are a voting technique (absolute majority or simple majority) and a behavior-knowledge space (BKS) technique in the prior art.

The voting technique will be obvious to those skilled in the art and will not be described. However, the BKS technique is briefly described below.

Suppose that K determinants select one of L decision candidates.

Then, a crystal vector composed of K elements is observed. The BKS technique generates a (K + 1) -dimensional table by accumulating the decision vector of the K-dimension and the actually determined value from the training sample data.

The BKS technique observes the decision of the K determinator for unknown inputs and assigns the decision with the highest frequency among the decisions that are true in the case to the generated table as the combined decision in that case do.

If no match is found, it is operated in such a way as not to make a combined decision.

Therefore, theoretically, to operate the BKS technique, Is required.

So far, many decision combining methods have been proposed for combining multiple recognizers, but most of these methods assume an independent relationship with the recognizer. Huang and Suen proposed a Behavior-knowledge Space (BKS) method to avoid independent families. However, this BKS method has a problem that the calculation and the required storage amount are geometrically complicated, and there are many rejections in crystal bonds.

However, unlike the above, there is a method of considering the dependency between the recognizers. This dependence of the cognitive period plays an important role in the probabilistic multiple decision method without independent assumptions.

That is, the method is such that the higher order rate distribution obtained from the training sample is best approximated to the product of the k-order dependence or the (k + 1) -order distribution, and then the multiple decisions are combined into an approximate distribution using the Bayesian method.

BACKGROUND OF THE PRIOR ART OF THE FIELD OF THE INVENTION

The background of the prior art of the pattern recognition field to which the present invention is applied will now be described. The ultimate goal of the research carried out in the field of character and pattern recognition is to improve recognition performance. In order to achieve this purpose, research has been conducted in two major directions.

One was research to improve the performance of the recognizer itself.

The other was research on combining multiple recognizers in a crystal-bonding method.

There are a number of recognizers or recognition algorithms to accomplish the above objects, but any recognizer or recognition algorithm among them is not as good as expected and has a single recognizer with both accuracy and robustness to handle challenges to real- Or recognition algorithms have not yet been provided.

The biggest difficulty in how to improve the recognizer itself lies in the fact that it is not easy to group them together into a single recognizer, although many features of various types can be used.

The multiple recognizer combination is based on the assumption that two are better than one. This is due to the fact that if multiple compensation features and recognizers are used simultaneously, these compensation features and recognizers improve recognition performance. Various knowledge sources are essential for carrying out research to solve the problem of real life pattern recognition. Several researchers have presented the results of the improved research on the recognition problem using various alternative algorithms.

In most cases, the multiple recognizer combinations are made by combining the determinations of these recognizers (i.e., the recognition results) in parallel or sequentially.

In general, the decision type of the recognizer depends on one of the three-step recognition results such as measurement score in the measurement step, ranking in the ranking step and top selection in the extraction step as shown in Fig.

It is known that the ranking provides more information than the top selection and is less dependent on the recognizer than the measurement score.

It is a well-known fact that, in systems that make group decisions or in systems that support group decisions, the accuracy of group decisions is more important than the accuracy of individual decisions. A number of experts with different backgrounds, diverse expertise and experience can make group decisions in a mutually collaborative manner, providing a much better decision and solution than a single expert

When a number of researchers are incompatible with their decisions, a number of methods have been proposed to reach consensus. For example, there are methods such as voting method, group opinion matching function, social selection function, social security function, and so on.

In this situation, the recognizer corresponds to a group of experts of a plurality of experts. In addition, multiple recognizer combinations are referred to as multiple expert combinations in other pattern recognition applications. As an interdisciplinary approach, some group decision making methods have been applied to multiple recognizer combinations, and improved research performance has been reported in several papers.

An example is the voting method and the Borda count method. The multiple recognizer combining process proceeds as follows:

1) When input x is given in parallel to _K recognizers (C ₁ , C ₂ , ..., C _K ), K determinations are observed according to each decision type.

2) Combine the observed observations based on the Consensus expected to produce the best combination decision.

3) For convenient and consistent combination of observed crystals, it is assumed that all crystals are represented in the same form in most crystal-combining methods.

In summary, multiple recognizer combinations form a consensus to achieve best perceived performance.

Many decision - combining methods have been proposed in the field of character recognition, but most of them have not focused on the dependency between the recognizers and assumed the recognition period independence.

Thus, the combining performance of multiple recognizers is degraded or biased when adding high dependency recognizers. Huang and Suen proposed a behavior-knowledge space (BKS) method that no longer requires independent assumptions. However, in order to apply this BKS method to the combination of K recognizers, the (K + 1) -order probability distribution consisting of one decision variable and K determinations is geometrically complex and, according to the theoretical analysis, Management is impossible. So we need an approximate method.

Lewis explored the problem of approximating the nth order binary distribution by multiplying a number of low order component distributions using the Hartmanis expansion concept. This product approximation method is based on information measurement and maximum entropy criterion for the similarity of two distributions.

Chow and Liu solved the optimization problem by Lewis and presented a procedure to optimally approximate the nth order binary distribution with (n-1) second-order component distributions. This optimization procedure is based on maximizing mutual information between the two component variables and the Maxi -mum Weight Spanning Tree (MWST) algorithm.

This mutual information is defined as a quantitative measure of how certain events (ie, decisions) are made that indicate some alternative possibilities. Dependency refers to the predictive ability of a recognizer to make decisions on other identifiers, and is statistically determined by calculating the average mutual information.

The k-th dependency is defined as the expected performance for the decision of the other k recognizers in one recognizer. Since it is difficult to measure the recognition period k-order dependency by determination at the ranking level, it is assumed that only the determination at the extraction level participates in evaluating the k-th dependence.

This dependence is the theoretical basis for the approximation of a higher order probability distribution as a product of lower order distributions. Since recognizers tend to be statistically dependent on other identifiers, it is assumed that identifiers that perform independently are vulnerable.

Therefore, from this point of view, it is preferable to consider the dependence of the recognition period on the multiple recognizer combination.

Most of the existing decision-combining methods assume that the recognizers perform independently of each other. And, in the multiple recognizer combination, the high-order probability distribution estimates are mostly assumed to be conditionally independent of each other for the given input. Conditional independent families have the advantage of requiring simple calculations and small amounts of storage, but often they are not in real situations. Adding a non-member recognizer that is highly dependent on the member recognizer causes the problem that the combined decision is biased toward the determination of the dependency recognizer and that the multiple recognizer combining performance can be degraded for the dependency recognizer. Furthermore, since the recognizer tends to be statistically dependent on other recognizers, the assumption that the recognizers perform independently is not desirable. Therefore, a decision-making method that does not have independent assumptions can be considered desirable.

On the other hand, the BKS method has the problem that the evaluation and storage of the high-order probability distribution is geometrically complicated and can not be controlled from the theoretical analysis point of view. In addition, the BKS method tends to have a high rejection rate for a large number of crystals that are not visible if there is sufficient and representative training data set in practical applications. To reduce the geometric complexity and potential rejection rates, the higher-order probability distributions should be approximated by the product of the lower-order component distributions. The dependence contributes to the identification such as the optimal approximation of the higher-order probability distribution.

And as an intermediate method of the above methods, there is a need for a new methodology that considers dependencies to overcome the drawbacks of the prior art methods and take advantage of them.

Chow and Liu proposed an approximation method of the nth order probability distribution as a product of the second order distribution considering the first order tree dependence like Lewis. However, it is often the case that one decision is based on two or more different recognizers.

In this case, the primary dependency is not adequate to adequately evaluate the high-order probability distribution. Therefore, 1 k K, a new method is required to optimally approximate the high-order probability distribution by the product of k-order dependence or (k + 1) -order distribution.

Hereinafter, a conventional study of multiple decision bonds relating to interdisciplinary background and expertise is described.

This paper is concerned with the study of multiple decision combinations in pattern recognition and group decision making, and explains some research on the approximation of higher order distribution based on dependency trees.

First, conventional studies on multiple decision combinations to improve decision quality are explored in the group support system and pattern recognition area including group decision making. And a study on the approximation of higher order discrete probability distribution as products of lower order component distribution is introduced.

These studies provide the theoretical basis and motivation for the dependency - based methodology that considers the cognitive period dependency and excludes independent assumptions.

Conventional multiple crystal bonding method

Decision-combining method of multiple recognizers

Decision combining of multiple recognizers, which are studied by many researchers in the field of pattern recognition, develops a number of decision-combining methods to achieve improved performance goals. The multiple decision binding method can be classified into three steps according to each type of decisions depending on one of the three measurement results, such as the measurement score of the class in the measurement step, the rank of the class in the ranking step, and the single selection in the extraction step. From the determination of the measurement score, ranking is assigned to each class by class ordering according to the measurement score, and ranking is performed. A single selection decision is made by selecting a category with the best measurement score or the highest ranking. The recognizer making a decision at the ranking decision plane can participate in the decision combination at the ranking step and the extraction step.

The decision-combining methods in the measurement phase include averaged Bayes recognizer, multiple recognizer integration using fuzzy or fuzzy logic, LCA (Linear Confidence Accumulation) method, use of Dempster-Shahfer theory, and use of neural network combiners.

In the case of the extraction step, common voting methods, use of the Bayesian method as an independent assumption, use of the Dempster-Shafer formula used for evidence estimation, and the BKS method.

In particular, in the use of the Bayesian method by Xu et al., Xu et al. Assume that the recognizer used in the independent feature set or the independently trained recognizers perform mutually independently. This approach causes problems when the recognizer is not functionally independent.

In the ranking step, the decision combination method includes a higher order method, a Borda count method, a use of a logistic regression method, and a CAL (Candidate Appearance Likelihood) method.

Method of bonding in measuring step

Fu used syntactic and statistical methods in pattern recognition. In his method, the set of pattern primitives is first extracted from the image data and represented as a symbol string. Also, the attributes of a primitive are represented by the parameter vector associated with this primitive. The (non) similarity score between two descriptors is defined as the sum of the syntactic difference between the symbol strings and the semantic difference between the associated parameter vectors. The use of attributes and the use of properties and primitives together are attractive, but they cause problems in the similarity of measurements.

These two differences are measured on an incompatible scale and must be properly normalized before combining. Also, the parameter vector is always associated with a primitive, so there is a lack of independence between the information contained in the two descriptor types.

Xu et al. Use a Bayesian recognizer and a posterior probability computed by a Bayes recognizer for decision combining at the measurement level. The majority of the measurement points are combined by averaging the measurement points or by selecting the median of the measurement points. Their research approach can be extended to k-NN (Nearest Neighbor) recognizers and differential recognition airflows.

Tahani et al. Proposed a method based on fuzzy integral to nonlinearly combine objective evidence in the form of a fuzzy membership function using subjective evaluation of the value of a decision source.

This fuzzy integral differs from other paradigms in terms of the objective evidence supplied by the various sources and the expected value of the subset of the sources considered in the fusion process.

The most important advantage of this method is not only that objective evidence is combined, but also the relative importance of different sources is taken into account. However, this method has a problem that the subjective evaluation is weak. Cho used multiple neural networks with different structures.

The decision of this multiple neural network is combined with the voting method, the Borda count method and the fuzzy integral. In his work, he argued that the fuzzy integral is better in terms of cognitive performance for online handwriting problems.

Huang and Suen introduced one approach for cognitive period combining, known as the LCA method, in which each recognizer can provide a bracket label and corresponding measures. In preliminary experiments, the LCA method showed better performance than the voting and Bayesian methods.

Huang et al. Used a multi-layer neural network to accumulate probabilistic measures using data transformation functions. Their methodology consists of two steps: data transformation and data recognition.

In the data modification step, the output value of each recognizer is transformed into a likelihood measurement form.

At the data recognition stage, the neural network was found to be very suitable for accumulating modified outputs to yield final recognition results.

Yamaoka et al. Proposed the integration of heuristic in the integration of recognition results using mathematical modeling method based on fuzzy logic. The output unit of each recognizer must be renormalized in order to fuse the recognition results in the measurement step to valid ones. This convergence algorithm consists of two steps: a data normalization and qualification process and a fuzzy integration process.

They support fuzzy logic in that they provide a means of expressing imprecise and incomplete recognition results and are flexible in expressing expert knowledge and statistical knowledge, but Bayesian methods are more sophisticated and complex due to the computation and structure of confusion tables . However, renormalization of different values on the scale is not easy.

Rogova presented a method of combining multiple neural network recognizers based on the Dempster-Shafer theory for evidence and presented two new set of support functions for proof calculation.

Lee and Srihari proposed a single layer neural network to combine all three types of recognizers and to use all available information in mixed-type situations. In addition, the neural networks they consider can handle elimination and correlation recognizers of redundant recognizers by adding control terms to the square root mean error learning rules. For all type combinations, the output of each recognizer at the ranking and extraction levels is re- Can be assigned to an input node.

Dimauro et al. Published a methodology for the evaluation of the binding process. They considered the performance of the combining process that was evaluated taking into consideration the characteristics of the combined set of recognizers. We introduced a well-defined recognizer similarity index for evaluation. The output of the recognizer is assumed to be a ranked candidate list with a confidence value for each of the ranked candidate lists. This systematic approach for evaluating the combining process expresses steps to support the design of an advanced multi-recognizer system.

How to combine in the ranking step

Hull and Ho supported rankings as recognition results. They proposed a ranking step method to combine the determinations of different identifiers. In particular, Ho explains the advantages of ranking usage compared to the measurement score and the top-level selection, and suggests some methods for candidate set reduction and reordering. A common way to reduce candidate sets is intersection and union. This approach aims to reduce multiple classes in a given set without loss of true class. Candidate set reordering includes useful decision-combining methods such as the high-ranking method, the Borda count method, and the logistic regression.

The Borda count method is well known as one of the social selection functions and is also known as a function in group decision making.

The Borda count method assigns the same weight to each participant recognizer, but the use of logistic regression allows assigning non-identical weights to the participant recognizer depending on the importance contributing to better performance. They reported the use of logistic regression in their paper to show better recognition performance than other methods.

Tsutsumida et al. Studied a multi-recognizer system for achieving high recognition performance by combining compensation recognition algorithms and obtained the result of IPTP (Characteristic Recognition Competition of Ministry of Posts and Telecommunications, Tokyo, Japan) Explored. They conducted a new experiment on the application of the 3-layer back-propagation learning method as a CAL method exploiting new experimental conditions and as a combining method to accumulate extension candidates. Thus, the results of various recognition algorithms have been transformed from the ranking list in the ranking step to the numeric list in the measurement step using the data transformation.

Coupling method in the extraction step

Mazurov et al. Discussed the application of the committee method to the two-class decision problem when the final decision was made by a multiple voting method. They showed a minimum committee existence axiom and an algorithm for constructing this axiom. This method is not easy to extend to multi-class problems.

Suen et al. Proposed the use of multiple recognizers for handwritten digit recognition when decisions are combined by multiple votes. This voting mechanism is performed by accepting the highest class by each recognizer. If the recognizer can not determine from the candidate bracket set, the votes of the recognizer are evenly divided into candidate brackets.

Nadal et al. Described several heuristic learning rules for decision combining by a similar set of recognizers. In multi-class problems such as character recognition, more information from recognizers is useful than just top-level selection.

Alpaydin proposed independently training multiple neural networks with different learning algorithms or different sets of parameters for the same learning algorithm and then casting votes in a weighted multiple form for multiple neural networks.

The voting method was explored static versus dynamic, and dynamic exploration also considered neural network complexity. The system assigns trust weights to each participating neural network according to its degree of success and complexity.

Lam et al. Discussed the theoretical considerations related to the use of multiple polls to combine the results of multiple identifiers. In particular, they examined the performance of combinatorial games when new recognizers were added and led to conditions under which group performance would be improved by the conditions. These studies are based on an understanding of why and how the decision maker combines decisions by multiple polls to produce improved recognition results, and from the work that is expected to occur, do.

Mandler et al. Have presented the advantages of using multiple recognition algorithms. They described a method for decision-combining of independent recognizers based on Bayes decision theory and Dempster-Shafer evidence theory. The statistical model and Bayes decision theory are used to transform the difference measure calculated by the recognizer into a confidence value associated with each class, and then combined by evidence theory rules. The great difficulty of this method is to transform from difference measure to confidence value.

Franke et al. Proposed two different methods for combining the results of different recognizers.

The first method is based on the Dempster-Shafer proof theory and the second method is a statistical method with some assumptions about the input data. In this way, each recognizer's vote is considered as an independent evidence source and has a pro or contra value as a binary value.

The second method assumes that the vote has a binary value and is described as an optimization problem that can be solved by a common mean-square-error method. They are tested for user-dependent recognition of online handwritten characters.

Xu et al. Proposed several methods for modeling the multiple-recognizer combination by voting method, Bayesian method, and Dempster-Shafer proof theory. Using the effective Dempster-Shahfer evidence theory, they collected evidence in groups that affect the same decision in each group, then combined the evidence separately for each group using the formula for the simple function by Barnett .

Thus, if L is the number of candidate classes, the computation of the entire procedure is reduced to an O (L) order. Several decision rules have been proposed for applying to voting methods.

In the use of the Bayesian method, the recognizers are assumed to be mutually independent since they use a set of independent features or because the recognizers are trained by independent training sets. Since each recognizer calculates its own posterior probability (or confusion probability) for the top choice, all the posterior probabilities (or confusion probabilities) are multiplied by mutually independent assumptions for decision combinations. This method causes problems when the recognizers are not mutually independent.

Lu et al. Also used the Dempster-Shafer proof theory for the integration of free handwritten digit recognition. They proposed multiple recognizer systems consisting of three different recognizers, such as neural networks, structural template map recognizers and polynomial recognizers. They proposed inference rules for more accurate and accurate recognition results based on algorithms and combining evidence for combining and modeling recognition results. The input of this system is different from that of Xu and Mandler in that it contains a bracket label and an associated trust value.

Huang et al. Proposed a new method called BKS (Behavior-knowledge Space) method which accumulates decisions from individual recognizers and induces final decision. The BKS method comprises a knowledge-modeling step of constructing a higher order frequency table by class for the decisions of apparently coherent recognizers from the training data and an operation step of deriving a final decision according to the frequency for the decisions of the given joint recognizers . If the behavior of the actual sample and the behavior of the recognizer for the same learning sample coincide, the BKS method claims that each recognizer can be proved to be the best method in that it provides only one class at the extraction level.

However, if a decision of an unseen recognizer is observed, the BKS method rejects its final decision. Moreover, a larger number of recognizers theoretically require more computation and storage. These facts are disadvantages of the BKS method as described above.

Kittler and Hatef proposed a common theoretical method of combining identifiers using discrete pattern representations. In other words, based on Bayesian decision rules, we provide a theoretical basis for combining existing recognizers. Furthermore, an analysis of the sensitivity of the combining method to the evaluation error was performed. Their experimental results show that the sum rule, which is the maximum limit assumption, and its derivative rules, that is, the minimum rule, the maximum rule, the median rule, and the combination rule developed under the maximum voting rule consistently outperform other combining methods.

Methods of binding crystals in other papers

Several other methods have been proposed for carrying out committee decisions in other papers. These include useful voting and Borda counting methods in multiple recognizer systems. There are also other coupled models including Bayes decision, Dempster-Shafer proof theory, fuzzy theory and generation rules.

How to Combine Group Decisions

Group consensus functions, social selection functions, and social security functions have been developed to combine the ranking given by the decision makers' committees. The multiple voting function is a specific group matching function.

Arrow discussed conditions for these functions and various assumptions based on probability axioms.

Black presented a history of mathematical theories for committee decisions and elections, and logic for committee conditions and elections.

Formal descriptions of social selection functions and social security functions are provided by Hwang et al. The Borda count method is a kind of social selection function. In the multi-class recognition problem, it is advantageous to consider more than the top choice by each recognizer. Thus, a ranking-based association method is more useful than a multiple voting method. Recognition is a graphical structure that expresses casual reasoning and is used to solve causal decision making problems. It is necessary to construct group recognition from individual recognition for a group of multiple recognizers when the recognizer expresses its knowledge from the viewpoint of awareness consisting of causal objects (concepts) and causal oriented arcs (causal relations). Constructing group awareness corresponds to combining multiple decisions of individual recognizers. According to the way of expressing casualities, Kosko proposed a way to express causal relations in fuzzy terms and combine them using fuzzy operators through inferential links of awareness. Zhang et al. Proposed a NPN (Negative-Positive-Negative) logic that can directly express negative causal relationships in recognition. This NPN logic and NPN relationship replace the fuzzy logic with NPN logic. They also proposed a combined function to combine multiple causal relationships of NPN awareness.

Method of combining in a recognizer system

There are many ways to combine knowledge from multiple sources or evidence in the recognizer system domain. Duda et al. Proposed several approaches to solving hypotheses based on multiple uncertain evidence by probability. In this approach, there is a way in which multiple evidence is used to update the probability of a hypothesis when association evidence is uncertain and leading association evidence is inconsistent. To combine multiple evidences, they assume that evidence is conditional independent of each other for a given hypothesis.

Konolige proposed the formalization of Bayesian theories known by information theory for recognizer systems that must deal with uncertainty. The theoretical correct implementation of the Bayesian approach requires knowledge of the joint probability function for all events in the model. However, it is also difficult to evaluate the joint probability function for a small set of event subsets. Thus, some approximations are used in a computationally practical way to update probabilities.

However, conditional independent assumptions can not correctly characterize the required joint distribution. In other words, the independent assumption is incompatible with many mutually exclusive and non-leaving hypotheses. And often the recognizer more trusts in assessing the probability of a hypothesized condition on multiple evidence events rather than a single evidence event. One of his ideas uses the principle of minimal information on the integrated distribution. After defining entropy and information measurements, we choose a distribution with minimal information measurement for all possible distributions as a joint distribution. It also links the minimum information principle and the independent concept by the product expansion axiom that describes the distribution by a set of subdistributions based on the maximum entropy. To combine multiple evidences, use the product extensions computed for joint probabilities of evidence events.

Discrete probability distribution approximation method

For most information system designs, the main problem is to evaluate the underlying n-dimensional probability distribution from finite sample numbers and to store the distribution in a limited amount of machine memory. The allowable equipment complexity and limitations on available samples require the distribution to be approximated with the use of simplified assumptions such as normalization or statistical independence of random variables as consideration.

Lewis considers the approximation problem of the nth order binary distribution as the product of several component distributions with low order distribution using the expansion concept by Hartmanis. Under suitably limited conditions, the product approximation has shown that it has minimal information properties. This approximation method is based on an information measure called relative entropy or similarity for the similarity of the two distributions and on a maximum entropy basis. Since the information in the actual distribution is the same for all approximations by the definition of the similarity measurement criterion, the best approximation is an approximation in which the sum of information is the largest in the approximate distribution. Thus, more than one proposed approximation can be compared, and the best approximation is selected without knowledge of the actual distribution above the given distribution by approximation. That is, a comparison process comprising an approximate selection including a maximum correlation amount. However, the problem of selecting a set of component distributions of a given complexity to construct the best approximation remains unresolved.

Brown has presented an iterative method that provides an optimal approximation to a joint probability distribution with a set of binary variables given a joint probability distribution of arbitrary variable subsets. Each step of the iterative method provides an improved approximation, and the procedure converges to provide an approximation that is a minimum information extension to the employed component distribution.

To solve the unresolved problem of Lewis, Chow studied how to approximate the n-th order distribution by multiplying the (n-1) second-order component distributions. A method for optimal approximation of the nth order discrete probability distribution by the product of the second order distribution or the distribution of the first order tree dependency has been proposed. To obtain the optimal set of (n-1) first-order dependencies among n variables, a procedure to derive the minimum difference approximation from the information was derived using the similarity. The optimal procedure is based on Kruskal's Maximum Weight Spanning Tree (MWST) algorithm and maximizing mutual information between two variables. Minimizing the similarity is equivalent to maximizing the overall weight of the primary dependency tree.

Wong and Liu presented a decision-based approach to recognizing discrete data. The distribution evaluation employed in this approach is modified from the dependency tree procedure based on a characteristic independent chi-square test. Wang and Wong addressed the high-order probability distribution in the recognition of discrete data and approximated the distribution by the product of the low-order component distribution, based on the error probability minimum-maxax recognition system. Therefore, we defined the measurement criteria of Classes-Patterns (CP) used to approximate using Bayes error minimization. The approach proposed by Chow and Liu is compared with Wang and Wong's approach.

Malvestuto has presented a heuristic learning procedure to approximate the n-dimensional discrete probability distribution with a decomposable model of a given complexity. The resolvable model is a special interaction model with a number of desirable characteristics; The decomposable model may be represented by a Markov trust network; The approximation generated by the decomposable model has a product form. The dependency graph associated with the decomposable model of rank 2 results in a tree called the primary dependency tree. However, to obtain an optimal approximation, all resolvable models with rank k must be evaluated.

As described above, in the determiner of the recognizer, recognizer, machine, and the like that selects one of the predefined decision candidates, existing methods of combining a plurality of decisions include a voting technique, a Bayesian technique based on an independent assumption, There is a behavior-knowledge space (BKS) technique that is not based on independent assumptions.

The independent voting method or the Bayesian method has a problem in that when a dependent decision is included, the coupling relation of a plurality of decisions is biased and the performance is deteriorated due to this.

However, the calculation for combining multiple crystals is simple, and the amount required for this calculation is also advantageous.

On the other hand, the BKS technique, which is not based on the independent assumption, has an advantage of not requiring an independent assumption, whereas a geometric storage amount is required in order to maintain a high probability distribution composed of a plurality of decisions to be combined.

In addition, there is a disadvantage in that a combined result can not be made for a pattern of a large number of crystals which are not seen, and the rejection rate increases.

For example, assuming that there are L decision candidates in the process of combining K decisions, the complexity of the Bayesian method based on independent assumptions is , And the complexity of the BKS technique is to be.

In addition, a method of approximating a higher-order probability distribution has already been proposed, but a method of approximating a higher-order probability distribution based on a higher-order higher order dependency relationship has not yet been proposed Be absent

Therefore, if one determiner makes decisions that are dependent on the determinations of two or more different determinants, there is no way to approximate the probability distribution taking into account this.

The present invention has been made to solve the above-mentioned problems, If And to provide an optimal product approximation method of a probability distribution due to a differential dependency.

Another object of the present invention is to provide And to provide a method for obtaining a constitutive distribution term in an optimal product approximation of a probability distribution by a differential dependence.

It is still another object of the present invention to provide a method of combining multiple decisions by dependency.

Brief Description of the Drawings Fig. 1 is a diagram showing three steps of a recognition result. Fig.

2 is a block diagram of a general multiple recognizer system;

FIG. 3 is a diagram illustrating a method of minimizing the degree of similarity based on a primary dependency relationship according to the present invention. FIG.

FIG. 4 is a diagram illustrating a method of minimizing the degree of similarity based on the secondary dependency relationship according to the present invention. FIG.

5 is a diagram illustrating the calculated average mutual information value.

Figure 6 is a diagram depicting a dependency tree by a primary dependency relationship;

Fig. 7 is a diagram showing the sum of the product approximation result and the average mutual information value according to the second dependency relation; Fig.

8 is a diagram showing a first approximation by minimizing a Bayes error rate.

Brief description of the symbols used in the description of the invention

K determinator set:

Set of L decision candidates:

input :

Degree of dependency:

In order to achieve the first object, The optimal product approximation method of the probability distribution due to the differential dependence,

K set of determiners ,

A set of L decision candidates ,

Input ,

The degree of dependency is ,

If it is,

Probability of force terrier end

= when,

= Lt; / RTI &

end Lt; / RTI >

If it is,

The And

= If the relationship is established,

The above-mentioned high-order probability distribution To In a method of optimally approximating a product of a low-order probability distribution by a differential dependence,

Obtaining a primary dependency relationship;

Obtaining a conditionally independent assumption;

Obtaining a secondary dependency relationship;

Obtaining a conditional primary dependency;

.

.

.

Obtaining a differential dependency; And

Conditional And a step of obtaining a differential dependence relationship.

In order to achieve the above-mentioned second object, The method of obtaining the constituent distribution term at the time of the optimal product approximation of the probability distribution by the differential dependence is as follows:

K set of determiners ,

A set of L decision candidates ,

Input ,

The degree of dependency is ,

,

Probability of force terrier end

= when,

= Lt; / RTI &

end Lt; / RTI >

when,

The And

= If the relationship is established,

A method for finding a constituent distribution term at an optimal product approximation of a probability distribution by a differential dependence,

Actual probability distribution , An approximate probability distribution when,

Defined as

Minimize the measure of closeness Quot;

for ㉠ do / * Primary dependency * /

/ * Here, the above is the first dependency relation If

And

Given as * /

for ㉡ do / * Secondary dependency * /

/ * Where the above is the second dependency relation If

And

Given as * /

........ ........

for ㉢ do / * ( -1) Dependency relation * /

/ * At this time, -1) As a dependency relation 0 i ( -1) (j), ..., i1 (j) j

And

Given as * /

while (㉣) do / * Car dependency relationship * /

/ * ≪ / RTI > 0 as a car dependency i (j), ..., i1 (j) j

And

Given as * /

.......

end

end

.......

end

end

, One primary dependency, one secondary dependency, ..., one ( -1) dependency, (K- )doggy And the dependency relation -1) nested for loops and one while loop.

According to another aspect of the present invention,

K set of determiners ,

A set of L decision candidates ,

Input ,

The degree of dependency is ,

When there is a relationship, A method for combining a decision method by performing a method of obtaining an optimal product approximation of a probability distribution due to a differential dependence and a constituent distribution term for this with a normal Bayesian decision method,

Referring to the training sample data, A first step of obtaining an optimal product approximate set of a probability distribution by a differential dependence; And

A second step of probabilistically combining the determinations of the plurality of determinants by applying the approximate set obtained in the above step to the combination formula with the Bayesian method

And is characterized by considering dependencies between determinants that make multiple decisions without requiring an independent assumption.

Now, the operation principle and preferred embodiments of the present invention will be described in detail.

In the present invention, only the parallel combination of multiple determinants is considered for the same input, assuming that the decision of the recognizer is at the highest class or class ranking. The multiple recognizer system is defined as a monolith of a determinator set and a decision-combining method in parallel as shown in Fig. In the present invention, some societal selection functions are applied to a multiple determinator combination that calculates the rank of a class as a recognition result. Further, the present invention is an intermediate method among the above-mentioned conventional methods, and is a new method considering the dependency to overcome the shortcomings of the conventional method and take merits.

The method of the present invention is based on a dependency-directed approximation method that optimally approximates a high-order probability distribution as a product of a (k + 1) -order distribution, and the optimal approximation distribution is combined with a Bayesian method It is based on the probabilistic combining method to be applied.

Although the method provided in the present invention has more complex computations and larger storage than an independent home based approach, this method no longer requires an independent assumption and has less computation and less storage capacity than the BKS method. In addition, this method reduces the high recognition rejection rate of the BKS method by the product approximation.

In the method provided by the present invention, the dependency-directed approximation method uses the measure of closeness proposed by Lewis above to calculate the product of the (k + 1) It is a method to optimally approximate the distribution.

Here, it is assumed that the dependency relationship is determined stochastically by observing only the determiner's determinations obtained from the training samples.

Since there is no way to optimally approximate a higher order probability distribution in the case of a higher order dependency (i.e., k 1), a method involving an algorithm for higher order dependency is systematically designed in the present invention. Therefore, this approximate distribution is applied to the Bayesian method and the binding formula is derived by approximate distribution and Bayesian theorem.

Problem definition

To formally express the combination of multiple identifiers, some notation is defined using probability theory and Bayesian methods, and a multiple determinator join process is formulated as follows: When x is given in parallel to _K determiners (e.g., C ₁ , C ₂ , ..., C _K ), the K-order decision vector D = C ₁ (x) = M ₁ , C ₂ M ₂ , ..., C _K (x) = M _K are observed,

M _i (1 i L) is one of L determinations or classes by M = {M ₁ , M ₂ , ..., M _L } at the extraction level, where the subscript i is independent of i of the i th recognizer (C _i ).

The main task of combining multiple crystals in a stochastic way

To determine the class m ^* that maximizes the posterior probability P ^* .

That is, the (K + 1) -order probability distribution

_{P (m, C 1 (x} ) = M 1, C 2 (x) = M 2, ..., C K (x) = M K)

Should be estimated from the training samples, where m is the class variable of the input x.

This is referred to as a (K + 1) -order probability distribution in the present invention.

The present invention provides a new way of calculating the higher order probabilities for multiple decision bonds without independent assumptions to achieve the task. However, the computation and storage of (K + 1) -order probability distributions is geometrically complex and unmanageable (for example, .

Therefore, an approximate approximation method is required to approximate the (K + 1) -order probability distribution by the product of the (k + 1) -order distribution considering k-order independence at 1kK.

In other words, the problem is to identify the optimal product of the higher-order probability distributions and probabilistically combine multiple decisions using the Bayesian method without independent assumptions.

A dependency-based methodology is proposed by considering the dependencies between recognizers and extending Chow and Liu's research to higher-order dependencies. This dependency-based methodology takes a weaker assumption than the independent assumption based on the combining method, and requires less recognition rejection rate, computational complexity (eg, And less storage capacity.

In order to demonstrate the superior performance of the dependency-based methodology, the present invention employs a plurality of recognizers, collected from the KAIST (Korea Advanced Institute of Science and Technology) and the standardized CENPARMI database of Concordia University of Canada, A recognition experiment was conducted for English characters.

Various sets of multiple recognizers were performed in each application area and evaluated by various decision methods. That is, in this methodology, dependency-based decision combining methods are compared to conventional methods such as the multiple voting method, the Borda count method, the BKS method, and the Bayesian method based on independent assumptions.

The present invention as described above is largely composed of three parts.

First, it presents the rationale for computing the optimal product approximation set based on the dependence relation of the specified order by referring to the experimental sample data.

Second, it is an algorithm to obtain the actual optimal product approximation set based on this basis.

Third, the Bayesian method is applied from the computed optimal product approximation set to substantially concatenate the decisions of a plurality of determinants based on the dependency relation.

The part that presents the above general evidence shows how to obtain the optimal product approximation distribution based on the higher order dependence by using similarity and mutual information definition.

The algorithm part actually obtains a product approximation set according to the above grounds, and the obtained product approximation set is utilized in the Bayesian method to obtain a plurality of decisions by a probabilistic method.

The present invention operates as follows.

First, we refer to the training sample data and calculate the optimal product approximation set based on the specified order of dependence based on the rationale and algorithm.

Second, the calculated optimal product approximation set is combined with the decision of the multiple determinator practically by applying the Bayesian method.

In particular, the algorithm for computing the optimal product approximation set is operated such that the higher order probability distributions are approximated by the product of the lower order distributions while satisfying the stochastic properties.

To this end, the constraint term is used in the product approximation formula and applied to obtain the possible higher order dependency set.

This will be described in detail as follows.

C = {C ₁ , C ₂ , ..., C _K }: a set of determiners (recognizers)

M = {M ₁ , M ₂ , ..., M _L }: a set of decision candidates,

x: input,

And M _i M Lt; / RTI >

Let C _j (x) = M _j be C _j , where 1 j (K + 1)),

here, Is approximated as follows.

Here, 0 i (j) j,

: Primary dependency,

: Conditional independent families,

And Here, 0 i2 (j), i1 (j) j,: secondary dependency,

And Here, 0 i2 (j) j,: conditional primary dependency,

.

.

.

And Here, 0 ik (j), ..., i1 (j) j, k-order dependency,

And

here, And conditional k-th dependency.

In approximating the higher-order probability distribution (P) to the product of the lower-order probability distribution as described above, the approximate distribution (P _a ) (K + 1) must be determined from the permutation of 1, ..., (K + 1) so that the difference from the actual distribution becomes small.

And we use the measure of Closeness proposed by Lewis as a measure to calculate the difference between the actual probability distribution and the approximate distribution.

The rationale for determining the optimal approximate distribution based on the second order dependency by applying this similarity is as follows.

In the following equation, M stands for Averaged Mutual Information or simply mutual information.

In order to simplify the expansion process, subscript n is omitted in the approximate distribution definition equation.

Here, if C is (C ₁ , ..., C _{K + 1} )

I (P (C), P _a (C)) =

From the above equation to minimize I (P (C), P a (C)) can be determined the best approximation distribution.

However, this minimization is equivalent to maximizing M (C _j ; C _{i2 (j)} , C _{i1 (j)} ).

Because the remaining terms belong to the constant. Using the above rationale, the algorithm for finding the optimal approximate distribution based on the second dependence relation is as follows.

In this algorithm, mutual information values are used as weights to obtain the optimal product approximation set. This is explained as follows.

Algorithm to find the optimal product approximation distribution based on the second dependence relation

Input: s sample data S ¹ , S ^{1 ...} , S ^s .

Output: Optimal product approximation set based on second-order dependence by similarity.

Way :

1. Calculate the secondary and tertiary distributions from the training sample data.

2. Compute the weights M (C _j ; C _{i (j)} ) and M (C _j ; C _{i2 (j)} and C _{i1 (j)} ) for all 2 pairs and 3 pairs of data obtained from the sample data.

3. Compute the maximum weighted sum of the first and second dependencies and obtain the associated optimal product approximation set.

max_total_weight = 0;

for n = 1 to Number of primary dependencies do

total_weight = 0;

Select one of the primary dependencies as a constraint;

total_weight = weight of selected dependency;

while (number of unselected determinants 0) do

Selects one of the non-selected determinants;

Selects one of the possible secondary dependencies associated with the selected determinator;

total_weight = weight of selected secondary dependency;

end

max_total_weight = MAX (max_total_weight, total_weight)

stores max_total_weight and its associated set of primary and secondary dependencies;

end

The maximum weighted sum max_total_weight and the associated primary and secondary dependency sets are obtained.

4. Output the above result as an optimal product approximation set.

Based on the algorithm for finding the optimal product approximation set based on the above second dependency relation, the optimal product approximation set based on the k-th dependence relation has one primary dependency, one secondary dependency, ..., one ( k-1) and (Kk) k-order dependency relations.

For example, the algorithm for finding the optimal product approximation set based on the k-th dependence relation can be composed of (k-1) nested for loops and one while loop.

for do / * Primary dependency * /

for do / * Secondary dependency * /

........ ........

for do / * (k-1)

while () do / * k dependency * /

.......

end

end

.......

end

end

In particular, the algorithm for computing the optimal product approximation set should be operated such that the higher order probability distribution approximates the product of the lower order distributions while satisfying the stochastic properties. To this end, the constraint term is used in the product approximation formula and applied to obtain the possible higher order dependency set.

The general reasoning section shows how to obtain the optimal product approximation distribution based on the high order dependence using the similarity and mutual information definition.

The algorithm part is to actually obtain the product approximation set according to the above grounds, and the obtained product approximation set is used in the Bayesian method, so that multiple decisions can be obtained by the probabilistic method.

Dependency-based methodology

The dependency-based methodology is outlined below. First, we discuss the theoretical background of dependency-based methodology.

A dependency-based methodology for dependency-based or multiple-decision combining is described. Basically, this methodology consists of two sequential steps: a stochastic combination using dependency-directed approximation and Bayesian equations. Dependency-Oriented Approximation 1 k K, the optimal product approximation of the (K + 1) -order probability distribution by k-order dependence is treated. And, stochastic combining using Bayesian equations applies the optimal product approximation identified by dependency-directed approximation to Bayesian equations for multiple crystal combining.

Since the recognizer is statistically dependent on other identifiers, it is not desirable to assume that the identifiers are performed independently. Therefore, in this view, consideration of the recognition period dependence in the multiple recognizer combination is desirable.

First, the background on Bayesian formulas is introduced in the next chapter for the use of probabilistic terms, and the concerns about dependency are explained. The advantages and disadvantages of dependency-oriented methodologies are also explained in terms of complexity analysis.

Background on the Bayesian Method

For the Bayesian method of recognizing a given input pattern x, some notation as defined below is needed.

The set of L individual determinations or classes is denoted as M = {M ₁ , M ₂ , ..., M _L }, and the set of K recognizers is denoted as C = {C ₁ , .., C _K }. In the Bayesian method, the output of each recognizer is treated as taking into account the error of the recognizer itself. The error of each recognizer is described by its own confusion matrix. Thus, the use of the Bayesian method requires a learning step in the confusion matrix for which the recognition period has been established. Elements of confusion matrix Is the number of samples of the class (M _i ) to which the label class (M _j ) is assigned by the recognizer (C _k ). The total number of samples of class M _i is , And the total number of samples to which the label class M _j is assigned by the recognizer C _k is to be.

= , i = 1, ..., L.

= , j = 1, ..., L + 1.

For an event (i. E., Decision) C _k (x) = M _j , the true value of this event is x M _i , i = 1, ..., L is true.

For the K recognizers, there are K second-order confusion matrices in the learning phase. When the same input pattern x is to be understood by the reader the K, the K event _{C K (x) = M j} k = 1, ..., K, j = 1, ..., L are generated. A problem is how to combine K individual events to select a winner class based on the group consensus. This problem is formulated by the following equation (3) which determines the winner class m ^* to maximize the posterior probability P ^* .

In the above Equation 3, Subscript Brackets Subscript . In order to estimate the (K + 1) -order probability, the collection of K determinations obtained from the training samples should be stored and stored in storage space. However, the calculation and storage of the (K + 1) -order probability distribution is geometrically complex and unmanageable. Therefore, (conditional) independent assumptions are applied for simple calculation of the (K + 1) -order probability distribution. However, without independence hypothesis, we will approximate the (K + 1) -th order distribution with a lower-order component distribution by dependence.

Background on independence and product approximation

Under the assumption that the recognizers perform independently of each other, Xu proposed a Bayesian method. In this method, if a non-member recognizer with a high dependency on the member recognizer is added to the member recognizer set, the join decision is biased towards the dependency recognizer. Furthermore, if the dependency recognizer has an error recognition result, the performance deteriorates. This is an important weakness of the independent family based method.

Dependency refers to the ability of a recognizer to predict one recognizer. The differential dependence is defined as the predictive performance of the K recognizer determinations for a single recognizer. These dependencies are statistically determined by the calculation of the average mutual information, the statistical relationship, or the interdependent redundancy criteria of the event covering method.

Since the average mutual information is derived from the similarity and this easily expands into a higher order dependency, the (average) mutual information is selected on a dependency basis. (Average) Mutual information is defined as a quantitative criterion that indicates the occurrence of a particular event (ie, decision) as to the likelihood of some alternative.

The dependence provides a theoretical basis for approximating a higher-order probability distribution as a product of lower-order probability distributions. In the present invention, it is assumed that the recognition period dependence is probabilistically measured by the determination of K recognizers from the training data set, and this dependency contributes to the approximation of the high probability distribution. In the higher-order probability distribution approximation by k-order dependence, two criterion can be applied to the higher-order probability distribution. One criterion is to minimize the degree of similarity.

This criterion yields one product approximation for each class. The similarity is mathematically defined as follows.

Another criterion is to minimize the Bayes error rate. Using the Bayes error rate, one product approximation is generated for all classes. Minimizing the Bayes error rate P _{e is} to minimize the conditional entropy of the decision variable when the recognizer is given as

,

The similarity is emphasized as the optimal approximation criterion because the optimal approximation criterion has better performance than the Bayes error rate. Without knowledge of the actual distribution, the higher order probability distribution is approximated optimally by the dependency relation, and the multiple decision is combined with the optimal approximation distribution using the Bayesian method in a probabilistic manner. The proposed method is applied to handwritten number and alphabet recognition in experiments.

Advantages and Disadvantages of the Method

It is assumed that there are K recognizers and L decisions.

(K + 1) -order probability distribution and calculates the (K + 1) -order probability from this distribution. As previously described, storing and estimating a (K + 1) -order probability distribution without approximation is geometrically complex and unmanageable from a theoretical analysis.

If the (K + 1) -order probability distribution accepts a conditional independence hypothesis, then the storage complexity of this distribution is to be. In other words,

.

This independent assumption-based approximation has simple computation and low storage capacity. However, independent homes are often overused for practical applications.

On the other hand, the BKS method should maintain a (K + 1) -order probability distribution without approximation. Thus, the storage complexity is And is geometrically complex.

Although the BKS method does not require an independent assumption, the disadvantage is that the BKS method is geometrically complex even for a small K. Moreover, since the BKS method can not make a decision if the joint K decision is observed to be inconsistent with the elemental component of the frequency table, a high rejection rate may occur.

As an intermediate method for both extreme methods, the dependency-based method approximates the (K + 1) -order probability distribution with low-order component distributions and combines multiple determinations without independent assumptions. Considering the k-th dependency relationship (where 1 k K), the storage complexity of this method is And has complexity between the independent home based method and the BKS method. In addition, this method can reduce meat angular rate by product approximation.

Dependency Oriented Optimal Approximation

When the high-order probability distribution is approximated, a criterion for measuring the degree of similarity between the approximation and the actual distribution is required. This criterion, which is subordinate to the information theory model, was developed by Lewis. This is called a measure of closeness or measure of divergence or relative entropy. The similarity is defined as the difference between the information contained in the approximate distribution and the information contained in the actual distribution. In the K-dimensional vector determined D simply represent a C _j (x) = M _j with C _j.

Without the approximation, the (K + 1) -order probability distribution is transformed into a product of lower-order distributions using the following chain rule.

=

Chow and Liu attempted to solve the optimization problem proposed by Lewis and used Kruskal's Maximum Weight Spanning Tree (MWST) algorithm to calculate the product of the first-order tree dependence or the (n-1) We approximate the n-order binary distribution of n variables best. Since they focus only on the distribution of the first-order tree dependencies, these methods are not a suitable way to consider higher order dependencies. However, k K, we propose a dependency-oriented method to approximate the (K + 1) -order probability distribution with the optimal product of given k-order dependence. The optimal product is then used to couple multiple crystals using the Bayesian method. It is assumed that this dependency is probabilistically determined only by observing the decision of the recognizer obtained from the sample. And the dependency order can be increased for better approximation.

Using the similarity, the optimal approximation of the probability distribution is to minimize the difference between the actual distribution and the approximate distribution. The details of the dependency-directed optimal approximation are discussed by the primary or secondary and k-th dependencies.

First-order dependency approximation

When the primary dependency is considered, the approximate distribution is defined as follows from the secondary distribution point of view.

,

Where 0 i (j) j.

therefore The Where n ₁ , n ₂ , ..., n _K , n _{K + 1} are permutations of integers 1, 2, ..., K, K + 1. By definition, P ( | C ₀ ) is P ( )to be.

For simplicity, let n _j be denoted by j, and let C ₁ , C ₂ , ..., C _K , and C _{K + 1 be} the (K + 1)

When the primary dependency-based method is applied by Chow and Liu to an approximation for K multiple decision bonds, the (K + 1) -order probability distribution P of the vector C is given by the similarity between P and P _a Is approximated by P _a by the optimal product of the secondary distribution or the first order distribution determined by minimizing.

As shown in Figure 3, I (P (C) , P a (C)) is to minimize both because the remaining term of the constant sum of the average mutual information M (C _j ; C _{i (j)} ).

The next step is then to identify the primary dependency optimal set from the (K + 1) ^{(k + 1) -2L} spanning tree in graph G with (K + 1) For all node pairs in graph G, after calculating the average mutual information, assigning this information to the edge weights, applying the MWST algorithm to the graph G, and finally selecting the best dependency tree. From the selected optimal dependency tree, And conditional permutation of this permutation Can be determined.

On the other hand, if Equation (9) All Assuming vortices, that is, C _j , are assumed to be conditionally independent of each other for a given C _{K + 1} , then the approximate distribution is defined in terms of the second order distribution as:

Here, C _j is forcibly conditioned only for C _{K + 1} .

This approximation is known as a conditional independent assumption that can be defined as a specific case of a first-order dependency approximation. The details are as follows.

Secondary dependency approximation

When the secondary dependency is considered, the approximate distribution is defined as follows from the third distribution point of view.

,

Where 0 i2 (j), i1 (j) j.

therefore And The Lt; RTI ID = 0.0 > The constants 1, 2, ... , K, K + 1. By definition, P ( | C ₀ , ) Is P ( , ).

Also, as shown in Fig. 4, the degree of similarity can be used to optimally approximate the distribution as the product of the second dependency relation.

In FIG. 4, I (P (C) , P a (C)) is to minimize the sum of the average mutual information that satisfy a given constraint so all remaining term is a constant M (C _j ; , ) Is maximized. Thus, the next step is to identify a second-order dependency-based optimal product approximation set from all allowable product sets. The process for identifying the optimal product approximation set based on the second dependence relation is explained as follows.

Algorithm to find the optimal product approximation distribution based on the second dependence relation

Input: s sample data S ¹ , S ^{1 ...} , S ^s .

Output: Optimal product approximation set based on second-order dependence by similarity.

Way :

1. Calculate the secondary and tertiary distributions from the training sample data.

2. Compute the weights M (C _j ; C _{i (j)} ) and M (C _j ; C _{i2 (j)} and C _{i1 (j)} ) for all 2 pairs and 3 pairs of data obtained from the sample data.

3. Compute the maximum weighted sum of the first and second dependencies and obtain the associated optimal product approximation set.

max_total_weight = 0;

for n = 1 to Number of primary dependencies do

total_weight = 0;

Select one of the primary dependencies as a constraint;

total_weight = weight of selected dependency;

while (number of unselected determinants 0) do

Selects one of the non-selected determinants;

Selects one of the possible secondary dependencies associated with the selected determinator;

total_weight = weight of selected secondary dependency;

end

max_total_weight = MAX (max_total_weight, total_weight)

stores max_total_weight and its associated set of primary and secondary dependencies;

end

The maximum weighted sum max_total_weight and the associated primary and secondary dependency sets are obtained.

4. Output the above result as an optimal product approximation set.

On the other hand, if Equation (11) All Is the same for Is given for C _{K + 1} , The approximate distribution is defined as follows from the third-order distribution point of view.

Here, 0 i2 (j) j. And C _{< K + 1} & gt _; , Where n ₁ , n ₂ , ... , n _K are permutations of integers 1, 2, ..., K. By definition, P ( | C ₀ , C _{K + 1} ) is P ( , C _{K + 1} ).

This approximation is known as a conditional primary dependency hypothesis that can be defined as a particular case of second-order dependency approximation. The details are as follows.

Using the new dependency-oriented approximation, the degree of dependency to be considered is computed in a systematic way for higher order probability distribution approximations It can be easily extended to the car. The primary dependency product set consists of one primary dependency, one secondary dependency, ..., one ( -1) and (K- )dog It depends on the interdependence.

,

Where 0 ... , , j.

The algorithm for the optimal set of the dependency relations can be applied to the proposed algorithm for the second dependency, And a recursive routine for finding one of the secondary dependencies -1) It can be devised by adding a nested for loop that deals with the dependency.

Product Approximation by Secondary Dependence

3 to 5 recognizer E1, E2, average cross for calculating primary and secondary dependency from quaternary probability distribution consisting of three decision variables in E3 C _1, C _2, C _3, and hypotheses decision variable C ₄ Information values are shown.

Applying the conditional independence hypothesis to the fourth-order probability distribution, we can obtain the results shown in the first line of the following equation, and apply the dependency tree method to the same fourth-order probability distribution as shown in FIG. The result can be obtained.

M = 6.717977

_{P a (C 1, C 2} , C 3, C 4) = P (C 1, C 4) P (C 3 | C 4) P (C 2 | C 43): M = 6.728495 ^*

In the above equation, * denotes the optimal result.

In order to consider more available information, the proposed algorithm for the second order dependency is applied to the same actual distribution as shown in FIG. 7 to obtain the following results.

From the results of Figure 7, as the optimum product set of the second _{dependency, P (C 4, C 2} ) P (C 1 | C 4, C 2) P (C 3 | C 4, C 2), P ( _{C 4, C 2) P (} C 3 | C 4, C 2) P (C 1 | C 4, C 2), P (C 4, C 3) P (C 2 | C 4, C 3) P ( _{C 1 | C 4, C 2} ), P (C 1, C 2) P (C 4 | C 1, C 2) P (C 3 | C 4, C 2), P (C 2, C 3) P (C ₄ | C ₂ , C ₃ ) P (C ₁ | C ₄ , C ₂ ). This is because the sets of optimal products have the largest weight of the secondary average mutual information.

First-order dependency approximation by minimizing Bayes error rate

Now, as another criterion, we apply a criterion that approximates Bayes error rate minimization to a higher order probability distribution by first order dependence. For approximation, we define decision-classifiers (DC) mutual information as shown above.

,

Here, m is a decision variable and C is a set of decision variables of the recognizer. As mentioned above, minimizing the Bayes error rate is to maximize the DC mutual information.

When the first order dependence is considered in the product approximation by minimizing the Bayes error rate, the approximate distribution is defined as follows in terms of the third order distribution.

Here, 0 i2 (j) j. And m is , Where n ₁ , n ₂ , ... , n _K are integers 1, 2, ... , K is a permutation. By definition, P ( | C ₀ , m) is P ( , m).

Furthermore, in addition to the expression of P _a , the probability distribution of the recognizer decision variables is defined as follows from the secondary distribution point of view.

Here, 0 i2 (j) j. And m is Where n ₁ , n ₂ , ..., n _K are permutations of integers 1, 2, ..., K.

If the probability distributions P _a (m, C) and P _a (C) are inserted into Equation 15 instead of P (m, C) and P (C) respectively, the following equation is obtained. 8, maximizing M (m; C) is the sum of average mutual information satisfying a given constraint because all the other terms are constant .

On the other hand, the approximation for the secondary dependency is as follows.

Applied to the Bayesian method

In order to couple multiple crystals in a probabilistic manner, The approximate distribution obtained from the optimal product set of the dependency relation is applied to the Bayesian method. Combination formulas are derived using Bayesian theorem and approximate distribution.

Application of primary dependency to Bayesian methods

For each class M _i , we use a first order dependency and Bayesian method, and one class term M _i Assuming that M is expressed as C _{K + 1} (x) = M _{K + 1} , the following formula can be obtained.

Where η is .

Where n ₁ , n ₂ , ..., n _K , n _{K + 1} are permutations of integers 1, 2, ..., K, K + 1. Depending on the Bel (M _{K + 1} ) value calculated by combining multiple decisions, the maximized posterior probability is selected. Then, the combined crystal is determined as the class m according to the determination rule E (D) given as follows. if,

Bel (M _i ), E (D) = m,

Otherwise, E (D) = L + 1 (implies rejection).

Application of Secondary Dependence to the Bayesian Method

If an optimal set of secondary dependencies is used for higher order dependencies, then the join method is expressed as:

Here, .

Where n ₁ , n ₂ , ... , n _K , n _{K + 1} are integers 1, 2, ... , K, K + 1. Further, the above decision rule E (D) is applied to a multiple decision combination on the basis of a secondary dependency relation.

Hereinafter, embodiments according to the present invention will be described.

Experiment

The first experiment was carried out with a set of standardized data in a totally amorphous offline handwritten number distributed at the CENPARMI Institute in Concordia, Canada. This data set is divided into two different sets of training data (A, B) and test data sets (T). Each dataset contains 200 characters per class for 10 numbers.

The second experiment was conducted with amorphous online handwritten numbers and alphabet letters collected from KAIST. In particular, it is intended to show proof of the influence of the high dependency recognizer. From these experiments, we will compare the dependency-based method with other decision-combining methods and evaluate the performance of the proposed method.

Recognition of amorphous offline handwritten numbers

Experiments on the results of combining the recognition recognizers NN1, NN2, NN3, R1, R2 as well as recognition results of individual recognizers are discussed. The rejection recognition result of the recognizer And is excluded from identifying the optimal set of differential dependencies. The recognizers NN1, NN2, NN3 are neural network recognizers. The recognizers R1, R2 are rule-based recognizers. These recognizers were developed by kAIST and Chonbuk University. In particular, recognizers NN1 and NN2 were developed without rejection type results.

The performance of the individual recognizer on the test data set (T) and the recognition rejection rate are shown in Table 1.

[Table 1]

Individual recognizer performance for test set T

Here, Classifier is the recognizer, Recognition rate is the recognition rate, and Rejection rate is the recognition rejection rate.

Combination methods such as multiple voting method, Borda count method, BKS method, conditional independent home based Bayesian method Experiments using the proposed methods as the Bayesian method based on the dependency relation were performed on the test data set (T).

In the present invention, the first experiment was performed by combining three recognizers selected from five recognizers. There are ten possible combinations in the three selected recognizers.

As shown in Table 2, the best recognition rate was obtained for each combination case by a secondary dependency-based Bayesian method (i.e., a secondary dependency relationship) (except for the Borda count method because it is a ranking-level combining method).

[Table 2]

Combining performance of 3 recognizers

In the present invention, the second experiment was performed by combining the four recognizers selected from the five recognizers. There are five possible combinations in the four selected recognizers.

As shown in Table 3, the best recognition rate for each combination case was obtained by conditional primary dependency-based Bayesian methods (i.e., conditional primary dependency) (except for the Borda count method too). Interestingly, the performance of the BKS method was degraded by the combination of four recognizers. It can be deduced that the insufficient data in constructing the BKS table increases the rate of recognition rejection.

[Table 3]

Combining performance of 4 recognizers

In the present invention, the third experiment was performed by combining all five recognizers. As shown in Table 4, the best recognition rate was obtained by a conditional primary dependency based Bayesian method (i.e., conditional primary dependency). Performance by the BKS method has been degraded due to the increased rejection rate of recognition.

[Table 4]

Combining performance of 5 recognizers

From all the experimental results, The Bayesian method based on the dependency relation showed better performance than the conditionally independent Bayesian method and BKS method. In most cases, the recognition rate obtained by the second-order dependency approximation method including the conditional first-order dependency is higher than the recognition rate by the first-order dependency approximation method. The degraded recognition rate of the BKS method was due to the fact that the training data set was not representative and not enough. The proposed new method 1, it requires more storage than the conditionally independent home based Bayesian method, Integrating the product dependence relation by the Bayesian method contributes to the improvement of the performance of multiple classifier coupling without independent assumptions.

Of the 53 samples rejected by the BKS method in Table 2, 21 samples are correctly recognized by conditionally independent hypothesis-based Bayesian methods or primary dependency-based Bayesian methods. However, 34 samples are correctly recognized by conditional primary dependency-based Bayesian methods or secondary dependency-based Bayesian methods. These results We explain the reason why the Bayesian method based on the dependency relation shows good recognition rate. The best recognition rate was achieved mainly by the Borda count method. However, the Borda count method combines decisions at the rank level, while the dependency based Bayesian method combines the decision at the extraction level. Therefore, it is a desirable effect that the dependency-based Bayesian method occasionally shows the best recognition rate.

Amorphous on-line handwritten digit recognition

This experiment was carried out with a data set of amorphous online handwritten numbers collected from KAIST. The training data set contains 4088 characters written by 13 people, and the test data set contains 988 characters written by 10 people. The creator of the training data set does not participate in the test data set. The recognizers (ie, H1, H2, H3) were developed at KAIST using the Hidden Markov Model developed by related data sets and statistical modeling methods. The performance of the individual recognizers for the test data set is shown in Table 5 together with the recognition rejection rate.

[Table 5]

Performance of individual recognizers for test sets

We tested the test data set for comparison with the same previous method as the dependency-oriented approach. The rejection recognition result of the recognizer was included in determining the optimal product set of the differential dependence. The performance of the combined recognizer is shown in Table 6.

As shown in Table 6, the best recognition rate was obtained by the Bayesian method based on the second dependency relationship or the Bayesian method based on the conditional first dependency relationship.

Of the 26 samples rejected by the BKS method in Table 6, 12 samples are correctly recognized by the conditionally independent hypothesis-based Bayesian method and 13 samples are correctly recognized by the primary dependency-based Bayesian method. However, 15 samples are correctly recognized by conditional primary dependency-based Bayesian methods or secondary dependency-based Bayesian methods. In addition, The reason why the differential dependency based Bayesian method shows good recognition rate is explained.

[Table 6]

The performance of the join recognizer for a set of test numbers

From the experimental results shown in Table 6, it can be seen that by the proposed dependency- The Bayesian method based on the dependency relation shows better performance than the Bayesian method and BKS method based on conditional independent assumption. Recognition rejection rate The error rate was increased by the lack of recognition rejection criterion in the decision rule E (D), while it was reduced by the Bayesian method based on the dependency relation. therefore, The dependency-based Bayesian method was not good for evaluating performance using the performance model (10 * E + R) or (8 * E + 2 * R). Here, E represents the error rate and R represents the recognition rejection rate. Therefore, it is appropriate to apply the Bayesian method based on к-dependence to cases where only mandatory recognition performance is an important concern. Otherwise, the rejection criterion must be incorporated into decision rule E (D) to minimize the error rate.

The degraded recognition rate of the BKS method is large enough and is due to the lack of representative training data sets. On the other hand, The improved recognition rate of the Bayesian method based on the dependency relation is improved by correctly processing and recognizing the rejected samples using the product approximation.

The proposed new method 1, it requires more storage than the conditionally independent home based Bayesian method, We integrate the product approximation by the difference dependence into the Bayesian method and contribute to the improvement of the performance of the multiple recognizer combination without independent assumption.

Add dependency recognizer to multiple recognizer systems

In addition to the above, three recognizers H1, H2, and H3 have been developed and tested for amorphous on-line handwritten numbers and alphabet recognition as a whole. These recognizers are components of the Base system. In order to demonstrate the efficiency of the dependency-based method, several multiple recognizer systems have been built by adding a high dependency recognizer generated by faking one of the component recognizers of the base system to the base system.

For example, the H1 falsification system consists of the component recognizer and recognizer H1 and the homogeneous recognizer. In order to identify the optimal product set of the dependency relations, 4088 numbers created by 13 people, 3749 lowercase letters created by 19 people (lowercase), and 2464 uppercase letters were used as training data sets. The test data set consisted of 988 numbers written by 10 people, 1684 lowercase letters by 9 people, and 1169 uppercase letters. The creator of the training data set is different from the creator of the test data set in each coverage area. The rejection recognition rate of the recognizer And is excluded in the identification of the optimal product set of the differential dependence. That is, only valid recognition results are considered. The recognition rates of the first candidate of the constituent recognizer for the test data set are shown in Table 7.

[Table 7]

Recognition rate of recognizer for test data set

From the experimental results on numerical data in Table 8, The dependency-based join method shows better performance than the conditionally independent assumption-based Bayesian method and the BKS method. The best recognition rate for the English lowercase data in Table 9 was obtained by Bayesian method based on conditional primary dependency for all the multiple recognizer systems. The best recognition rate for the English capital letter data in Table 10 was obtained by Bayesian method based on conditional first order dependency for all the multiple recognizer systems except the H3 forged system.

[Table 8]

Recognition rate of multiple recognizer systems on numerical data

For the H3 counterfeit system, the best recognition rate was obtained by the Bayesian method based on the second dependence relation.

When higher order dependencies are considered, the recognition rate increases as shown in Table 8 through Table 10. However, the identification complexity for the optimal product set also increases. In the Bayesian Bayesian method based on the conditional independent assumption, the identification complexity for the optimal product set has the computational complexity O (1) regardless of the number of recognizers, since the number of recognizers is not required. In the case of a primary dependency, O (NlogN) is required as the identification complexity for the optimal product set, where N is the total number of edges of the graph G, to be.

Also, in the case of a second dependency relationship, O (NN) is required as the identification complexity for the optimal product set, where the first N is equal to the first dependency and the second N is the selected first dependency The total number of relationship-based allowable secondary dependencies, to be. Where C (d, 2) is a statistical combination function. Although it is complicated to consider higher order dependencies, it is desirable to consider the higher order dependence because the recognition rate can be increased and the computational complexity in identification is only performed once in the training phase. However, higher order dependency considerations do not always guarantee high performance.

[Table 9]

Recognition Rate of Multiple Recognition Systems in Lowercase English Data

In summary, the performance of the BKS method does not change even when a higher order dependency recognizer is added to the base system. In most cases, the recognition rate obtained by the higher-order dependency-based Bayesian method is higher than the recognition rate obtained by the lower-order dependency-based Bayesian method. The low recognition rate obtained by the Bayesian method based on the second dependency relation for lowercase and uppercase English data is sufficient and due to the lack of representative training data set.

[Table 10]

Recognition Rate of Multiple Recognition System in English Capitalization Data

Integrating the differential dependencies into the Bayesian method, especially when higher order dependencies are involved, improves the performance of multiple recognizer combinations. Conditionally independent home-based Bayesian methods Differences in recognition rates between differential dependency based Bayesian methods are generally statistically valid by t_ test at significance level 0.01. For this fact, an example of performing a t_ test in a base system to demonstrate whether the correctness improvement due to dependency considerations is statistically valid for English capitalization data is described below. Degradation due to addition of a higher order dependency recognizer can be an example of the above problem. It is therefore an important issue to apply the voting method and the social selection function to the multiple recognizer system in order to add the arbitrary recognizer to the multiple recognizers in a simple manner and to achieve good performance.

H ₀ : The mean accuracy of the second-order dependency-based Bayesian method is less than or equal to the mean accuracy of the conditionally independent assumption Bayesian method (ie, , And D is the recognition rate per author)

H _a : An alternative to H ₀ (ie, μ _D 0)

If n is the number of writers,

= 2.4396.

here,

= 0.8578,

= 1.11272.

Since T = 2.4396 t _0.025 = 2.306, H ₀ can be rejected as degrees of freedom 8. This means that the average accuracy of the second-order dependency-based Bayesian method improves to a belief interval of 97.5%.

In the present invention, the following conclusions are derived from the above-described experimental examples.

The recognizer performance for multiple crystal bonds is superior to that for individual identifiers.

The voting combination is similar to the social selection function. Their recognition performance is greatly degraded by the high dependency recognizer.

The recognition performance of the BKS method decreases as the number of recognizers increase, but the recognition performance does not change even though a high dependency recognizer is added.

All bonding methods using Bayesian methods outperform other bonding methods.

Integrating the differential dependence into the Bayesian method improves the performance of multiple recognizer combinations. Especially when high dependency recognizers are involved. Experimental results for capital letter data are supported by showing statistically valid t test results.

The recognition rate obtained by higher order dependency is generally higher than the recognition rate obtained by lower order dependence.

The best recognition rate obtained was obtained by using the Borda count method for the CENPARMI database on the KAIST database.

INDUSTRIAL APPLICABILITY The present invention is applicable to an application field in which a plurality of determiners are simultaneously performed in a determiner such as a recognizer, a recognizer, and a sensor machine for performing discrete crystal, and each of the determinations is obtained and a plurality of obtained crystals are combined in a stochastic manner.

For example, if you want to use multiple identifiers in the field of pattern recognition, combine the decisions of that recognizer, and want to combine the opinions or decisions of multiple identifiers in the field of group decision making, you can use multiple sensors and combine the results And to make a decision.

As the determiner, not only a character recognizer using a neural network but also a knowledge based recognizer or a feature based recognizer can be used.

In addition The optimal approximation method of the probability distribution by the dependency relation and the method of combining multiple decisions, that is, from the high- It can be applied to the field for obtaining the optimal approximate distribution by the product of the low-order probability distribution based on the differential dependence and to the field for combining multiple decisions based on the dependency relation using this approximate distribution.

In addition, the present invention can be applied to a case where a plurality of determiners such as an image recognition field, a DSS, a discrete decision maker, a sensor, a machine, etc. are performed and the determinations are combined. It will be apparent to those skilled in the art that the present invention is not limited to the above-described embodiments, but may be applied to other fields as long as the spirit of the present invention is maintained.

With the above arrangement, the present invention does not take an independent assumption, thereby avoiding the problems caused by independent assumptions.

Also, in order to approximate the high-order probability distribution by the product of the low-order probability distribution, it is possible to obtain an optimal product approximation distribution set based on the order dependence relationship while changing the dependency order.

Also, the storage complexity required is larger than that of the independent family, but it is smaller than that of the BKS technique. Because O (L ² ) O (L ^{k + 1} ) O (L ^{K + 1} ).

In addition, we extended the existing research results on the product approximation by suggesting a methodology for handling higher - order dependency, not just the first dependency relation.

In addition, it has been shown that combining the decisions of multiple determinants based on higher order dependence is superior in performance.

In the field of pattern recognition, the present inventors have excelled the present invention through an experiment in which a plurality of recognizers are used to combine the determinations.

Claims (21)

K set of determiners , A set of L decision candidates , Input , The degree of dependency is , If it is, Probability of force terrier end = when, = Lt; / RTI & end Lt; / RTI > If it is, The And = If the relationship is established, The above-mentioned high-order probability distribution To In a method of optimally approximating a product of a low-order probability distribution by a differential dependence, Obtaining a primary dependency relationship; Obtaining a conditionally independent assumption; Obtaining a secondary dependency relationship; Obtaining a conditional primary dependency; . . . Obtaining a differential dependency; And Conditional And a step of obtaining a difference dependence between the first and second states, Optimal Product Approximation Method of Probability Distribution by Differential Relations. 2. The method according to claim 1, wherein the actual probability distribution And approximate probability distribution The difference with Defined as The feature of being measured using the Measure of Closeness, Optimal Product Approximation Method of Probability Distribution by Differential Relations. 3. The method of claim 2, wherein when approximating an optimal product of a probability distribution by a second order dependency, , ≪ / RTI > Optimal Product Approximation Method of Probability Distribution by Differential Relations. 4. The method of claim 3, wherein an optimal product approximation of a probability distribution is obtained when the similarity is minimized. Optimal Product Approximation Method of Probability Distribution by Differential Relations. The method of claim 3, Lt; / RTI > is maximized, an optimal product approximation of the probability distribution is obtained. Optimal Product Approximation Method of Probability Distribution by Differential Relations. 2. The method of claim 1, If = And Lt; RTI ID = 0.0 > Optimal Product Approximation Method of Probability Distribution by Differential Relations. 7. The method of claim 6, = Lt; RTI ID = 0.0 > Optimal Product Approximation Method of Probability Distribution by Differential Relations. 8. The method of claim 7, If = And Lt; RTI ID = 0.0 > Optimal Product Approximation Method of Probability Distribution by Differential Relations. 9. The method of claim 8, If = And Lt; RTI ID = 0.0 > Optimal Product Approximation Method of Probability Distribution by Differential Relations. 10. The method of claim 9, The car dependence If = And Lt; RTI ID = 0.0 > Optimal Product Approximation Method of Probability Distribution by Differential Relations. 11. The method according to claim 10, The car dependence If = And Lt; RTI ID = 0.0 > Optimal Product Approximation Method of Probability Distribution by Differential Relations. K set of determiners , A set of L decision candidates , Input , The degree of dependency is , , Probability of force terrier end = when, = Lt; / RTI & end Lt; / RTI > when, The And = If the relationship is established, A method for finding a constituent distribution term at an optimal product approximation of a probability distribution by a differential dependence, Actual probability distribution , An approximate probability distribution when, Defined as Minimize the measure of closeness Quot; for ㉠ do / * Primary dependency * / / * Here, the above is the first dependency relation If And Given as * / for ㉡ do / * Secondary dependency * / / * Where the above is the second dependency relation If And Given as * / ........ ........ for ㉢ do / * ( -1) Dependency relation * / / * At this time, -1) As a dependency relation 0 i ( -1) (j), ..., i1 (j) j And Given as * / while (㉣) do / * Car dependency relationship * / / * ≪ / RTI > 0 as a car dependency i (j), ..., i1 (j) j And Given as * / ....... end end ....... end end , One primary dependency, one secondary dependency, ..., one ( -1) dependency, (K- )doggy And the dependency relation -1) nested for loops and one while loop, A method of finding the constituent distribution term at the time of the optimal product approximation of the probability distribution by the differential dependence. 13. The method of claim 12, = 2, the constituent distribution term at the optimal product approximation, Reading ^s sample data S ¹ , S ^{1, ...} , S ^s as input; Calculating second and third probability distributions from training sample data; _Calculating weight values M (C _j ; C _{i (j)} ) and M (C _j ; C _{i2 (j)} and C _{i1 (j)} ) for all 2 pairs and 3 pairs of data obtained from the sample data; Calculating a maximum weighted sum of the first and second dependency relations and obtaining an associated optimal product approximation set; And outputting the result obtained in the step as an optimal product approximation set. A method of finding the constituent distribution term at the time of the optimal product approximation of the probability distribution by the differential dependence. 14. The method of claim 13, wherein obtaining the associated optimal product approximation set comprises: max_total_weight = 0; for n = 1 to the number of primary dependencies do total_weight = 0; Selecting one of the primary dependencies as a constraint; total_weight = weight of selected dependency; while (number of unselected determinants 0) do Selecting one of the non-selected decision units; Selecting one of possible possible secondary dependencies associated with the selected determinator; total_weight = weight of selected secondary dependency; end max_total_weight = MAX (max_total_weight, total_weight) storing max_total_weight and related primary and secondary dependency sets; end Obtaining a maximum weighted sum max_total_weight and its associated primary and secondary dependency set, A method of finding the constituent distribution term at the time of the optimal product approximation of the probability distribution by the differential dependence. 14. The method of claim 13, And The And conditions for, and _{n 1, n 2, ...,} n K, n K + 1 is an integer 1,2, ..., K, the permutations of the K + 1, P ( | C ₀ , ) Is P ( | ), N _j is denoted by j, and C ₁ , C ₂ , ..., C _K , and C _{K + 1} are (K + 1) i2 (j), i1 (j) j, , ≪ / RTI > A method of finding the constituent distribution term at the time of the optimal product approximation of the probability distribution by the differential dependence. 14. The method of claim 13, And The And conditions for, and _{n 1, n 2, ...,} n K, n K + 1 is an integer 1,2, ..., K, the permutations of the K + 1, P ( | C ₀ , ) Is P ( | ), N _j is denoted by j, and C ₁ , C ₂ , ..., C _K , and C _{K + 1} are (K + 1) i2 (j) j, Gt; and < RTI ID = 0.0 > 1, < / RTI > A method of finding the constituent distribution term at the time of the optimal product approximation of the probability distribution by the differential dependence. K set of determiners , A set of L decision candidates , Input , The degree of dependency is , When there is a relationship, A method for combining a decision method by performing a method of obtaining an optimal product approximation of a probability distribution due to a differential dependence and a constituent distribution term for this with a normal Bayesian decision method, Referring to the training sample data, A first step of obtaining an optimal product approximate set of a probability distribution by a differential dependence; And A second step of probabilistically combining the determinations of the plurality of determinants by applying the approximate set obtained in the above step to the combination formula with the Bayesian method Wherein a dependency between the determinants that make a plurality of decisions is taken into consideration without requiring an independent assumption. 18. The method of claim 17, wherein the dependency observes only the determinants of the determinants obtained from the training samples. 19. The method according to claim 18, wherein the coupling equation when combining the Bayesian method and the secondary dependence is expressed as? Lt; / RTI > Which is calculated by combining a plurality of crystals, Values, and combining the plurality of crystals by selecting a maximized posterior probability, depending on the value. 20. The method of claim 19, wherein the determiner is one of a character recognizer, a knowledge based recognizer, or a feature based recognizer using a neural network. 21. The method according to claim 20, wherein, when using the reject recognition result of the recognizer, Characterized in that it identifies the optimal set of differential dependencies by excluding only those of the rejected classes and using only those of the valid classes or including those of the valid classes and the rejected classes, / RTI >