JP2012164008A - Handwritten word recognition device and model learning device for handwritten word recognition - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a handwritten word recognition device which can be greatly improved in recognition precision of handwritten words while taking an environment into consideration even when exactly different fonts are mixed together in one class.SOLUTION: The handwritten word recognition device which recognizes handwritten words present in an image on postal matter in a postal matter processing system is greatly improved in recognition precision of the handwritten words while taking an environment of character deformations etc., due to both adjacent characters into consideration by enabling a plurality of Gaussian distributions to be allocated to one state even when exactly different fonts are mixed together in one class.

Description

  Embodiments of the present invention include, for example, a handwritten word recognition device that recognizes a handwritten word existing in an image on a mail in a mail processing system, and a model learning device for handwritten word recognition used in the handwritten word recognition device About.

  For example, a hidden Markov model (HMM) is generally used for speech recognition, and a method using an HMM has also been proposed for handwritten character recognition.

  In speech recognition, it is known that it is effective to create a phoneme model for each environment because the phoneme is distorted depending on adjacent phonemes (referred to as “environment”). However, when the number of environments is large, the number of learning patterns per phoneme model decreases, resulting in overlearning. Therefore, a method is adopted in which environments are clustered and the number of learning patterns is increased by sharing models between environments belonging to the same cluster to prevent overlearning.

  Use of the above method for handwritten character recognition is expected to improve recognition performance. However, in handwritten character recognition, completely different fonts such as block and cursive are mixed in one class, and the above-mentioned environment clustering method does not assume such a case, so an effect cannot be obtained. .

  For example, an example of a lowercase letter “r” in English letters is given. Since the letter “r” is completely different in the block form and the cursive form, it is considered that two distributions that are largely separated are formed as shown in FIG. And the difference in the environment is considered to be only a small difference in each distribution. In this example, it is assumed that the character on the left side of “r” is an environment, and five types of environments a to e exist, and are represented as ar to err. At this time, when clustering is performed while ignoring the environment, division that reflects the font is possible as shown in C and D of FIG. 9, but if only the environment is divided, a plurality of fonts are used as shown in A and B of FIG. Therefore, the estimation accuracy is significantly reduced compared to the former.

H. Bunke, S. Bengio, A. Vinciarelli, “Offline recognition of unconstrained handwritten texts using HMMs and statistical language models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 26, Issue 6, June 2004, p.p. 709-720 Junichi Takami, Shigeki Hiyama, “Automatic Generation of Hidden Markov Networks by Sequential State Partitioning”, IEICE Transactions D-II, Vol. J76-D-II, No. 10, pp. 2155--2164, Oct 1993.

  Therefore, the present invention provides a handwritten word recognition apparatus and model learning for handwritten word recognition that can significantly improve the recognition accuracy of handwritten words in consideration of the environment even when completely different fonts are mixed in one class. An object is to provide an apparatus.

  The handwritten word recognition apparatus according to the embodiment includes an image capturing unit that captures an image including a handwritten word on a recording medium, a word extracting unit that extracts a word image from the image captured by the image capturing unit, A first feature extracting means for extracting the feature from the word image extracted by the word extracting means; and storing a character model for each character, wherein each character model is a model population and an environment-specific character for each environment. The model population and each environment-specific character model are each composed of a plurality of states, and each state of the model population is composed of at least two G (natural number) Gaussian distributions, Each state of each environment-specific character model is composed of at least two or more M (natural numbers, M ≦ G) Gaussian distributions, and each state of the model population Model storage means corresponding to any of the combinations of M selected from the G Gaussian distributions constituting the feature, the features extracted by the first feature extraction means, and the model storage means And a model matching unit that performs matching processing with each character model and uses the result as a recognition result.

The block diagram which shows roughly the structure of the handwritten word recognition apparatus which concerns on embodiment. The figure which shows an example of an input image. The figure which shows an example of the extracted word candidate. The block diagram which shows roughly the apparatus structure at the time of the recognition process of a handwritten word. The block diagram which shows roughly the apparatus structure at the time of a model learning process. The figure explaining the model which concerns on embodiment. The figure explaining an example of the environment division | segmentation using GMM which concerns on embodiment. The figure explaining the data distribution at the time of mixing of a different font. The figure explaining the environment division example of data distribution.

First, before describing the embodiment, an outline of the present embodiment will be briefly described.
As a method for solving the above-described conventional problems, it is conceivable that the distribution of each environment is not a single Gaussian distribution but a mixed Gaussian distribution (GMM), and each Gaussian distribution is shared. This will be described with reference to FIG. 7 using an example similar to FIG. 8 and FIG. First, since the distribution is greatly different between the two fonts, it is natural to divide the fonts. In FIG. 7, it is divided into left and right parts. Since the left side (“r” in the block body) has a large number of patterns, the environment can be further divided. In FIG. 7, it is divided into a group E (lower side) including three environments ar, br, and cr, and a group F (upper side) including two environments dr and er. The

  On the other hand, the right side (cursive “r”) has a small number of patterns and cannot be further divided into one group G. As a result, for example, the distribution representing ar is the GMM of group E and group G, and the distribution representing dr is the GMM of group F and group G.

  Therefore, in the present embodiment, there are G Gaussians as a whole, and each environment considers a model represented by GMM using M of them, and uses a method of estimating the maximum likelihood with the EM algorithm.

Next, a simultaneous optimization method of environment clustering and GMM (Gaussian Mixture Model) estimation applied to this embodiment will be described.

Hereinafter, embodiments will be described with reference to the drawings.
FIG. 1 schematically shows the configuration of a handwritten word recognition apparatus according to the present embodiment. The handwritten word recognition device includes an image input unit 11 serving as an image capturing unit that captures an image including a handwritten word on a recording medium, and a word extracting unit that extracts a word image from the image captured by the image input unit 11. A word extraction unit 12, a video coding system (VCS) 13 in a mail processing system, a word as word image accumulating means for accumulating a word image instructed by the video coding system 13 as a word image for learning An image storage unit 14; a feature extraction unit 15 as first and second feature extraction means for extracting the features from the word image stored in the word image storage unit 14 or the word image extracted by the word extraction unit 12; Extracted by the model storage unit 16 and the feature extraction unit 15 as model storage means for storing a character model for each character. The feature extracted by the model matching unit 17 and the feature extracting unit 15 as model matching means for performing a matching process between the character model stored in the model storage unit 16 and the character model in the model storage unit 16 A first probability calculation unit 18 as a first probability calculation means for calculating a posterior probability of appearing from each state of the first, a second probability calculation means as a second probability calculation means for calculating a posterior probability that each of the environments is the combination. The second probability calculation unit 19 is a third probability calculation unit that calculates a posteriori probability that the feature extracted by the feature extraction unit 15 appears from each Gaussian distribution on the condition that each environment is each combination. From the probabilities calculated by the third probability calculation unit 20, the first, second, and third probability calculation units 18, 19, and 20 and the features extracted by the feature extraction unit 15. Gaussian parameter updating unit 21 serving as a Gaussian parameter updating unit for calculating a parameter of the Uus distribution and updating a parameter of the Gaussian distribution stored in the model storage unit 16 based on the calculation result, second and third probability calculations Weight parameter update unit as a weight parameter update unit that calculates a weight parameter for model learning from each probability calculated by the units 19 and 20 and updates the weight parameter stored in the model storage unit 16 based on the calculation result 22 is comprised.

Hereinafter, each part will be described in detail.
The image input unit 11 inputs, for example, an image including a handwritten word (English character word in this example) handwritten on a mail as shown in FIG. 2, and is configured by a video camera or the like.

  The word extraction unit 12 extracts a word candidate (word image) by performing known image processing on the image input by the image input unit 11. FIG. 3 shows an example of word candidates extracted from the image of FIG.

For example, the video coding system 13 displays an image of a mail piece for which address information (word) could not be recognized by a mail piece sorting apparatus (not shown) on the display unit, and corrects the word that could not be recognized by the operator's coding work. Input.
The word image storage unit 14 stores the corresponding word image together with the correct answer (word) input by the video coding system 13.

  The feature extraction unit 15 extracts the feature from the word image extracted by the word extraction unit 12 or the word image stored in the word image storage unit 14. In this case, a plurality of features are extracted from one word. Various methods have been proposed for the feature extraction method. For example, the methods disclosed in the following documents can be used.

J. et al. A. Rodriguez, F.M. Perronnin, “Local gradient histogram features for word spotting in unconstrained handwrought documents,” International Conference of Frontiers
The model storage unit 16 stores character models 31, 31,... Corresponding to each character. Each character model 31 includes, for example, one model population 32 and L (two or more) environment-specific character models 33 _{1 to} 33 _L for each environment. Further, the model population 32 and the environment-specific character models 33 _{1 to} 33 _L are each composed of N (two or more) states. Further, each state of the environment-specific character models 33 _{1 to} 33 _L is composed of M (two or more) Gaussian distributions (Gaussian), and each state of the model population 32 is G (M ≦ G) Gaussians. It consists of a distribution. Further, the M Gaussian distributions constituting each state of the environment-specific character models 33 _{1 to} 33 _L may be any combination of M selected from the G Gaussian distributions constituting each state of the model population 32. It corresponds to crab.

  The model matching unit 17 performs a matching process between the feature extracted by the feature extraction unit 15 and each character model 31 stored in the model storage unit 16, and uses the result that maximizes the matching score as the recognition result.

  The first probability calculation unit 18 calculates a posterior probability that the feature extracted by the feature extraction unit 15 appears from each state of each character model 31 in the model storage unit 16. The calculation method can be calculated using, for example, the calculation formula disclosed in the document “Machine Learning for Audio, Image and Video Analysis Theory and Applications”, Springer, 2008.

  The second probability calculation unit 19 calculates a posteriori probability that each environment is each combination (each GMM). The third probability calculation unit 20 calculates the posterior probability that the feature extracted by the feature extraction unit 15 appears from each Gaussian distribution on condition that each environment is each combination (each GMM).

  The Gaussian parameter updating unit 21 calculates the parameters of the Gaussian distribution from the probabilities calculated by the first, second, and third probability calculating units 18, 19, and 20 and the features extracted by the feature extracting unit 15. Based on the calculation result, the parameters of the Gaussian distribution stored in the model storage unit 16 are updated.

  The weight parameter updating unit 22 calculates weight parameters for model learning from the probabilities calculated by the second and third probability calculation units 19 and 20, and is stored in the model storage unit 16 based on the calculation results. Update weight parameters.

Next, handwritten word recognition processing in the above configuration will be described.
At the time of handwritten word recognition processing, the device configuration shown in FIG. 1 is as shown in FIG. 4, and the video coding system 13, the word image storage unit 14, the first probability calculation unit 18, and the second probability calculation unit 19 are used. The third probability calculation unit 20, the Gauss parameter update unit 21, and the weight parameter update unit 22 are not used.

  First, the image input unit 11 inputs an image including a handwritten word on a postal matter. FIG. 2 shows an example of the input image. Next, the word extraction unit 12 extracts a word candidate (word image) by performing known image processing on the image input by the image input unit 11. FIG. 3 shows an example of word candidates extracted from the image of FIG.

  Next, the feature extraction unit 15 extracts the feature from the word image extracted by the word extraction unit 12. In this case, a plurality of features are extracted from one word. Next, the model matching unit 17 performs a matching process between the feature extracted by the feature extraction unit 15 and each character model 31 stored in the model storage unit 16, and obtains a recognition result with a result that maximizes the matching score. To do.

Next, the model learning process will be described.
During the model learning process, the apparatus configuration shown in FIG. 1 is as shown in FIG. 5, and the image input unit 11, the word extraction unit 12, and the model matching unit 17 are not used.

  First, in the video coding system 13, a correct answer is taught to a word image that could not be recognized by a mail sorting apparatus (not shown) and stored in the word image storage unit 14. The feature extraction unit 15 extracts features from the word images stored in the word image storage unit 14. In this case, a plurality of features are extracted from one word.

  Next, the first probability calculation unit 18 calculates a posterior probability that the feature extracted by the feature extraction unit 15 appears from each state of each character model 31 in the model storage unit 16. Each feature belongs to a state where the posterior probability is maximized. That is, a specific character model, a specific environment, and a specific state are assigned to each feature.

Here, the selected features belonging to environment l correspond to x _l1 ,..., X _lNl that are elements of x _{l in} the “simultaneous optimization method of environment clustering and GMM estimation” described above. Note that “data” in the “simultaneous optimization method of environment clustering and GMM estimation” described above corresponds to “feature” in the present embodiment.

  Next, the second probability calculation unit 19 calculates a posteriori probability that each environment is each combination (each GMM). That is, the calculation of Expression (93) in the “simultaneous optimization method of environment clustering and GMM estimation” described above is performed.

  Next, the third probability calculation unit 20 calculates the posterior probability that the feature extracted by the feature extraction unit 15 appears from each Gaussian distribution on condition that each environment is each combination (each GMM). That is, the calculation of Expression (97) in the “simultaneous optimization method of environment clustering and GMM estimation” described above is performed.

  Next, the Gaussian parameter updating unit 21 calculates the parameters of the Gaussian distribution from the probabilities calculated by the first, second, and third probability calculating units 18, 19, and 20 and the features extracted by the feature extracting unit 15. Then, the parameters of the Gaussian distribution stored in the model storage unit 16 are updated based on the calculation result.

  That is, the calculation of Equation (78) in the “simultaneous optimization method of environment clustering and GMM estimation” described above is performed, and the parameters (means and covariance matrix) of the Gaussian distribution stored in the model storage unit 16 are updated. To do.

  Next, the weight parameter updating unit 22 calculates weight parameters for model learning from the probabilities calculated by the second and third probability calculation units 19 and 20, and stores them in the model storage unit 16 based on the calculation results. Update the weight parameter.

  In other words, the equations (83) and (88) in the “simultaneous optimization method of environment clustering and GMM estimation” described above are calculated, and model learning related to the model population 32 stored in the model storage unit 16 is performed. Update the weight parameter.

  In the above embodiment, a case has been described in which the word image storage unit 14 stores a word image that has been recognized by the video coding system 13 and has not been recognized as a word image for learning. The word image in which the correct answer inputted by other word image input devices is taught is not limited to 13, and may be accumulated as a learning word image. Furthermore, not only the word image that could not be recognized, but also the recognized word image, that is, the correct answer was found, so that the word image can also be used as a learning word image.

  Moreover, although the said embodiment demonstrated the case where the one feature extraction part 15 extracts the characteristic from the word image accumulate | stored in the word image storage part 14, or the word image extracted by the word extraction part 12, a word A dedicated feature extraction unit may be provided for each of the image storage unit 14 and the word extraction unit 12.

  Further, in the above embodiment, each feature belongs to the state where the posterior probability is maximum in the first probability calculation unit 18; however, the posterior probability is regarded as a weight without belonging to a specific state, and the subsequent calculation is performed. May be performed.

  As described above, according to the above-described embodiment, even when completely different fonts are mixed in one class, a plurality of Gaussian distributions can be assigned to one state. The recognition accuracy of handwritten words can be remarkably improved in consideration of the environment such as character deformation.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 11 ... Image input part (image taking means), 12 ... Word extraction part (word extraction means), 13 ... Video coding system (VCS), 14 ... Word image storage part (word image storage means), 15 ... Feature extraction part (Feature extraction means), 16 ... model storage section (model storage means), 17 ... model matching section (model matching means), 18 ... first probability calculation section (first probability calculation means), 19 ... second Probability (second probability calculation means), 20 ... third probability calculation section (third probability calculation means), 21 ... Gauss parameter update section (Gauss parameter update means), 22 ... weight parameter update section (weight parameter update) Means), 31 ... character model, 32 ... model population, 33 _{1 to} 33 _L ... environment-specific character model.

Claims (5)

Image capturing means for capturing an image including a handwritten word on a recording medium;
Word extraction means for extracting a word image from the image captured by the image capture means;
First feature extraction means for extracting the feature from the word image extracted by the word extraction means;
A character model for each character is stored. Each character model is composed of a model population and an environment-specific character model for each environment, and each of the model population and each environment-specific character model is composed of a plurality of states. Each state of the model population is composed of at least two G (natural numbers) Gaussian distributions, and each state of each environment-specific character model has at least two M (natural numbers, M ≦ G). Model storage means constituted by a number of Gaussian distributions and corresponding to any of a combination of M selected from the G number of Gaussian distributions constituting each state of the model population;
A model matching unit that performs a matching process between the feature extracted by the first feature extraction unit and each character model stored in the model storage unit, and sets the result as a recognition result;
A handwritten word recognition apparatus comprising: A handwritten word recognition model learning device used in the handwritten word recognition device according to claim 1,
Word image storage means for storing a word image for learning;
Second feature extraction means for extracting features from the word images stored by the word image storage means;
First probability calculation means for calculating a posterior probability that the feature extracted by the second feature extraction means appears from each state of each character model in the model storage means;
A second probability calculating means for calculating a posteriori probability that each environment is the combination;
Third probability calculating means for calculating a posterior probability that the feature extracted by the second feature extracting means appears from each Gaussian distribution on the condition that each environment is the respective combination;
Gaussian distribution parameters are calculated from the probabilities calculated by the first, second, and third probability calculating means and the features extracted by the second feature extracting means, and the model storage means is based on the calculation result. Gaussian parameter updating means for updating the parameters of the Gaussian distribution stored in
A weight parameter for model learning is calculated from the respective probabilities calculated by the second and third probability calculating means, and based on the calculation result, the model learning for the model population stored in the model storing means is calculated. A weight parameter updating means for updating the weight parameter;
A model learning apparatus for recognizing handwritten words, comprising:   3. The word image for learning of handwritten words according to claim 2, wherein the word image for learning stored in the word image storage means is a word image instructing a correct answer obtained from a video coding system in a mail processing system. Model learning device. Image capturing means for capturing an image including a handwritten word on a recording medium;
Word extraction means for extracting a word image from the image captured by the image capture means;
First feature extraction means for extracting the feature from the word image extracted by the word extraction means;
A character model for each character is stored. Each character model is composed of a model population and an environment-specific character model for each environment, and each of the model population and each environment-specific character model is composed of a plurality of states. Each state of the model population is composed of at least two G (natural numbers) Gaussian distributions, and each state of each environment-specific character model has at least two M (natural numbers, M ≦ G). Model storage means constituted by a number of Gaussian distributions and corresponding to any of a combination of M selected from the G number of Gaussian distributions constituting each state of the model population;
A model matching unit that performs a matching process between the feature extracted by the first feature extraction unit and each character model stored in the model storage unit, and sets the result as a recognition result;
Word image storage means for storing a word image for learning;
Second feature extraction means for extracting features from the word images stored by the word image storage means;
First probability calculation means for calculating a posterior probability that the feature extracted by the second feature extraction means appears from each state of each character model in the model storage means;
A second probability calculating means for calculating a posteriori probability that each environment is the combination;
Third probability calculating means for calculating a posterior probability that the feature extracted by the second feature extracting means appears from each Gaussian distribution on the condition that each environment is the respective combination;
Gaussian distribution parameters are calculated from the probabilities calculated by the first, second, and third probability calculating means and the features extracted by the second feature extracting means, and the model storage means is based on the calculation result. Gaussian parameter updating means for updating the parameters of the Gaussian distribution stored in
A weight parameter for model learning is calculated from the respective probabilities calculated by the second and third probability calculating means, and based on the calculation result, the model learning for the model population stored in the model storing means is calculated. A weight parameter updating means for updating the weight parameter;
A handwritten word recognition apparatus comprising:   5. The handwritten word recognition apparatus according to claim 4, wherein the learning word image stored in the word image storage means is a word image taught with a correct answer obtained from a video coding system in a mail processing system. .

Images