CN107590497A - Off-line Handwritten Chinese Recognition method based on depth convolutional neural networks - Google Patents

Abstract

The present invention proposes an off-line handwritten Chinese character method using a deep convolutional neural network. Specifically, a spatial transformation network is first used to rotate, translate and scale the handwritten Chinese characters in the HWDB1.1 database. The original arbitrary position, proportion and direction of the Chinese character pictures are corrected. The corrected handwriting dataset is then fed into a convolutional neural network for recognition and classification. The twist parameter increments for rotating, translating, and scaling the original image are calculated by a convolutional neural network or neural network that is continuously adjusted through backpropagation. The delta of the warp parameter and the warp parameter update parameters in a composite way. Finally, we built the network framework of the handwritten Chinese character recognition of the present invention on the TensorFlow deep learning framework platform. And compared with the network framework of handwritten Chinese character recognition using convolutional neural network alone, after training with a large number of data sets, the test results show that the recognition rate of handwritten Chinese characters has been significantly improved.

Description

Offline Handwritten Chinese Character Recognition Method Based on Deep Convolutional Neural Network

technical field

The invention belongs to the technical field of image classification, in particular to an off-line handwritten Chinese character recognition method based on a deep convolutional neural network.

Background technique

Off-line handwritten Chinese character recognition is a sub-direction in the field of pattern recognition. Offline means that the handwritten characters to be processed are two-dimensional images of handwritten characters collected by image capture devices such as scanners or cameras, hereinafter referred to as handwritten Chinese character recognition. Most of the research papers and technical reports published before 2011 focus on how to select features and matching methods to adapt to the variation of handwritten Chinese characters. It is in the design of feature extraction algorithms and classifiers. Traditional HCCR steps include: graph normalization, feature extraction, dimensionality reduction, and classifier training. Moreover, due to the large number of Chinese characters, complex structures, many similar characters, and changing writing styles, these traditional methods not only have complicated steps, but also if the selected features are not suitable in the feature extraction step, the recognition effect will be seriously affected. Although the methods based on MQDF and DLQDF have achieved good results, they have reached their bottleneck.

In recent years, due to the major breakthroughs in image recognition based on deep learning, Deep Convolutional Neural Network (DCNN) has also been applied in the field of handwritten Chinese character recognition. In the 2013 ICDAR international competition, the Fujitsu team won the championship with a recognition rate of 94.77% using the CNN model. Later, a handwritten Chinese character recognition model using improved models and preprocessing methods appeared. The most common improvement method is to increase the tolerance of the network to the deformation of different handwritten Chinese characters, including data enhancement technology and spatial pooling. These two methods have the following disadvantages. The former generates new samples based on the geometric deformation of the original data. The method is the most widely used method in handwritten Chinese character recognition. If only the learning ability of the model with exponential growth of samples can be improved, it is also a problem that cannot be ignored. In the latter, pooling operations can corrupt image details. Moreover, with the explosive growth of data, although the deep convolutional neural network has achieved great success, there is no criterion to solve the geometric changes of the same type of data. For example, the same handwritten "Han" character has different sizes and stroke shapes written by different people. The variety of data transformation is the key factor affecting the recognition effect of deep convolutional neural network.

The improved methods include: preprocessing method, distorting and deforming Chinese characters to expand the data set, and improving the generalization ability of the convolutional neural network; feature extraction, combining traditional Gabor direction features, HoG features, etc., using different DCNN models. The currently used models include GoogLeNet composed of Szegedy et al., ResLeNet designed by K He, X Zhang et al. These models are all designed for image classification. There is still a certain difference from handwritten Chinese character recognition. Although it has achieved good results, the network structure is deep, the model is complex, and debugging is difficult. Moreover, the degree of distortion and deformation of Chinese characters is not easy to grasp, and the angle cannot be selected adaptively. In 2015, four Cambridge PhDs from DeepMind, a cutting-edge AI company under Google,

The researchers designed a spatial transformation network (Spatial Transformer Network), which can achieve adaptive rotation, translation, and scaling. However, there are also some problems in directly using the framework composed of reverse spatial transformation network and convolutional neural network for handwritten Chinese character recognition. For example, after the reverse space transformation network, although the direction of the Chinese characters is corrected, the strokes of the corrected Chinese characters are thicker. Moreover, the samples in the processing process are samples that have been scaled, rotated and translated by multiple layers, and then directly input to the convolutional neural network for recognition. Because the scaled sample cuts out the edge information of the original image, it will cause incomplete fonts when used in handwritten Chinese character recognition, which will seriously affect the recognition effect.

Because most of the current networks ignore the spatial distortion of actual handwritten Chinese characters, because the actual writing environment, style, direction, position, and size are different, resulting in a variety of sample sets. However, CNNs still lack robustness to spatial variations of input samples. The traditional normalization method only converts samples into normalized Chinese characters of a specified size. Although it plays a pivotal role in the classification task, the normalization method cannot guarantee that the HCCR task is optimal. The reverse synthesis space transformation network can align the input picture or the learned feature space according to the feature transformation parameters of the task learning picture without labeling the key points, thereby reducing the spatial rotation, translation, scale, and distortion. The effect of isogeometric transformations on classification and recognition.

Contents of the invention

Aiming at the above problems and the characteristics of Chinese characters, the present invention adopts algorithms such as reverse synthesis space transformation algorithm and deep convolutional neural network to recognize offline handwritten Chinese characters. This method is robust to different writing styles and different writing environments. Among them, we use the reverse synthesis space transformation network to solve the problems of font distortion, deformation, and tilt caused by various writing styles. And designed the corresponding deep convolutional neural network. Deep convolutional neural networks can efficiently extract features and classify them. Finally, the output of the reverse synthesis space transformation network is used as the input of the convolutional neural network to realize the recognition of handwritten Chinese characters.

Technical scheme of the present invention and flow sequence are as follows:

(1) Build a TensorFlow deep learning framework platform for deep convolutional neural networks;

(2) Convert the GNT format data of the HWDB1.1 dataset into binary and store it in PKL format

(3) Read the PKL format data and perform normalization processing, and convert it into training set, cross-validation set and test set;

(4) Realize the reverse synthesis space transformation network and convolutional neural network in the TensorFlow framework platform, and use the output of the reverse synthesis space transformation network as the input of the convolutional neural network;

(5) direction synthesis space transformation network frame model is set and added in the convolutional neural network framework designed by the inventor himself, forming the model structure of inverse synthesis space transformation network plus convolutional neural network;

(6) Train and test the constructed network framework.

(7) Use the TensorBoard tool to visualize intermediate results, training and testing results. Comparatively analyze the effectiveness of the network structure of the present invention.

The detailed process of said step 2): HWDB1.1 is a dataset of off-line handwritten Chinese characters of the CASIA-HWDB database. It was collected by the Chinese Academy of Sciences during 2007-2010. The storage format is the unique binary GNT format. So we need to convert this dataset into a usable format first. Because this data set has a variety of writing fonts, it is currently one of the most authoritative data sets that verify the model's recognition effect.

Said step 3): reverse and normalize the picture, the former can make most of the data in the picture zero, speed up the calculation speed, and the latter can improve the convergence speed of the network.

The step 4): the specific process of inversely synthesizing the space transformation network is: multiplying the training sample image with the initialized distortion parameter p to obtain the first corrected image. Feed this into a geometric predictor, which is built from a small convolutional neural network, or neural network. The forward propagation of the network predicts the distortion increment Δp, and the backpropagation BP algorithm updates the parameters of the geometric predictor, and further iteratively updates the distortion parameters. After every update of Δp, the parameter p is updated synthetically. Afterwards, the matrix composed of P is multiplied by the training sample, and the result is a corrected picture, and the corrected picture is input to the geometric predictor, so that Δp and p are iteratively updated. Until the number of cycles given in advance is satisfied, the final P value is multiplied by the original training sample, and the obtained result is used as the input of the convolutional neural network designed by the present invention for further feature extraction and classification.

Said step 5): the convolutional neural network adopts the first two convolutions plus a pooling layer, followed by two convolutional layers, and two fully connected layers after the convolutional layer.

Compared with the current handwritten Chinese characters based on the deep convolutional neural network, 1) the present invention adopts the reverse synthesis space transformation network to convert the handwritten Chinese character set into easy-to-recognize Chinese character pictures through twisting and deformation, effectively improving the recognition of handwritten Chinese characters Effect. And avoid man-made design parameters such as distortion and deformation of the picture. 2) The convolutional neural network adopts a network structure of two convolutional layers plus one pooling layer plus two convolutional layers. More convolutional layers and small convolution kernels are used to effectively extract the stroke features of Chinese characters.

Description of drawings

Figure 1 Flowchart of reverse synthesis space transformation network

Figure 2 CNN_4 network structure

Figure 3 Comparison diagram of scheme 1 before and after the reverse space transformation network

(a) Original handwritten Chinese characters

(b) Handwritten Chinese characters corrected by ICSTN_2

Figure 4: Comparison of scheme 2 before and after the reverse space transformation network

(a) Original handwritten Chinese characters

(b) Handwritten Chinese characters corrected by ICSTN_2

Figure 5. Comparison of scheme three before and after the reverse space transformation network

(a) Original handwritten Chinese characters

(b) Handwritten Chinese characters corrected by ICSTN_4

Figure 6 Scheme 1 Cost function and test accuracy change curve

(a) Using ICSTN_1+CNN_4 loss function

(b) Test accuracy using ICSTN_1+CNN_4

Figure 7 The cost function and test accuracy change curve of scheme 2

(a) Using ICSTN_2+CNN_4 loss function

(b) Test accuracy using ICSTN_2+CNN_4

Figure 8 The three-cost function and test accuracy change curve of the scheme

(a) ICSTN_4+CNN_4 loss function

(b) Test accuracy using ICSTN_4+CNN_4

Figure 9 CNN_4 cost function and test accuracy change curve

(a) Using CNN_4 loss function

(b) CNN_4 test accuracy

specific implementation

Below in conjunction with accompanying drawing, the present invention is described in further detail.

The specific construction method of the network, as well as the algorithm principle and effect of the new theory adopted by the present invention are given below. The implementation of the present invention is further described. The example that the present invention adopts is 200 kinds of Chinese characters of HWDB1.1, and each category has 300 Chinese characters, totally 60000 samples, wherein training set, cross-validation set and testing machine are respectively: 48000, 6000, 6000. The specific process of the network operation designed by the present invention is:

(1) Data preprocessing, inverting the acquired binary image and normalizing it to improve the convergence of the network. And uniformly adjust the image size to 32*32, and then distribute it as training set, cross-validation set and test set;

(2) Input the training sample to the next step deformation parameter, denoted as I(x), x=(x,y) means that any pixel point in the image is located. p=[p ₁ p ₂ . . . p _n ] represents a twist parameter. In the present invention, the affine transformation is p=[p ₁ p ₂ p ₃ p ₄ p ₅ p ₆ ], and the relevant secondary coordinate table transformation matrix is:

(3) W(p) transforms the original image:

ImWarp(x)=I(x)·W(p) (2)

(4) Because image coordinates are non-integral and cause image discontinuity through twisting and transforming, the present invention carries out bilinear interpolation through transformed image ImWarp(x):

V _i ^c : represents the i-th pixel of the c-th sample;

(x _i , y _i ): Indicates the i-th pixel coordinate of the training sample ImWarp;

W: Indicates the width of the image;

H: Indicates the height of the image;

(5) Input the image V(x) into the convolutional neural network or neural network. The present invention refers to this network as a geometric predictor (why the neural network or convolutional neural network is used to learn the distortion increment will be deduced in detail later) , the geometric predictor is used to learn the distortion increment Δp of the image, and there are three schemes for the composition of its network structure. Finally, the present invention will simulate and realize the effect of each scheme one by one. The three options are:

a) Scheme 1: The structure of the network is: Conv(7×7,4)+Conv(7×7,8)+P+FC(48)+FC(8), where Conv represents the convolutional layer, and P represents Pooling layer, FC means fully connected layer, the size and number of convolution kernels in parentheses, and the number of neurons in parentheses in fully connected layer;

b) Scheme 2: The structure of the network is: Conv(9×9,8)+FC(48)+FC(8);

c) Scheme 3: The structure of the network is: FC(48)+FC(8)

The forward propagation of the geometric predictor outputs the warp increment Δp, and the backpropagation updates the network parameters to further generate a new Δp.

(6) Updating the distortion parameter, the update method of the distortion parameter p is synthesis. Under the specific formula:

The corresponding transformation matrix is:

W(p _out )＝W(ΔP)·W(p _in ) (5)

(7) If the number of cycles recurN>1, go to step (3) until the end of the cycle. The inverse synthesis spatial transformation network can be cycled multiple times to achieve the best results in aligning and warping images.

(8) After the loop ends, the transformation matrix W(p _out ) composed of the finally updated distortion parameter p is applied to the original image j, namely:

Im=I(x)·W(P _out ) (6)

, and then input the image into the convolutional neural network;

(9) The framework of the convolutional neural network is shown in Figure 2. The network structure is Conv(3×3,8)+Conv(3×3,16)+P+Conv(3×3,32)+Conv(3×3 ,64)+P+FC(100)+FC(200), what each parameter represents is the same as above. Next, we will use a convolutional layer, a pooling layer, and a fully connected layer to give an example.

The convolutional layer, the main content of the convolutional layer is to use a trainable convolution kernel to perform convolution operations on the input data, and output the results in a certain combination form, which is essentially the feature extraction of the input data. In general, in order to obtain nonlinear characteristics of the model and limit the output within a given range, the output of the convolution is converted with a non-linear function (Non-linear Function). This function is also called the activation function (Activation Function). This example uses the Relu activation function with a faster convergence speed. The specific operation formula of the convolutional layer is:

Among them, Im is the output of the reverse space transformation network, which is the training sample of distorted Chinese characters corrected by the reverse space transformation network, W is the convolution kernel, b is the convolution layer bias; U is the convolution layer output, and f is Relu activation function, Y is the output of the convolution output U through the activation function, which is the output of the entire convolution layer and the input of the next layer.

The main task of the pooling layer is to sample the input data samples in two-dimensional space, also known as the downsampling operation. The specific calculation process is as follows:

Pooling layers make the network robust to translation, rotation, etc. Make the network more tolerant.

Fully connected layer, the function of the fully connected layer is to reduce the input two-dimensional feature matrix to a one-dimensional feature vector, which is convenient for the output layer to perform classification processing.

Output layer, the function of the output layer is to classify according to the one-dimensional vector output of the above fully connected layer. This column uses the softmax cross entropy loss

The distortion offset Δp of the image can be calculated by the convolutional neural network model because for the reverse synthesis KL image alignment algorithm:

Equation (9) is the objective function of reverse composite image alignment, where I is the original image, T is the target image, p: the distortion parameter, Δp: the required distortion increment; I(p) and T(Δp) respectively represent the Image of the effect of warping parameters. Then, the above formula (9) is approximated by the first-order Taylor:

Its least squares solution is:

where the + sign represents the generalized inverse. Next, the parameter p is updated in the following way:

Among them, is the synthesis operation of the distortion parameter p, the solution of formula (10) is a linear regression form, which can be transformed into a general formula:

Δp=R·I(p)+b (13)

where R is a linear regression that estimates the linear relationship between the image and the geometry of the image, and b is the bias. Mainly to obtain parameters R and b. We can parameterize the linear regression model as a neural network or a convolutional neural network, and the output of the network is Δp.

The reverse synthetic spatial transformation network does not directly predict the geometric transformation of the image by the network, but uses the geometric predictor to update the warp parameter p multiple times before classification, and after each cycle, the warp parameter p is saved after outputting the image, This will avoid edge effects when resampling the original image. The geometric predictor {R,b}, the objective function optimized and learned with supervised gradient descent, its objective function is:

Δp: Warp increment, is the output of the geometry predictor

M: the number of samples of the image

From the formula, (12) and (13) can get:

Use the synthetic algorithm to update the parameters, instead of updating the first-order partial derivative of the image to the parameter every time, the gradient is calculated in the backpropagation, and the partial derivative of the distortion parameter is a closed mathematical expression, P _in and P _out are the input respectively Warp parameters for output:

I is the identity matrix, which enables gradients to be backpropagated to the geometric predictor.

In order to intuitively illustrate the feasibility and effectiveness of the network framework we constructed, the present invention is verified through experimental simulation. And verified the three schemes of the geometric predictor network we designed one by one. The framework composed of each scheme adopts the same parameter configuration during training: epoch=16, the initial learning rate is 0.001, and the learning rate is multiplied by a factor of 0.8 every two epochs. In order to compare the effectiveness of using the inverse synthesis space, the present invention realizes only the recognition effect of the convolutional neural network through separate simulation.

Firstly, the present invention visualizes the training samples before and after the inverse spatial transformation network. The models corresponding to schemes 1, 2, and 3 are: ICSTN_1+CNN_4, ICSTN_2+CNN_4, and ICSTN_4+CNN_4. The number after ICSTN_ is the model depth. Figures 3, 4, and 5 are the comparison charts of three different geometric predictor models before and after sample transformation. On the whole, after the reverse synthesis space transformation network, each Chinese character has the same size, uniform strokes, and neat arrangement, and some distorted strokes have been corrected, which is much more recognizable than the original Chinese characters. The difference between different models is also quite significant, and the overall trend is that the deeper the geometric predictor network, the clearer the spatially transformed Chinese character strokes, and the less over-rotation, the more distorted strokes are corrected. As shown in Figure 5(b), Chinese characters such as "jin", "jiu", and "jin" have been properly corrected and deformed. And the effect of Fig. 3 is the worst, there are more Chinese characters to produce inclination. In summary, it can be concluded that the deeper the geometric predictor network of the reverse synthesis space transformation network, the more effective it is to correct the distorted Chinese characters.

Next, analyze the iteration curves of each network framework: Figures 6, 7, and 8 are the loss function curves and test accuracy curves of the three models composed of the three schemes. The number of iterations on the ordinate is in units of 50, that is, the results of every 50 training iterations are displayed, so there are a total of 50,000 iterations and 10 epochs. From (a) of the three figures, it can be seen that the ICSTN_2+CNN_4 and ICSTN_4+CNN_4 models converge faster, and the model ICSTN_4+CNN_4 has the best model stability because the curve has fewer sharp points. But the loss of the ICSTN_2+CNN_4 model is closest to zero. From (b) of the three figures, it can be seen that ICSTN_2+CNN_4 has the highest recognition rate. Then why does the model formed by Scheme 3 have the best effect of correcting Chinese characters, but does not get the best recognition rate? The main reason is that the ICSTN_4+CNN_4 model has many parameters and has not been fully trained under the same epoch. Compared with the model composed of schemes 1, 2, and 3, the CNN model has a large gap. Compared with (a) (b) in Figure 9 and (a) (b) in Figures 6, 7, and 8, the value of the loss function convergence The largest, the recognition rate is only 82.08%. The recognition effect of CNN_4 is not good because the model is shallow.

Finally, it can be seen from Table 1 that from the perspective of the time consumed by training the model, the deeper the inverse spatial transformation network, the more time it consumes. Therefore, factors such as comprehensive efficiency and recognition rate, ICSTN_2+CNN_4 have the best effect and material rate. from the accuracy of the test. The present invention has clear advantages. The training time difference between the CNN_4 model and the ICSTN_1+CNN_4 model is small, but the difference in recognition effect is obvious. ICSTN_1 uses only one layer of neural network. Therefore, the present invention realizes the combination of the reverse spatial transformation network and the shallower convolutional neural network but obtains a higher recognition effect.

Table 1 Recognition effect of different network structures

model class Recognition accuracy (%) Time-consuming (h) CNN_4 82.08 5.16 ICSTN_4+CNN_4 95.83 17.72 ICSTN_2+CNN_4 96.38 13.40 ICSTN_1+CNN_4 95.32 6.92

Claims (2)

1. A kind of 1. method of the Off-line Handwritten Chinese Recognition based on depth convolutional neural networks, it is characterised in that including following step Suddenly：

   TensorFlow deep learning frameworks are built on Windows；

   Prepare data set, data set is converted into TensorFlow input form, and pretreatment is normalized in picture；

   Inverse composition spatial alternation network and the convolutional neural networks of depth are built under TensorFlow environment, using reverse Blended space network carries out twist distortion to handwritten Chinese character, is corrected and alignd.And output it as convolutional neural networks Input；

   Finally the network of inverse composition spatial alternation network and convolutional neural networks formation is trained using substantial amounts of data, Test.And the picture during network processes and final testing result are visualized by TensorBoard instruments, analyze its work With, and contrast recognition effect.
2. 2. according to step described in claims 1, handwritten Chinese character is corrected using inverse composition spatial alternation, its feature exists In：

   The homogeneous matrix formed using warp parameters enters line translation alignment, exemplified by warp parameters are in a manner of affine, ginseng to original image Number p=[p₁ p₂ p₃ p₄ p₅ p₆], corresponding homogeneous transform matrix：

   <mrow> <mi>W</mi> <mrow> <mo>(</mo> <mi>p</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfenced open = "[" close = "]"> <mtable> <mtr> <mtd> <mrow> <mn>1</mn> <mo>+</mo> <msub> <mi>p</mi> <mn>1</mn> </msub> </mrow> </mtd> <mtd> <msub> <mi>p</mi> <mn>2</mn> </msub> </mtd> <mtd> <msub> <mi>p</mi> <mn>3</mn> </msub> </mtd> </mtr> <mtr> <mtd> <msub> <mi>p</mi> <mn>4</mn> </msub> </mtd> <mtd> <mrow> <mn>1</mn> <mo>+</mo> <msub> <mi>p</mi> <mn>1</mn> </msub> </mrow> </mtd> <mtd> <msub> <mi>p</mi> <mn>6</mn> </msub> </mtd> </mtr> <mtr> <mtd> <mn>0</mn> </mtd> <mtd> <mn>0</mn> </mtd> <mtd> <mn>1</mn> </mtd> </mtr> </mtable> </mfenced> </mrow>

   Acted on image, twist distortion is carried out to image I (x), include the operation such as the rotation of image, translation, scaling.

   ImWarp (x)=I (x) W (p)

   ImWarp (x) is the picture corrected.Parameter p is to be updated to ask for by the way of iteration, and p distortion increases Amount Δ p is that this network is referred to as into geometry fallout predictor by a convolutional neural networks or neural computing, the present invention.By Inspired in inverse composition spatial alternation principle by inverse composition Lucas-Kanade algorithms：

   <mrow> <munder> <mrow> <mi>m</mi> <mi>i</mi> <mi>n</mi> </mrow> <mrow> <mi>&amp;Delta;</mi> <mi>p</mi> </mrow> </munder> <mo>|</mo> <mo>|</mo> <mi>I</mi> <mrow> <mo>(</mo> <mi>P</mi> <mo>)</mo> </mrow> <mo>-</mo> <mi>T</mi> <mrow> <mo>(</mo> <mi>&amp;Delta;</mi> <mi>P</mi> <mo>)</mo> </mrow> <mo>|</mo> <msubsup> <mo>|</mo> <mn>2</mn> <mn>2</mn> </msubsup> </mrow>

   Shift onto and convert by a series of, its solution is：

   Δ p=RI (p)+b

   Last solution is linear regression forms, it is possible to which its solution procedure parameter is turned into neutral net or convolutional Neural net Network；

   Seeking the network structure of distortion increment Delta p geometry fallout predictor has three kinds of schemes, and two schemes are convolutional neural networks structures, One neutral net for individual layer.There is detailed structure explanation in specification.We eventually verify each scheme and divided one by one Analyse its validity.

   The Δ p of geometry fallout predictor renewal synthesizes with p before, undated parameter p.The p value of renewal can be existed from new role afterwards On image, loop iteration asks for optimal p value.Defined according to synthesis, update mode is as follows：

   By the way of synthesis it is the gradient that need not remove to ask sample image every time come the benefit of undated parameter.Greatly accelerate anti- To the calculating speed of propagation.

   It will be finally input to by the picture corrected in convolutional neural networks and carry out feature extraction and classifying.Convolutional neural networks Structural order is respectively：Two convolutional layers, then a pond layer, two convolutional layers, a pond, it is finally two full connections Layer.Final output neuron number is identical with the classification of present example.