CN112818951B - A method of ticket identification - Google Patents

Abstract

The invention discloses a ticket recognition method, which relates to the technical fields of text detection, text recognition and structured information extraction. It solves the technical problem that the existing model cannot effectively extract structured information. The key point of the technical solution is to carry out the CTPN network The text line position detection model is trained to locate the key information in the ticket and is robust to various forms of tickets (tables, etc.); the data is synthesized through rules of high-frequency words and text content in specific fields to expand the It provides training data for the text recognition model and improves the accuracy of the recognition model; it is based on a convolutional neural network and has good parallelism, and can use high-performance GPU (Graphics Processing Unit, graphics processor) to accelerate calculations.

Description

A method of ticket identification

Technical field

The present disclosure relates to the technical fields of text detection, text recognition and structured information extraction, and in particular, to a method of ticket recognition.

Background technique

Ticket recognition refers to the technology of identifying images containing text information in different fields such as invoices, ID cards, bank cards, etc. that are common in daily life and extracting structured information from them. Since tickets include many fields and their formats are complex, it brings many difficulties to identification and structured extraction.

The ticket structured recognition task can be subdivided into research tasks in multiple fields such as text detection and text recognition. The main method in the current field of text detection is to combine the target detection or segmentation algorithm in deep learning with the text detection task, such as EAST. This algorithm uses the FCN (Fully Convolutional Networks) structure commonly used in semantic segmentation. In fact, The text box parameters are regressed based on the idea of regression, and the FCN architecture is used to complete the feature extraction and feature fusion operations. Then the EAST model predicts the regression parameters of a set of text lines at each position in the image, and finally uses non-polar The large value suppression operation extracts lines of text from the input image. This method greatly simplifies the process of text detection, but currently similar methods still have problems such as poor detection of long text and poor detection of small text areas, and these problems are the more critical issues in ticket recognition.

Currently, there are two main methods in the field of text recognition: character recognition and sequence recognition. When using the character recognition method for text recognition, you first need to segment a single character from the image, then classify the single character image through a classifier, and finally merge it into a text line-level recognition result; while the text recognition algorithm based on sequence recognition divides the entire As the smallest unit of recognition, text lines are automatically aligned to complete the recognition of the entire text sequence. At the same time, the Seq2Seq model of natural language processing and the attention mechanism are introduced to improve the recognition effect. However, both methods have their own problems. The character recognition method requires character-level supervision information, so a lot of annotation work is required; the robustness of the method based on sequence recognition is greatly affected by the training data, and for images with complex backgrounds and similar characters are prone to misrecognition.

Therefore, for the task of structured ticket recognition, the current method does not take into account the extraction of structured information, and the messy information obtained cannot be directly used in subsequent work, so the above problems still need to be researched and solved.

Contents of the invention

The present disclosure provides a method for ticket recognition. Its technical purpose is to establish a model that can effectively extract structured information to address issues such as different image styles, inconsistent table formats, and unclear printing in tickets.

The above technical objectives of the present disclosure are achieved through the following technical solutions:

A method of ticket recognition, a model training process and a text recognition process. The model training process includes:

S100: Collect data for text line detection and text image recognition; wherein the data includes text line images;

S101: Collect high-frequency words that appear in various ticket scenarios, establish a keyword database based on the high-frequency words, and count the rules of text content in specific fields in the high-frequency words. According to the high-frequency words and the Rules randomly generate extended data;

S102: Train the CTPN network through the text line image to obtain a text line position detection model;

S103: Train the recognition network through the data and the expanded data to obtain a text recognition model with a self-attention mechanism;

The text recognition process includes:

S200: Input the image of the ticket into the text line position detection model. The text line position detection model detects the text line position in the ticket and outputs the text image with the detected text line position;

S201: Input the text image into the text recognition model for text recognition, recognize the text through the self-attention mechanism of the text recognition model to obtain a recognition result, and structure the recognition result according to the keyword database Extract and get valid information.

The beneficial effects of the present disclosure are: the present invention obtains a text line position detection model by training the CTPN network, thereby locating the key information in the ticket, and is robust to various forms of tickets (tables, etc.); through high The rule-synthesized data of frequent words and specific field text content in them expands the training data of the text recognition model and improves the accuracy of the recognition model; based on the convolutional neural network, it has good parallelism and can utilize high-performance GPU ( Graphics Processing Unit, graphics processor) accelerates calculations.

Description of the drawings

Figures 1 and 2 are flow charts of the model training process of a ticket identification method according to the present invention;

Figures 3 and 4 are flow charts of the text recognition process of a ticket recognition method according to the present invention;

Figure 5 is the structure diagram of the text recognition model;

Figure 6 is a schematic flowchart of text line positioning, text recognition, and structured extraction provided by an embodiment of the present invention.

Detailed ways

The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings. In the description of the present disclosure, it should be understood that the terms "first", "second" and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated. The number of technical characteristics serves only to distinguish the different components.

Figures 1 and 2 are flow charts of the model training process of a ticket recognition method according to the present invention. As shown in Figures 1 and 2, the model training process includes: S100: Collecting data for text line detection and text image recognition. Data; wherein the data includes text line images.

Specifically, to collect data for text line detection and text image recognition, a large number of public, high-precision annotated text line detection sets and text image recognition data containing multiple languages can be obtained by searching in the field of text detection and recognition research. set. Screen out the data that is significantly different from the ticket recognition scenario from the collected data set, mark and eliminate the abnormal data obtained, and use the collected data for the CTPN (Connectionist Text Proposal Network, Connectionist Text Proposal Network) network and recognition network training.

S101: Collect high-frequency words that appear in various ticket scenarios, establish a keyword database based on the high-frequency words, and count the rules of text content in specific fields in the high-frequency words. According to the high-frequency words and the Rules randomly generate augmented data.

Specifically, randomly generating expanded data according to the high-frequency words and the rules includes: (1) combining the high-frequency words whose word frequency is not less than a preset threshold to generate text. (2) Combine the text into a specific format that matches the text in the ticket. (3) Randomly select a blank or noisy image as the background, render the text that conforms to a specific format onto the image, and obtain an image of the text, that is, obtain the expanded data.

The above data, whether it is data or expanded data, is actually image data. The CTPN network and recognition network are directly trained by extracting the characteristics of the image data.

S102: Train the CTPN network through the text line image to obtain a text line position detection model.

S103: Train the recognition network using the data and the expanded data to obtain a text recognition model.

Figures 3 and 4 are flow charts of the text recognition process of a ticket recognition method according to the present invention. As shown in Figures 3 and 4, the text recognition process includes: S200: Input the image of the ticket into the text line position detection model , the text line position detection model detects the text line position in the ticket, and outputs a text image with the detected text line position.

S201: Input the text image into the text recognition model for text recognition, recognize the text through the self-attention mechanism of the text recognition model to obtain a recognition result, and structure the recognition result according to the keyword database Extract and get valid information.

Specifically, performing structured extraction to obtain effective information includes: calculating the edit distance between each keyword and the recognition result, generating an edit distance matrix, and matching the pair with the smallest edit distance for each keyword Recognition results: determine the position of the keyword in the recognition result according to the paired recognition result, and obtain the effective information. When the keyword does not match the pair recognition result, the default value is returned; that is to say, the recognition rate is not 100%. When the keyword cannot match the pair recognition result with the smallest edit distance, it will return Represented by a default value. The output of the deep neural network is matched with the minimum edit distance to obtain keyword information, which effectively improves the reliability of the results.

As a specific implementation, step S102 includes:

S102-1: The CTPN network includes a convolutional neural network, an LSTM (Long Short-Term Memory) network and a 1×1 convolution layer connected in sequence; each text line includes at least two text line components, A plurality of preset anchor boxes with a fixed width of 16 and different heights are preset in the convolutional neural network for positioning the text line component.

S102-2: The initial learning rate of the CTPN network training is 0.001, the momentum is 0.9, and the text line image is put into the CTPN network for training.

In the forward propagation process of the CTPN network, first feature extraction is performed on the input text line image through a convolutional neural network (such as VGG16), and a first feature map of size N×C×H×W is obtained. Then use 3×3 convolution at the position corresponding to each preset anchor box on the first feature map to obtain a second feature map with a size of N×9C×H×W, and then convert the The dimension is transformed to NH×W×9C, and then the second feature map with the dimension NH×W×9C is sent to the LSTM network to learn the sequence features of each row in the second feature map, and the output is NH×W ×256 third feature map, and transform the dimension of the third feature map to N×512×H×W, and finally put the third feature map with dimension N×512×H×W into a 1×1 volume The prediction result is obtained after convolution of the multilayer layer; where N represents the number of text line images processed each time, H represents the height of the text line image, W represents the width of the text line image, and C represents the forward propagation of the text line image in the network. number of channels.

S102-3: After obtaining the prediction result, calculate the loss of the CTPN network according to the first loss function, then use the optimizer SGD (stochastic gradient descent, stochastic gradient descent) to update the parameters of the CTPN network, and then add the text line The image is put into the CTPN network after updated parameters for training, and this process is repeated repeatedly until the best prediction result is obtained, and the best model parameters corresponding to the best prediction result are saved to obtain the text line position detection model;

Wherein _{, the} first loss _{function is :} Loss ₌ λ _v The loss function Smooth L1 Loss between the actual anchor box center point coordinates and height; L _conf represents the confidence loss, that is, the binary cross entropy loss of whether the text line component is included between the preset anchor box confidence and the actual anchor box; L conf _x represents the abscissa offset loss, that is, the loss function Smooth L1Loss between the offset value of the transverse coordinate and width of the text line in the predicted anchor box and the offset value of the transverse coordinate and width of the text line in the actual anchor box; λ _v , λ _conf , λ _x represent weight;

The output results of the text line component at each of the preset anchor box positions include: v _j , v _h , _si , x _side , where v _j and v _h represent the center point of the preset anchor box. coordinates and height, s _i represents the confidence of the text line component included in the preset anchor box, x _side represents the offset value of the horizontal coordinate and width of the text line component.

As a specific implementation, step S103 includes:

S103-1: The recognition network includes a feature extraction network, a feature fusion network, a coding network, a fully connected layer and a decoding algorithm that are connected in sequence, as shown in Figure 5.

S103-2: The initial learning rate of the recognition network is 0.0001, and the beta value of the optimizer Adam is (0.9, 0.999). The data and the expanded data are put into the recognition network for training;

In the forward propagation process of the recognition network, the image with a size of H×W is extracted through the feature extraction network to obtain the first feature;

Then, the first features are fused through the feature fusion network, and the fused first features are sampled so that the height of the fused first features is 1 to obtain the second features;

Input the second feature into the encoding network for encoding to obtain encoding features;

Input the encoding features into the fully connected layer for decoding, and obtain the decoding result;

Finally, the decoding results are aligned through the decoding algorithm to obtain the recognition result;

Wherein, the feature extraction network is a Resnet50 network, the feature fusion network is a FPEM (FeaturePyramid Enhancement Module, feature pyramid enhancement module) network, the encoding network is an Encoder network, and the decoding algorithm is CTC (Connectionist Temporal Classification, connection timing Classification) algorithm, the loss function of the CTC algorithm is Y represents the decoding result, Y' represents the correctly annotated recognition result, t represents the sequence length of the encoding feature, k represents the alignment function of the CTC network, C:k(c)=Y' represents the set All sequences c in C can obtain the correctly labeled recognition result Y' through the CTC algorithm, p represents the probability, and p(c _t |Y) represents the probability of obtaining the sequence c _t of length t under the premise of Y.

The Resnet50 network is a residual network that extracts image visual features, and the FPEM network is a convolutional network that fuses multi-stage image visual features. By fusing multi-stage features, the receptive field of the model can be increased and the accuracy of the model can be improved. The Encoder network is a feature encoding network based on the self-attention mechanism. The use of the self-attention mechanism allows the model to more accurately extract effective information from the features, thus improving the robustness of the text recognition model. The CTC algorithm is a decoding algorithm for the output sequence. For example, the output sequence is "cccaaat", and after alignment with the CTC algorithm, it is "cat".

The Encoder network is the encoder part of the model Transformer that is widely used in the fields of natural language processing and computer vision. This part of the model benefits from the superpositionable encoding module and has excellent feature capture performance. The encoding module includes Multi-Head Attention (multi-head attention). Attention) and Feed Forward (feed forward) two parts, the mathematical expression of the Multi-Head Attention part is as follows:

Multi-Head Attention(x)=x+Self-Attention(FC(x),FC(x),FC(x));

The input of the Encoder is input into the Self-Attention (self-attention) module as Q, K, V after passing through three layers of fully connected layers FC respectively. d _k is the dimension of the input, and T represents the matrix transpose; the feedforward part consists of layer 1 It is composed of a fully connected layer FC, a layer of Relu activation function and a layer of fully connected layer FC.

S103-3: After obtaining the recognition result, calculate the loss of the recognition network through the loss function of the CTC algorithm, then use the optimizer Adam to update the parameters of the recognition network, and then combine the data and the expanded data Invest in the recognition network after updated parameters for training, repeat this process repeatedly until the best recognition result is obtained, and save the best model parameters corresponding to the best recognition result to obtain the text recognition model.

Figure 6 is a schematic flowchart of text line positioning, text recognition, and structured extraction provided by an embodiment of the present invention. A single ticket image is input into a text line position detection model (CTPN model) loaded with optimal parameters to obtain text line detection. As a result, redundant text boxes are filtered out through the confidence threshold, and the text positioning box of the key area on the image is obtained.

When identifying the content of a text line, the height of the text line image is generally adjusted to 32 pixels and then sent to the text recognition model for recognition. This specifically includes: (1) scaling the text line image while maintaining the original aspect ratio. The scaled image Height h'=32, image width w'=w×(h'/h), where w and h are the original width and height of the image. (2) Input a single image into the text recognition model loaded with optimal parameters to obtain the recognition vector. (3) Process the identification vector through the CTC decoding algorithm to obtain the text sequence with the highest confidence.

Then perform structured extraction to obtain effective information, including: (1) Calculating the edit distance between each keyword and the text recognition result. The larger the edit distance, the lower the matching degree; (2) Generating an edit distance matrix for each keyword Find the pairing with the smallest edit distance; (3) Determine the position of the keyword in the recognition result based on the pairing, and obtain the text content. Finally, the key information will be extracted and organized into structured data output according to the corresponding type. If the keyword is not matched, the default value obtained from statistics will be used to supplement it.

The above are exemplary embodiments of the present disclosure, and the protection scope of the present disclosure is defined by the claims and their equivalents.

Claims (3)

1. A method of ticket recognition, characterized by a model training process and a text recognition process, and the model training process includes: S100: Collect data for text line detection and text image recognition; wherein the data includes text line images; S101: Collect high-frequency words that appear in various ticket scenarios, establish a keyword database based on the high-frequency words, and count the rules of field text content in the high-frequency words. According to the high-frequency words and the rules Randomly generate extended data; S102: Train the CTPN network through the text line image to obtain a text line position detection model; S103: Train the recognition network through the data and the expanded data to obtain a text recognition model with a self-attention mechanism; The text recognition process includes: S200: Input the image of the ticket into the text line position detection model. The text line position detection model detects the text line position in the ticket and outputs the text image with the detected text line position; S201: Input the text image into the text recognition model for text recognition, recognize the text through the self-attention mechanism of the text recognition model to obtain a recognition result, and structure the recognition result according to the keyword database Extract and obtain effective information; Wherein, in step S101, extended data is randomly generated according to the high-frequency words and the rules, including: Combine the high-frequency words whose word frequency is not less than a preset threshold to generate text; Combining said text into a specific format consistent with the text in the ticket; Randomly select a blank or noisy image as the background, render the text that conforms to a specific format onto the image, and obtain an image of the text, that is, obtain the expanded data; The step S102 includes: S102-1: The CTPN network includes a convolutional neural network, an LSTM network and a 1×1 convolution layer connected in sequence; each text line includes at least two text line components, which are preset in the convolutional neural network Multiple preset anchor boxes with a fixed width of 16 and different heights are used to position the text line components; S102-2: The initial learning rate of the CTPN network training is 0.001, the momentum is 0.9, and the text line image is put into the CTPN network for training; In the forward propagation process of the CTPN network, first feature extraction is performed on the input text line image through the convolutional neural network to obtain a first feature map of size N×C×H×W, and then in Use 3×3 convolution on the first feature map corresponding to the position of each preset anchor frame to obtain a second feature map with a size of N×9C×H×W, and then transform the dimensions of the second feature map is NH×W×9C, and then the second feature map with the dimension NH×W×9C is sent to the LSTM network to learn the sequence features of each row in the second feature map, and the output is NH×W×256 The third feature map, and the dimension of the third feature map is transformed to N×512×H×W, and finally the third feature map with the dimension of N×512×H×W is put into the 1×1 convolution layer The prediction result is obtained after convolution; among them, N represents the number of text line images processed each time, H represents the height of the text line image, W represents the width of the text line image, and C represents the channel of the text line image in the forward propagation of the network. number; S102-3: After obtaining the prediction result, calculate the loss of the CTPN network according to the first loss function, then use the optimizer SGD to update the parameters of the CTPN network, and then put the text line image into the CTPN network after the updated parameters Perform training, repeat this process repeatedly until the best prediction result is obtained, and save the best model parameters corresponding to the best prediction result, thereby obtaining the text line position detection model; Wherein _{, the} first loss _{function is :} Loss ₌ λ _v The loss function SmoothL1 Loss between the actual anchor box center point coordinates and height; L _conf represents the confidence loss, that is, the binary cross-entropy loss of whether there is a text line component between the preset anchor box confidence and the actual anchor box; L _x Represents the horizontal coordinate offset loss, that is, the loss function Smooth L1 Loss between the offset value of the horizontal coordinate and width of the text line in the predicted anchor box and the offset value of the horizontal coordinate and width of the text line in the actual anchor box; λ _v , λ _conf , λ _x represent weight; The output results of the text line component at each of the preset anchor box positions include: v _j , v _h , _si , x _side , where v _j and v _h represent the center point of the preset anchor box. Coordinates and height, s _i represents the confidence of the text line component included in the preset anchor box, x _side represents the offset value of the horizontal coordinate and width of the text line component; The step S103 includes: S103-1: The recognition network includes a feature extraction network, a feature fusion network, a coding network, a fully connected layer and a decoding algorithm that are connected in sequence; S103-2: The initial learning rate of the recognition network is 0.0001, and the beta value of the optimizer Adam is (0.9, 0.999). The data and the expanded data are put into the recognition network for training; In the forward propagation process of the recognition network, the image with a size of H×W is extracted through the feature extraction network to obtain the first feature; Then, the first features are fused through the feature fusion network, and the fused first features are sampled so that the height of the fused first features is 1 to obtain the second features; Input the second feature into the encoding network for encoding to obtain encoding features; Input the encoding features into the fully connected layer for decoding, and obtain the decoding result; Finally, the decoding results are aligned through the decoding algorithm to obtain the recognition result; Wherein, the feature extraction network is the Resnet50 network, the feature fusion network is the FPEM network, the encoding network is the Encoder network, the decoding algorithm is the CTC algorithm, and the loss function of the CTC algorithm is Y represents the decoding result, Y' represents the correctly annotated recognition result, t represents the sequence length of the encoding feature, k represents the alignment function of the CTC network, C:k(c)=Y' represents the set All sequences c in C can obtain the correctly labeled recognition result Y' through the CTC algorithm, p represents the probability, and p(c _t |Y) represents the probability of obtaining the sequence c _t of length t under the premise of Y; S103-3: After obtaining the recognition result, calculate the loss of the recognition network through the loss function of the CTC algorithm, then use the optimizer Adam to update the parameters of the recognition network, and then combine the data and the expanded data Invest in the recognition network after updated parameters for training, repeat this process repeatedly until the best recognition result is obtained, and save the best model parameters corresponding to the best recognition result to obtain the text recognition model. 2. A ticket identification method as claimed in claim 1, characterized in that in step S201, the structured extraction is performed to obtain effective information, including: Calculate the edit distance between each keyword and the recognition result, generate an edit distance matrix, match the paired recognition result with the smallest edit distance for each keyword, and determine where the keyword is based on the paired recognition result. The position in the recognition result is obtained to obtain the effective information; When the keyword does not match the pair recognition result, the default value is returned. 3. A method of ticket recognition according to claim 2, characterized in that, in the step S201, when the text line image is recognized by the text recognition model, the height of the text line image is adjusted to 32 pixels are then sent to the text recognition model for recognition.