CN111507328A - Text recognition and model training method, system, device and readable storage medium - Google Patents

Abstract

The invention discloses a text recognition and model training method, system, equipment and readable storage medium. In the coding stage of text recognition, the invention extracts image features of a picture to be recognized through a dense convolutional neural network, so that the extracted features are more Abstract, contains richer semantic information; by adding two-dimensional position encoding information to image features, image features containing position information are generated, and the added two-dimensional position encoding can more accurately locate the characters in the image when decoding the image features. position, so that the corresponding text characters can be more accurately recognized, and the accuracy of curved text recognition can be improved; in the decoding stage, the image features containing position information are decoded through the transformer decoding layer containing the two-dimensional attention mechanism. , can make full use of the two-dimensional spatial information of the image, and use a weakly supervised method for training, which can further improve the accuracy of curved text recognition.

Description

Text recognition and model training method, system, device and readable storage medium

technical field

The present invention relates to the technical field of image processing, and in particular, to a text recognition and model training method, system, device and readable storage medium.

Background technique

In daily work or life, it is often necessary to use computer technology to recognize the text on paper documents, for example, the text on various bills, the identity information on the certificate entity, etc. Image-based text recognition has become a computer vision technology. an important research topic.

At present, the recognition of text information printed on paper mainly adopts Optical Character Recognition (Optical Character Recognition, hereinafter referred to as: OCR) technology, which uses optical technology and computer technology to read the text printed or written on paper, and convert it. into a format that humans can understand. The processing steps of OCR mainly include: image preprocessing, layout analysis, text positioning (or image cutting), character cutting and recognition, etc.

Due to the variety of text fonts and shapes in natural scenes, and the existence of occlusion, uneven lighting, excessive noise, etc., especially for many curved texts in natural scenes, such as trademarks, seals, etc., often contain very Important information requires high recognition accuracy. However, the prior art has a low accuracy rate for the recognition of curved text in natural scenes, and how to improve the accuracy of recognition of curved text in natural scenes has become an urgent technical problem to be solved.

SUMMARY OF THE INVENTION

The present invention provides a text recognition and model training method, system, device and readable storage medium to overcome the above technical problems in the prior art and improve the accuracy of curved text recognition in natural scenes.

A text recognition method provided by the present invention includes:

Extract the image features of the image to be recognized through a dense convolutional neural network;

Adding two-dimensional position coding information to the image features to generate image features including position information;

Through the transformer decoding layer including the two-dimensional attention mechanism, the image features including the position information are decoded to obtain the recognition result.

The present invention also provides a text recognition model comprising: an encoding module and a decoding module; the encoding module is used for: extracting image features of a picture to be recognized through a dense convolutional neural network, and adding two-dimensional position encoding information to the image features , to generate image features including position information; the decoding module includes a transformer decoding layer including a two-dimensional attention mechanism, and the transformer decoding layer including a two-dimensional attention mechanism is used to decode the image features including position information. processing to obtain the identification result;

The method includes:

Obtaining a training set for text recognition in natural scenes, the training set includes at least a plurality of pieces of curved text training data, and each piece of the curved text training data includes: a sample picture containing the curved text and its corresponding text annotation information;

The text recognition model is trained through the training set.

The present invention also provides a text recognition system, including:

an encoding module for extracting image features of the picture to be identified through a dense convolutional neural network, adding two-dimensional position encoding information to the image features, and generating image features including position information;

The decoding module is used for decoding the image features including the position information through the transformer decoding layer including the two-dimensional attention mechanism to obtain the recognition result.

The present invention also provides a text recognition device, comprising:

A processor, a memory, and a computer program stored on the memory and executable on the processor; wherein, the processor implements the text recognition method and/or text recognition described above when the processor runs the computer program Model training method.

The present invention also provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program can be executed to execute the above-mentioned text recognition method and/or text recognition model training method.

In the coding stage of text recognition, the present invention extracts the image features of the picture to be recognized through a dense convolutional neural network, so that the extracted features are more abstract and contain more abundant semantic information; by adding two-dimensional position coding to the image features information, generate image features including position information, and the added two-dimensional position code can more accurately locate the position of characters in the image when decoding the image features, so that the corresponding text characters can be more accurately identified, and the curved text can be improved. The accuracy of recognition; in the decoding stage, the image features containing the position information are decoded through the transformer decoding layer including the two-dimensional attention mechanism, which can fully utilize the two-dimensional spatial information of the image and use a weak supervision. The training method can further improve the accuracy of curved text recognition.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a flowchart of a text recognition method provided in Embodiment 1 of the present invention;

2 is a schematic structural diagram of a traditional transformer model provided by Embodiment 1 of the present invention;

3 is a schematic structural diagram of a text recognition model provided in Embodiment 1 of the present invention;

4 is a flowchart of adding two-dimensional position coding according to Embodiment 2 of the present invention;

FIG. 5 is a flowchart for determining a two-dimensional attention vector according to Embodiment 3 of the present invention;

6 is a schematic diagram of a flow chart for determining a two-dimensional attention vector according to Embodiment 3 of the present invention;

7 is a flowchart of a text recognition model training method provided in Embodiment 4 of the present invention;

8 is a schematic structural diagram of a text recognition system according to Embodiment 5 of the present invention;

9 is a schematic structural diagram of a text recognition system according to Embodiment 6 of the present invention;

FIG. 10 is a schematic structural diagram of a text recognition device according to Embodiment 7 of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are the present invention. Some, but not all, embodiments are disclosed. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The terms "first", "second", etc. involved in the present invention are only used for descriptive purposes, and should not be understood as indicating or implying relative importance or implying the number of indicated technical features. In the description of the following embodiments, the meaning of "plurality" is two or more, unless otherwise expressly and specifically defined.

At present, in the field of scene bending text recognition, the difficulty mainly lies in the "alignment" of each text character and the image text area (hereinafter referred to as the "alignment" operation), that is, how to accurately identify the text characters in the image text area. The above "align" operation is relatively simple compared to regular straight text compared to curved text. For the above technical difficulty, the present invention adopts the following four methods to "align" the text area: extracting image features with convolutional neural network, adding two-dimensional position coding to the extracted image features, decoding with transformer The layer (ie, transformer-decoder) extracts the correlation between characters and implements the above-mentioned "alignment" operation with image features. The "alignment" between character features and image features uses a two-dimensional attention module. Among them, the convolutional neural network extracts image features and transformer-decoder as the basic module, the two-dimensional attention module is the core of "alignment" between text characters and image text areas in the transformer-decoder, and the two-dimensional position encoding is for two-dimensional attention. The force module specifically adds processing that enhances the "alignment" effect.

The text recognition method provided in this embodiment is implemented by using a text recognition model, the adopted model architecture is an encoder-decoder architecture, and the text recognition model includes an encoding module and a decoding module. In the encoder stage, the image features of the picture to be recognized are first extracted through the convolutional neural network, and then the two-dimensional position encoding is added. In the decoder stage, the output from the encoder is accepted through the transformer-decoder, and the two-dimensional attention mechanism is used to decode the recognition result.

In order to make the technical solutions of the present invention clearer, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

FIG. 1 is a flowchart of a text recognition method provided by Embodiment 1 of the present invention, FIG. 2 is a schematic structural diagram of a traditional transformer model provided by Embodiment 1 of the present invention, and FIG. 3 is a schematic structural diagram of a text recognition model provided by Embodiment 1 of the present invention . As shown in Figure 1, the text recognition method in this embodiment includes:

Step 10: Extract the image features of the image to be recognized through a dense convolutional neural network.

People’s cognition of images is abstract and layered. First, they understand color and brightness, then local details such as edges, corners, and lines, and then more complex information and structures such as textures and geometric shapes. The concept of the whole object. The research on the visual mechanism of visual neuroscience has verified this conclusion. The visual cortex of the animal brain has a hierarchical structure. The convolutional neural network can be regarded as an imitation of the human visual mechanism. It consists of multiple convolutional layers. Each convolutional layer is a convolutional kernel that scans the image from left to right and from top to bottom, and outputs feature maps. , that is, to extract the local features of the image. With the gradual increase of the convolutional layer, the receptive field (the size of the area mapped by each pixel in the feature map on the input image) gradually increases, and the extracted features are more abstract, and finally the abstract representation of the image at different scales is obtained. Since the convolutional neural network shines at the 2012 imageNet image recognition challenge, the convolutional neural network has continued to develop, and has been widely used in various fields. It has achieved the current best performance on many problems, and has now become the Mainstream for extracting image features.

In this embodiment, a dense convolutional neural network (Dense Convolutional Network, DenseNet for short) is used to extract the image features of the image to be recognized, to obtain the image features of the image to be recognized, that is, the feature map of the image to be recognized.

Exemplarily, before performing text recognition, the text recognition model can be pre-trained, and the encoding module and the decoding module can be trained during the model training process, so as to obtain the trained density used for extracting image features in the encoding stage. Convolutional Neural Network, and a trained Transformer Decoder Layer with a 2D Attention Mechanism.

In this step, the image features of the image to be recognized are extracted through a pre-trained dense convolutional neural network.

Step 20: Add two-dimensional position coding information to the image features to generate image features including position information.

The transformer model is completely composed of the Attention mechanism. The Attention mechanism was proposed by the Bengio team in 2014 and has been widely used in various fields of deep learning in recent years, such as in the direction of computer vision for capturing images. The receptive field of , or used in Natural Language Processing (NLP for short) to locate key tokens or features. Transformer abandons the traditional Convolutional Neural Network (CNN), Recurrent Neural Network (Recurrent Neural Network) , referred to as RNN), the entire network structure is completely composed of the Attention mechanism. However, the Attention mechanism does not contain position information, that is, there is no difference in the transformer when the words in a sentence are in different positions, which is of course unrealistic. Therefore, the introduction of position information in the transformer has a more important role than CNN, RNN and other models, so it is necessary to add a position vector to each feature vector in the image features obtained through the convolutional neural network to add position information. However, the traditional position encoding acts on one dimension, and the image features in this embodiment are two-dimensional, so the one-dimensional position encoding cannot be directly used.

In this embodiment, after the image features (that is, the feature map) of the image to be recognized are extracted, two-dimensional position coding information is added to the image features, which can strengthen the position representation of the two-dimensional space in the image features, thereby further enhancing the The ability to "align" image features with character features.

Step 30: Decode the image features including the position information through the transformer decoding layer including the two-dimensional attention mechanism to obtain the recognition result.

The entire network structure of the traditional transformer model is composed of the Attention mechanism, and the transformer encoding layer is composed of the Self-Attenion layer and the Feedforward Neural Network (FNN for short). As shown in Figure 2, the traditional transformer coding layer (Encoder#1 and Encoder#2 shown in Figure 2 represent two coding layers) includes a Self-Attenion layer and a feedforward network layer (Feed Forward). The traditional transformer decoding Layer (2X in Figure 2 indicates that there are two decoding layers stacked, and each decoding layer is shown in the dashed box in the decoding module part of the figure) including the Self-Attenion layer, the encoder-decoder Attenion layer and the feedforward network layer (FeedForward) , each sublayer in the transformer decoding layer (including the Self-Attenion layer, the Encoder-DecoderAttenion layer and the feedforward network layer) has an "Add&Normalize" layer in the middle, which represents the residual connection and layer normalization step. A transformer-based trainable neural network can be built by stacking transformer layers. The proposal of transformer solves two shortcomings of RNN. RNN-related algorithms can only be calculated sequentially from left to right or from right to left. This feature firstly limits the parallel ability of the model, and secondly, in the process of sequential calculation Particularly long-term dependent phenomena information is lost. The transformer is not a sequential structure similar to RNN, so it has better parallelism, and the distance between any two positions in the sequence is reduced to a constant, which solves the problem of long-term dependence.

The encoding module in this embodiment uses DenseNet to extract image features, and adds two-dimensional position encoding information; the decoding module in this embodiment is formed by stacking multiple transformer decoding layers, and the output of the former transformer decoding layer is used as the latter transformer decoding layer. The input of the layers, each transformer decoding layer includes: a masked multi-head attention layer, a 2D attention layer and a feed-forward neural network layer.

Exemplarily, as shown in FIG. 3 , the transformer decoding module in this embodiment may be formed by stacking two or three transformer decoding layers. “3X” in FIG. 3 indicates that three transformer decoding layers are stacked. The "Masked Multi-Head Attention" means a masked multi-head attention layer, "2D Attention" means a two-dimensional attention layer, and "Feed Forward" means a feed-forward neural network layer.

As shown in Figure 3, each sub-layer in the transformer decoding layer (multi-head attention layer with mask, 2D attention layer and feed-forward neural network layer) has an “Add&Normalize” layer in the middle, which represents the residual connection and Layer normalization step. The transformer decoding module also includes a linear layer (Linear layer) and a Softmax layer.

In addition, for model training, the transformer decoding module also includes an embedding layer. In the training phase, the ground-truth characters are converted into vector representations in a high-dimensional space through the embedding layer, which is used as the input of the first transformer decoding layer.

In this embodiment, the transformer decoding module is used in the decoder stage to extract character features and "align" with the image features extracted in the encoder stage. Different from the traditional transformer decoding module, in this embodiment, the encoder-decoder Attention in the original "alignment" operation is replaced by a two-dimensional attention mechanism to further strengthen the alignment of the character features extracted in the decoding stage and the image features extracted in the encoding stage. capacity, and finally output through a feed-forward neural network.

The traditional Attention module that implements the "alignment" operation in text recognition in natural scenes needs to pool the extracted image features vertically in the encoder stage, which loses the spatial information in the image and cannot make good use of two-dimensional image features. Spatial information. In this embodiment, the 2D Attention mechanism for curved text is used, which can make full use of the spatial information of the image, and use a weakly supervised method to "align" character features and image features to realize the recognition of curved text.

In the coding stage of text recognition, the embodiment of the present invention extracts the image features of the image to be recognized through a dense convolutional neural network, so that the extracted features are more abstract and contain more semantic information; by adding two-dimensional position coding to the image features information, generate image features containing position information, and the added two-dimensional position coding can more accurately locate the position of characters in the image when decoding the image features, thereby improving the accuracy of curved text recognition; in the decoding stage, by including The transformer decoding layer of the two-dimensional attention mechanism decodes the image features containing position information, which can make full use of the two-dimensional spatial information of the image, and use a weakly supervised method for training, which can further improve the accuracy of curved text recognition. Rate.

FIG. 4 is a flowchart of adding two-dimensional position coding according to the second embodiment of the present invention. On the basis of the above-mentioned first embodiment, in this embodiment, as shown in FIG. 4 , the above step 20 adds two Dimension position coding information, and generate image features including position information, which can be achieved by adopting the following steps 201-203:

Step 201: Generate a two-dimensional position code of each pixel in the image feature.

Specifically, this step can be implemented in the following manner:

Determine the position encoding weights in the horizontal and vertical directions according to the image features; for any pixel in the image features, generate the one-dimensional position encoding of the pixel in the horizontal and vertical directions respectively; The position encoding weight is the weighted summation of the one-dimensional position encoding of the pixel in the horizontal direction and the vertical direction to obtain the two-dimensional position encoding of the pixel.

表示，其中下标h表示竖直方向，pos_h为该像素在竖直方向上的排列位置，该像素在水平方向上的一维位置编码可以用

表示，其中下标w表示水平方向，pos_w为该像素在水平方向上的排列位置，特征图的深度可以用D表示，那么该像素在竖直方向的一维位置编码可以采用如下公式一或公式二计算：Exemplarily, for a pixel with a height h and a width w in an image feature (that is, a feature map), the two-dimensional position code of the pixel can be represented by P _hw , and the one-dimensional position code of the pixel in the vertical direction can be expressed by

Representation, where the subscript h represents the vertical direction, pos_h is the arrangement position of the pixel in the vertical direction, and the one-dimensional position encoding of the pixel in the horizontal direction can be used

where the subscript w represents the horizontal direction, pos_w is the arrangement position of the pixel in the horizontal direction, and the depth of the feature map can be represented by D, then the one-dimensional position encoding of the pixel in the vertical direction can use the following formula 1 or formula Second calculation:

表示竖直方向第pos_h个像素对应的一维位置编码向量中的第2i个分量，

表示竖直方向第pos_h个像素对应的一维位置编码向量中的第2i+1个分量，i为非负整数，下标2i和2i+1取值为[0，D-1]，其中D为特征图的深度，即像素对应一维位置编码向量的偶数(2i)分量通过公式一计算该分量值，像素对应一维位置编码向量的奇数(2i+1)分量通过公式二计算该分量值。Among them, pos_h is the arrangement position of the pixel in the vertical direction. For example, if there are 20 pixels in the vertical direction in the feature map, the value of pos_h is [0, 19];

represents the 2ith component in the one-dimensional position encoding vector corresponding to the pos_hth pixel in the vertical direction,

Indicates the 2i+1th component in the one-dimensional position encoding vector corresponding to the pos_hth pixel in the vertical direction, i is a non-negative integer, and the subscripts 2i and 2i+1 are [0, D-1], where D is the depth of the feature map, that is, the pixel corresponds to the even-numbered (2i) component of the one-dimensional position coding vector, and the component value is calculated by formula 1, and the pixel corresponds to the odd-numbered (2i+1) component of the one-dimensional position coding vector. The component value is calculated by formula 2 .

和

在k给定的情况下为线性关系，可以表示竖直方向上像素点的相对位置关系。The reason for taking the trigonometric function is

and

When k is given, it is a linear relationship, which can represent the relative position relationship of pixels in the vertical direction.

Similarly, the one-dimensional position encoding of the pixel in the horizontal direction can be calculated using the following formula 3 or formula 4:

表示水平方向第pos_w个像素对应的一维位置编码向量中的第2i个分量，

表示水平方向第pos_w个像素对应的一维位置编码向量中的第2i+1个分量，i为非负整数，下标2i和2i+1取值为[0，D-1]，其中D为特征图的深度，即像素对应一维位置编码向量的偶数(2i)分量通过公式三计算该分量值，像素对应一维位置编码向量的奇数(2i+1)分量通过公式四计算该分量值。Among them, pos_w is the arrangement position of the pixel in the horizontal direction. For example, if there are 48 pixels in the horizontal direction in the feature map, the value of pos_w is [0, 47];

represents the 2ith component in the one-dimensional position encoding vector corresponding to the pos_wth pixel in the horizontal direction,

Indicates the 2i+1th component in the one-dimensional position encoding vector corresponding to the pos_wth pixel in the horizontal direction, i is a non-negative integer, and the subscripts 2i and 2i+1 are [0, D-1], where D is The depth of the feature map, that is, the pixel corresponding to the even (2i) component of the one-dimensional position coding vector is calculated by formula 3, and the pixel corresponding to the odd (2i+1) component of the one-dimensional position coding vector. The component value is calculated by formula four.

Exemplarily, the following formula 5 can be used to determine the position encoding weight of the image feature in the horizontal direction, and the following formula 6 can be used to determine the position encoding weight of the image feature in the vertical direction:

为可学习线性权重，可以通过模型训练得到优选值，g(E)为整个特征图平均池化后的结果。where α(E) and β(E) represent the position encoding weights in the vertical and horizontal directions, respectively,

In order to learn the linear weight, the preferred value can be obtained through model training, and g(E) is the result of the average pooling of the entire feature map.

Further, after determining the position encoding weights in the horizontal direction and the vertical direction, and the one-dimensional position encoding of the pixel in the horizontal direction and the vertical direction, the following formula 7 can be used to determine the two-dimensional position encoding of the pixel:

表示该像素在竖直方向的一维位置编码，

表示该像素在水平方向的一维位置编码。Among them, P _hw represents the two-dimensional position code of the pixel,

represents the one-dimensional position encoding of the pixel in the vertical direction,

Indicates the one-dimensional position encoding of the pixel in the horizontal direction.

Step 202 , generating a position encoding tensor of image features.

Among them, the two-dimensional position encoding of each pixel is a vector. After obtaining the two-dimensional position encoding vector of each pixel in the image feature, the two-dimensional position encoding vector of all pixels is spliced into a tensor, in which the The position of the two-dimensional position-encoding vector in the tensor corresponds to the position of the pixel in the image.

Step 203: Add the position encoding tensor of the image feature to the image feature to obtain the image feature including the position information.

In this embodiment, for the two-dimensional attention module in the decoding stage, two-dimensional position coding is added to the image features in the encoding stage, and the position representation of the two-dimensional space in the image features is strengthened, so that the two-dimensional attention module can play a better effect. , thereby enhancing the ability to "align" text characters with image text areas.

On the basis of the above Embodiment 1 or Embodiment 2, in this embodiment, as shown in FIG. 3 , the text recognition model includes at least one transformer decoding layer including a two-dimensional attention mechanism, and each transformer decoding layer includes: Code multi-head attention layer, two-dimensional attention layer and feed-forward neural network layer.

Among them, the processing process of each transformer decoding layer includes: processing the input character features through a masked multi-head attention layer to obtain the first character feature; A character feature is determined, a two-dimensional attention vector is determined, and a two-dimensional attention vector is added to the first character feature to obtain a second character feature; the second character feature is input into the feedforward neural network layer.

Further, the first character feature may include one or more feature vectors. If the first character feature includes multiple feature vectors, the two-dimensional attention layer determines the first character feature according to the image feature and the first character feature including position information. A two-dimensional attention vector corresponding to each feature vector of the character feature, and then adding the corresponding two-dimensional attention vector to each feature vector of the first character feature to obtain the second character feature.

FIG. 5 is a flowchart of a text recognition provided in Embodiment 3 of the present invention, and FIG. 6 is a schematic diagram of a flowchart of a two-dimensional attention vector determination provided by Embodiment 3 of the present invention. As shown in FIG. 5 and FIG. 6 , the two-dimensional attention vector is determined by the two-dimensional attention layer according to the image features including the position information and the first character feature. Specifically, the following steps 301-303 can be used to achieve:

Step 301: Perform a first convolution process on the image features including position information to obtain a first tensor of H×W×d, where H, W and d represent the height, width and depth of the first tensor, respectively.

The first convolution process refers to passing the input image features through a 3×3 convolution.

Step 302, the first character feature includes at least one feature vector, and according to the first tensor, determine the weight value of each feature vector of the first character feature with respect to the image feature containing the position information, and each feature vector of the first character feature with respect to the image feature containing the position information. The weight value of the image feature of the position information is the weight value of each pixel of the image feature including the position information.

Specifically, the first character feature may include one or more feature vectors, and in this step, the weight value of each feature vector of the first character feature with respect to the image feature including the position information may be determined according to the first tensor.

In this embodiment, as shown in FIG. 6 , for each feature vector of the first character feature, the weight value of the feature vector with respect to the image feature including the position information can be determined in the following manner:

Perform a second convolution process on the feature vector (through a 1×1 convolution as shown in Figure 6) to obtain a 1×1×d second tensor; expand the height and width of the second tensor, The third tensor of H×W×d is obtained, and the height, width and depth of the third tensor are consistent with the first tensor; the third tensor is added to the first tensor, and the activation function is used for processing ( In Fig. 6, the tanh function is used as an example for illustration), and the fourth tensor of H×W×d is obtained; the third convolution processing is performed on the fourth tensor (as shown in Fig. 6, through a 1× 1 convolution) to obtain a fifth tensor of H×W×1 (not shown in Figure 6); perform two-dimensional softmax processing on the fifth tensor to obtain the weight of the feature vector with respect to the image features containing position information value, which is an H×W×1 tensor.

Among them, the second convolution and the third convolution are different 1×1 convolutions. The activation function may use a hyperbolic tangent function (tanh function), a sigmoid function, or other similar activation functions, which are not specifically limited in this embodiment.

Step 303: The image features including the position information are weighted and summed according to the weight value of each pixel to obtain a two-dimensional attention vector corresponding to each feature vector of the first character feature.

There may be one or more feature vectors of the first character feature, and generally, the number of feature vectors in the first character feature is consistent with the number of characters.

In this step, for each feature vector of the first character feature, the weight value of the feature vector with respect to the image feature including position information is regarded as the weight value of each pixel of the image feature including position information, and the weight value of the image feature including position information The image features are weighted and summed according to the weight value of each pixel point to obtain the two-dimensional attention vector corresponding to the feature vector.

This embodiment provides a specific implementation method for determining the two-dimensional attention vector. In this embodiment, the 2D Attention mechanism for curved text is used, which can make full use of the spatial information of the image and use a weakly supervised method to "align" "Character features and image features, further strengthen the alignment ability of the character features extracted in the decoding stage and the image features extracted in the encoding stage, which can further improve the accuracy of curved text recognition.

FIG. 7 is a flowchart of a method for training a text recognition model according to Embodiment 4 of the present invention. This embodiment provides a method for training a text recognition model. The text recognition model includes: an encoding module and a decoding module; the encoding module is used for: extracting image features of a picture to be recognized through a dense convolutional neural network, and adding a two-dimensional position to the image features Encoding information to generate image features containing position information; the decoding module includes a transformer decoding layer containing a two-dimensional attention mechanism, and the transformer decoding layer containing a two-dimensional attention mechanism is used to decode the image features containing position information to obtain identification. result.

The text recognition model provided in this embodiment is used to implement the text recognition method provided by any of the foregoing embodiments. For details of the specific implementation of the text recognition method, refer to the foregoing embodiment, which will not be repeated in this embodiment.

In this embodiment, before executing the above text recognition method, a text recognition model can be pre-trained. As shown in FIG. 7 , the text recognition model training method specifically includes the following steps:

Step 40: Acquire a training set for natural scene text recognition, the training set includes at least a plurality of pieces of curved text training data, and each piece of curved text training data includes: a sample picture containing the curved text and its corresponding text annotation information.

Exemplarily, in this embodiment, the training set includes as rich as possible sample pictures in natural scenes and their text annotation information.

Step 50: Train the text recognition model through the training set.

The encoding module in this embodiment uses DenseNet to extract image features, and adds two-dimensional position encoding information; the decoding module in this embodiment is formed by stacking multiple transformer decoding layers, and the output of the former transformer decoding layer is used as the latter transformer decoding layer. The input of the layers, each transformer decoding layer includes: a masked multi-head attention layer, a 2D attention layer and a feed-forward neural network layer.

Exemplarily, as shown in FIG. 3 , the transformer decoding module in this embodiment may be formed by stacking two or three transformer decoding layers. “3X” in FIG. 3 indicates that three transformer decoding layers are stacked. The "Masked Multi-Head Attention" means a masked multi-head attention layer, "2D Attention" means a two-dimensional attention layer, and "Feed Forward" means a feed-forward neural network layer.

As shown in Figure 3, each sub-layer in the transformer decoding layer (multi-head attention layer with mask, 2D attention layer and feed-forward neural network layer) has an “Add&Normalize” layer in the middle, which represents the residual connection and Layer normalization step. The transformer decoding module also includes a linear layer (Linear layer) and a Softmax layer.

In addition, the transformer decoding module also includes an embedding layer.

In the training phase, the ground-truth characters are converted into vector representations in a high-dimensional space through the embedding layer, which is used as the input of the first transformer decoding layer.

The text recognition model provided in this embodiment is essentially a classification model, and the final output result of the decoder is a probability tensor. Therefore, in this embodiment, a multi-class cross entropy loss function can be used to calculate the model loss, and the loss function formula is H(p, q )=-∑p(x)logq(x), where p(x) is 1 when it represents the correct answer, 0 otherwise, and q(x) represents the predicted probability of the correct answer item. This way, for each sample, only the part of the loss that predicts the correct term is considered.

In addition, in this embodiment, Adam is used as the optimization method to optimize the model. Adam is a first-order optimization algorithm that can replace the traditional Stochastic Gradient Descent (SGD) process. It can iteratively update based on training data. Neural network weights. Adam algorithm is different from traditional stochastic gradient descent. Stochastic gradient descent maintains a single learning rate (i.e. alpha) to update all weights, and the learning rate does not change during training. Adam designs independent adaptive learning rates for different parameters by calculating the first-order moment estimation and second-order moment estimation of the gradient. At the same time, the Adam algorithm is easy to implement, and has high computational efficiency and low memory requirements. Therefore, in this embodiment, the Adam algorithm is preferably used as the optimizer.

FIG. 8 is a schematic structural diagram of a text recognition system according to Embodiment 5 of the present invention. As shown in FIG. 8 , the text recognition system in this embodiment includes an encoding module 801 and a decoding module 802 .

Specifically, the encoding module 801 is configured to extract image features of the picture to be recognized through a dense convolutional neural network, add two-dimensional position encoding information to the image features, and generate image features including position information.

The decoding module 802 is used for decoding the image features including the position information through the transformer decoding layer including the two-dimensional attention mechanism to obtain the recognition result.

The above-mentioned functional modules are respectively used to complete the operation functions corresponding to the first embodiment of the method of the present invention, and also achieve similar functional effects, and details are not repeated here.

On the basis of the fifth embodiment above, in this embodiment, the encoding module 801 is further configured to: generate a two-dimensional position code of each pixel in the image feature, and generate a position code tensor of the image feature; The encoded tensor is added to the image features to obtain image features containing location information.

Optionally, the encoding module 801 is further configured to: determine the position encoding weights in the horizontal direction and the vertical direction according to the image feature; for any pixel in the image feature, generate a one-dimensional image of the pixel in the horizontal direction and the vertical direction respectively. Position encoding: According to the position encoding weights in the horizontal and vertical directions, the one-dimensional position encoding of the pixel in the horizontal and vertical directions is weighted and summed to obtain the two-dimensional position encoding of the pixel.

In this embodiment, the text recognition model includes at least one transformer decoding layer including a two-dimensional attention mechanism, and each transformer decoding layer includes: a masked multi-head attention layer, a two-dimensional attention layer and a feedforward neural network layer.

Specifically, the decoding module 502 is also used for:

The input character features are processed by the masked multi-head attention layer to obtain the first character feature; the two-dimensional attention layer is used to determine the two-dimensional attention vector according to the image features containing the position information and the first character feature, and in A two-dimensional attention vector is added to the first character feature to obtain the second character feature; the second character feature is input into the feedforward neural network layer.

Optionally, the decoding module 502 is further configured to: perform a first convolution process on the image features including the position information to obtain a H×W×d first tensor, where H, W and d represent the values of the first tensor respectively. height, width and depth; the first character feature includes at least one feature vector, and the weight value of each feature vector of the first character feature with respect to the image feature containing the position information is determined according to the first tensor, and each feature of the first character feature is The weight value of the vector with respect to the image feature containing the position information is the weight value of each pixel of the image feature containing the position information; the image features containing the position information are weighted and summed according to the weight value of each pixel to obtain the first The 2D attention vector corresponding to each feature vector of character features.

Optionally, the decoding module 502 is also used for:

Perform the second convolution process on the feature vector to obtain a second tensor of 1×1×d; expand the height and width of the second tensor to obtain the third tensor of H×W×d, the third tensor It is consistent with the height, width and depth of the first tensor; the third tensor is added to the first tensor, and the activation function is used to process it to obtain the fourth tensor of H×W×d; for the fourth tensor The third convolution process is performed on the quantity to obtain the fifth tensor of H×W×1; the two-dimensional softmax processing is performed on the fifth tensor to obtain the weight value of the feature vector with respect to the image feature containing the position information.

The above-mentioned functional modules are respectively used to complete the operation functions corresponding to the second and third embodiments of the method of the present invention, and also achieve similar functional effects, and the details are not repeated here.

FIG. 9 is a schematic structural diagram of a text recognition system according to Embodiment 6 of the present invention. On the basis of Embodiment 5 above, in another embodiment of the present invention, as shown in FIG. 9 , the text recognition system may further include a model Training module 803. The text recognition model includes: an encoding module 801 and a decoding module 802; the encoding module 801 is used to: extract the image features of the picture to be recognized through a dense convolutional neural network, add two-dimensional position encoding information to the image features, and generate an image containing the position information The decoding module 802 includes a transformer decoding layer including a two-dimensional attention mechanism, and the transformer decoding layer including a two-dimensional attention mechanism is used to decode the image features including the position information to obtain the recognition result. The model training module 803 is used to: obtain a training set for text recognition in natural scenes, the training set includes at least a plurality of pieces of curved text training data, and each piece of curved text training data includes: a sample picture containing curved text and its corresponding text annotation information; The training set trains the text recognition model.

In addition, in another embodiment of the present invention, the model training module may be implemented as a single system.

The above encoding module 801 and the decoding module 802 are used to complete the operation functions corresponding to any one of the first to third method embodiments of the present invention, and the above-mentioned model training module 803 is used to complete the operation functions corresponding to the fourth method embodiment of the present invention. The function effect, the details will not be repeated.

FIG. 10 is a schematic structural diagram of a text recognition device according to Embodiment 7 of the present invention. As shown in FIG. 10 , the device 100 includes a processor 1001 , a memory 1002 , and a computer program stored on the memory 1002 and executable on the processor 1001 .

Wherein, when the processor 1001 runs the computer program, the text recognition method and/or the text recognition model training method provided by any of the above method embodiments is implemented.

Embodiments of the present invention further provide a computer-readable storage medium, such as: ROM/RAM, magnetic disk, optical disk, etc., where the computer-readable storage medium stores a computer program, and the computer program can be used by a terminal device, a computer or A hardware device such as a server executes the above text recognition method and/or text recognition model training method.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims (11)

1. A text recognition method, comprising:

extracting image features of the picture to be identified through a dense convolutional neural network;

adding two-dimensional position coding information to the image characteristics to generate image characteristics containing position information;

and decoding the image features containing the position information through a transform decoding layer containing a two-dimensional attention mechanism to obtain a recognition result.

2. The method according to claim 1, wherein the adding two-dimensional position code information to the image feature to generate the image feature containing the position information comprises:

generating a two-dimensional position code for each pixel in the image feature and generating a position code tensor for the image feature;

and adding the position coding tensor of the image features and the image features to obtain the image features containing the position information.

3. The method of claim 2, wherein generating a two-dimensional position code for each pixel in the image feature comprises:

determining position coding weights in the horizontal direction and the vertical direction according to the image features;

for any pixel in the image characteristics, respectively generating one-dimensional position codes of the pixel in the horizontal direction and the vertical direction;

and according to the position coding weights in the horizontal direction and the vertical direction, carrying out weighted summation on the one-dimensional position codes of the pixel in the horizontal direction and the vertical direction to obtain the two-dimensional position code of the pixel.

4. The method according to any one of claims 1 to 3, comprising at least one of said transformer decoding layers comprising a two-dimensional attention mechanism, each of said transformer decoding layers comprising: a multi-headed attention layer with a mask, a two-dimensional attention layer, and a feedforward neural network layer.

5. The method of claim 4, wherein the decoding the image feature including the position information by a transform decoding layer including a two-dimensional attention mechanism to obtain an identification result comprises:

processing the input character features through a multi-head attention layer with a mask to obtain first character features;

determining a two-dimensional attention vector through a two-dimensional attention layer according to the image feature containing the position information and the first character feature, and adding the two-dimensional attention vector to the first character feature to obtain a second character feature;

inputting the second character feature into the feed-forward neural network layer.

6. The method of claim 5, wherein determining a two-dimensional attention vector from the image feature containing location information and the first character feature by a two-dimensional attention layer comprises:

performing first volume processing on the image features containing the position information to obtain a first tensor of H × W × d, wherein H, W and d respectively represent the height, width and depth of the first tensor;

the first character features comprise at least one feature vector, a weight value of each feature vector of the first character features relative to the image features containing the position information is determined according to the first vector, and the weight value of each feature vector of the first character features relative to the image features containing the position information is the weight value of each pixel point of the image features containing the position information;

and weighting and summing the image features containing the position information according to the weight value of each pixel point to obtain the two-dimensional attention vector corresponding to each feature vector of the first character features.

7. The method according to claim 6, wherein determining a weight value of any one feature vector of the first character features with respect to the image features containing the position information according to the first vector comprises:

performing second convolution processing on the eigenvector to obtain a second tensor of 1 × 1 × d;

expanding the height and width of the second tensor to obtain a third tensor of H × W × d, wherein the height, the width and the depth of the third tensor are consistent with those of the first tensor;

adding the third tensor to the first tensor, and processing by adopting an activation function to obtain a fourth tensor of H × W × d;

performing a third convolution processing on the fourth tensor to obtain a fifth tensor of H × W × 1;

and performing two-dimensional softmax processing on the fifth tensor to obtain a weight value of the feature vector about the image feature containing the position information.

8. A method for training a text recognition model, wherein the text recognition model comprises: an encoding module and a decoding module; the encoding module is configured to: extracting image features of a picture to be identified through a dense convolutional neural network, adding two-dimensional position coding information into the image features, and generating image features containing position information; the decoding module comprises a transformer decoding layer containing a two-dimensional attention mechanism, and the transformer decoding layer containing the two-dimensional attention mechanism is used for decoding the image features containing the position information to obtain an identification result;

the method comprises the following steps:

acquiring a training set for natural scene text recognition, wherein the training set at least comprises a plurality of pieces of bent text training data, and each piece of bent text training data comprises: the method comprises the steps of obtaining a sample picture containing a bent text and corresponding text marking information;

and training the text recognition model through the training set.

9. A text recognition system, comprising:

the coding module is used for extracting image features of the picture to be identified through a dense convolutional neural network, adding two-dimensional position coding information into the image features and generating image features containing position information;

and the decoding module is used for decoding the image features containing the position information through a transform decoding layer containing a two-dimensional attention mechanism to obtain an identification result.

10. A text recognition apparatus, comprising:

a processor, a memory, and a computer program stored on the memory and executable on the processor;

wherein the processor, when running the computer program, implements the method of any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which can be executed to perform the method according to any one of claims 1 to 8.

Images