CN114581906B - Method and system for text recognition in natural scene images - Google Patents

Abstract

The invention relates to the technical field of data identification, and discloses a text identification method and a text identification system for natural scene images, wherein the method comprises the following steps: acquiring a natural scene image to be identified; performing text recognition on the natural scene image to be recognized by adopting the trained deep learning model to obtain a recognized text; firstly, correcting a natural scene image to be recognized by a deep learning model, and then extracting a feature vector of the image after correction; and then respectively extracting visual features and semantic features from the feature vectors of the image, performing feature fusion on the two features, and finally performing text recognition on the fused features. The method can identify the scene texts in any shapes, has wide application scenes and strong generalization capability of the model, and can be applied to various scenes of text identification.

Description

Method and system for text recognition in natural scene images

technical field

The invention relates to the technical field of data recognition, in particular to a text recognition method and system for natural scene images.

Background technique

The statements in this section merely provide background related to the present disclosure and do not necessarily constitute prior art.

Text recognition is one of the branches of computer vision research, which belongs to pattern recognition and artificial intelligence, and is an important part of computer science. It uses computer technology to recognize text in natural scenes and convert it into a format that can be displayed by computers and understood by humans. Through text recognition, information processing speed can be greatly accelerated.

Due to the sequential nature of the text itself, there is a huge semantic gap between two-dimensional text images and one-dimensional sequential texts in the task of scene text recognition, which also makes scene text recognition uniquely difficult. Human recognition of text in a scene is influenced by a variety of factors, such as context, color, and even culture.

For example, when people see McDonald's iconic symbol with a yellow label on a white background, they will subconsciously know that it is McDonald's. Often people can judge what the text here is without reacting to what is written after seeing the scene for the first time. But for computers, these complex scenes are side effects for text recognition. In reality, the scene is extremely complex, and there are huge differences in fonts, styles, backgrounds, etc. These interferences make the computer often mistakenly recognize certain characters, and the recognition of a wrong character will often lead to the recognition result completely deviating from the meaning of the text.

The current text recognition methods can be roughly divided into two directions. One direction is to use the powerful feature extraction capabilities of neural networks in the deep learning era to optimize visual features to obtain higher feature extraction capabilities, and the other direction is based on text. It has its own semantics, and the features extracted by the feature extractor from the two-dimensional visual image are semantically enhanced or the results obtained by the visual model are corrected.

In recent years, the models with the best results are mostly semantic-based models. These methods usually first use a feature extractor to extract a two-dimensional image into a visual feature map, and then use the semantic model to further encode the feature map extracted by the visual model. , to get semantic features. Then, the visual features and semantic features are comprehensively used to obtain the final recognition result. But in this way, the semantic model is highly dependent on visual features.

This way of coupling semantic features with visual features has two disadvantages: First, the semantic model is only used to correct the results obtained by the visual model. The semantic model is trained end-to-end in the entire process, but it is actually used as a post-processing method independent of the entire model, which leads to a larger model, which makes it difficult to train the gradient chain. Second, using the semantic model to correct the results of the visual model, the recognition ability of the model is indeed enhanced through this structure, but there are various erroneous text information in natural scenes, for example, for the recognition and correction of handwritten test papers, for errors After using the semantic model, the model will automatically correct it to the correct word, which undoubtedly deviates from the original intention of the correction work.

SUMMARY OF THE INVENTION

In order to solve the deficiencies of the prior art, the present invention provides a text recognition method and system for natural scene images, and proposes a semantically independent text recognition network, which is different from the previous model that only uses the truncated gradient to decouple the visual and semantic models. Instead, it is adjusted from the model structure to achieve a complete decoupling of the structure. A new semantic module is designed to make full use of semantic information. A new visual-semantic feature fusion module is designed, which makes visual features and semantic features fully interact, and makes full use of visual features and semantic features. A gate mechanism is used to fuse the enhanced visual information with the semantic information to obtain the final prediction result.

In a first aspect, the present invention provides a text recognition method for natural scene images;

Text recognition methods for natural scene images, including:

Obtain the natural scene image to be recognized;

For the natural scene image to be recognized, the trained deep learning model is used for text recognition, and the recognized text is obtained;

Among them, the deep learning model firstly corrects the natural scene image to be recognized, and then extracts the feature vector of the image after correction; then extracts visual features and semantic features from the feature vectors of the image, respectively. The features are fused, and finally the fused features are used for text recognition.

In a second aspect, the present invention provides a text recognition system for natural scene images;

Text recognition systems for natural scene images, including:

an acquisition module, which is configured to: acquire a natural scene image to be recognized;

A recognition module, which is configured to: use the trained deep learning model to perform text recognition on the natural scene image to be recognized, and obtain the recognized text;

Among them, the deep learning model firstly corrects the natural scene image to be recognized, and then extracts the feature vector of the image after correction; then extracts visual features and semantic features from the feature vectors of the image, respectively. The features are fused, and finally the fused features are used for text recognition.

Compared with the prior art, the beneficial effects of the present invention are:

1) The present invention can recognize scene texts of any shape, has a wide range of application scenarios, and has strong generalization ability of the model, and can be applied to various text recognition scenarios.

2) The present invention adjusts the model structure, thereby reducing the coupling degree of the semantic module to the visual module, making it an independent part, thereby making the model easier to train end-to-end without requiring complex multi-stage or pre-training processes.

3) The present invention proposes a new semantic module for text recognition, through which the semantic information of text can be processed equivalently with the visual module, and preliminary semantic recognition results can be obtained in this branch to guide model training .

4) The present invention proposes a new visual-semantic fusion method, which enables the interaction of equivalent semantic features and visual features, and can make full use of the information of both, and uses the gate mechanism to make the final decision on the information fused by the two, thereby get better recognition results.

Description of drawings

The accompanying drawings forming a part of the present invention are used to provide further understanding of the present invention, and the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute an improper limitation of the present invention.

Fig. 1 is the functional module schematic diagram of the whole network;

Figure 2 is a schematic diagram of feature fusion of visual features and semantic features.

Detailed ways

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

Embodiments of the invention and features of the embodiments may be combined with each other without conflict.

All data acquisition in this embodiment is based on compliance with laws and regulations and the user's consent, and the legal application of the data.

Example 1

This embodiment provides a text recognition method for natural scene images;

Text recognition methods for natural scene images, including:

S101: Obtain a natural scene image to be recognized;

S102: For the natural scene image to be recognized, use the trained deep learning model to perform text recognition to obtain the recognized text;

Among them, the deep learning model firstly corrects the natural scene image to be recognized, and then extracts the feature vector of the image after correction; then extracts visual features and semantic features from the feature vectors of the image, respectively. The features are fused, and finally the fused features are used for text recognition.

Further, as shown in FIG. 1, the deep learning model includes: a correction module, the correction module is connected with the input end of the backbone network, and the output end of the backbone network is respectively connected with the input end of the visual feature extraction module and the semantic The input end of the feature extraction module is connected; the output end of the visual feature extraction module and the output end of the semantic feature extraction module are both connected with the input end of the visual semantic feature fusion module, and the output end of the visual semantic feature fusion module is connected to the prediction The input end of the module is connected, and the output end of the prediction module outputs the text recognition result.

Further, the rectification module is implemented by using a thin plate spline interpolation algorithm TPS (Thin Plate Spline), and is used to rectify the curved image into a regular shape.

It should be understood that curved texts widely exist in natural scenes, and these dramatic changes in image background, appearance, and layout bring significant challenges to text recognition tasks, which traditional Optical Character Recognition (OCR) methods cannot handle effectively. It is difficult to train a recognition model that can cope with various scenarios. Therefore, in this embodiment, the thin plate spline interpolation algorithm TPS is used to correct the image to obtain relatively regular text. The key problem of the thin plate spline interpolation algorithm TPS is to locate 2J control points and the transformation matrix between the control points, where J is a hyperparameter. These 2J control points are obtained by means of regression. Specifically, after the input image is sent to a 3-layer convolutional neural network to extract image features, 2J outputs are obtained using the full connection operation, and these 2J outputs correspond to its control point. The transformation matrix is the analytical solution obtained by the norm between the pixel point and the control point and the norm between the control points. Through this step, a preliminary corrected text image can be obtained.

Further, the backbone network is implemented by a ResNet (Residual Neural Network) convolutional neural network, which is used to extract the spatial features of natural scene images, and the spatial features are embedded with position information by using position coding. The spatial features of the embedded position information are enhanced to obtain the feature vector of the image.

Among them, the position code refers to:

；（1.1）

; (1.1)

；（1.2）

; (1.2)

指的就是对应的位置编码，是一个一维的向量。in,

Refers to the corresponding position code, which is a one-dimensional vector.

对应的是第

个位置，第

个维度的值。in,

corresponding to the

position, the

dimension value.

；（1.3）

; (1.3)

、键

、值

三者都是自身，这种方式可以深度挖掘出特征图内部特征的关系。公式化描述如下：It should be understood that, due to its style background, font style, and noise in the shooting process, the scene text image usually adopts a deep convolutional neural network as the image encoder. In this embodiment, ResNet45 is selected as the backbone network. The residual connection of the network reduces the model degradation caused by the deep network, and also reduces the problem of the disappearance of the network gradient. In addition, due to the sequential nature of the text, this embodiment uses the position encoding and Transformer to obtain enhanced features after using the residual network. Using the position encoding can introduce the position information of the image, so that the neural network pays more attention to the features in the two-dimensional feature map. The positional relationship between, for Transformer, the query here

,key

,value

All three are themselves, and in this way, the relationship between the internal features of the feature map can be deeply excavated. The formulation is described as follows:

；（1.4）

; (1.4)

是公式(1.1)中得到的位置编码，

指的是输入图像，

指的是残差神经网络，

指的是Transformer网络，该步的操作指的是将图像输入ResNet45网络中后得到了特征，将特征与位置编码相加后送Transformer网络中。in,

is the position code obtained in formula (1.1),

refers to the input image,

refers to the residual neural network,

Refers to the Transformer network. The operation of this step refers to inputting the image into the ResNet45 network to obtain features, adding the features and position codes and sending them to the Transformer network.

Further, the visual feature extraction module adopts the second Transformer neural network to separate the visual part and the semantic part of the image feature vector, and uses the position attention mechanism module to decode the visual part to obtain visual features;

、键

、值

替换为不同元素；其中，查询

被替换为位置编码，键

被替换为UNet网络的输出值，值

被替换为

的恒等映射。The position attention mechanism module is a query of the self-attention mechanism self-attention

,key

,value

replaced with a different element; where the query

is replaced by the position code, key

is replaced by the output value of the UNet network, value

is replaced with

The identity map of .

For the location attention mechanism, see Equation (1.6)-Equation (1.8).

Among them, after the visual features obtained by the second Transformer network are decoded using the position attention mechanism module, the fully connected layer is used to predict the decoding results, and the cross-entropy loss in formula (1.10) is used for supervised training.

得到了

，公式如下所示：Exemplarily, in the visual feature extraction module, the Transformer is used in this embodiment to further extract high-dimensional visual features, thereby reducing the dependence of the next decoding on the backbone network, so that the decoding of the semantic part is decoupled from the decoding of the visual part. The features of the visual part are two-dimensional features and the features of the semantic part are one-dimensional result sequences, which is the fundamental difference between the two. Through this step according to the characteristics

Got

, the formula is as follows:

；（1.5）

; (1.5)

都为公式（1.4）中得到

的恒等映射，

是特征的维度，

是超参数，

指的是

函数。通过该步，使得视觉部分提取更深的解码，使得视觉部分的解码与语义部分分离。Among them, in formula (1.5)

Both are obtained in formula (1.4)

The identity mapping of ,

is the dimension of the feature,

are hyperparameters,

Refers

function. This step enables the visual part to extract deeper decoding, so that the decoding of the visual part is separated from the semantic part.

、

和

使用的都是

的恒等映射，公式（1.8）中

、

和

都使用各不相同的编码方式。Next, use the position attention mechanism module to convert the visual features into character sequences. The self-attention formula is also used here, but it is different from (1.5)

,

and

are used

The identity map of , in Equation (1.8)

,

and

All use different encodings.

与公式（1.6）中的

不同，公式（1.5）的查询旨在编码二维视觉特征的位置关系因此实际采用的是特征

，而这里的

编码的则是单词中字符之间的序列关系，因此是对位置顺序采用了词嵌入，公式化描述如下：Equation (1.5)

with the formula (1.6)

Different, the query of formula (1.5) aims to encode the positional relationship of the two-dimensional visual features, so the feature is actually used

, and here the

The encoding is the sequence relationship between the characters in the word, so the word embedding is used for the position order, and the formula is described as follows:

（1.6）

(1.6)

使用的是词嵌入函数，

是字符的顺序，包括[0,1,…N]，

就是对应的位置编码。in,

The word embedding function is used,

is the sequence of characters, including [0,1,…N],

is the corresponding position code.

则是使用了UNet网络，这里的UNet网络没有使用字符级别的分割来引导训练，而是作为一种特征增强的方式，这里通过UNet网络得到与原特征图大小一样的新的特征，公式化描述如下：

The UNet network is used. The UNet network here does not use character-level segmentation to guide training, but as a way of feature enhancement. Here, new features of the same size as the original feature map are obtained through the UNet network. The formula is described as follows :

（1.7）

(1.7)

是公式（1.5）得到的特征，

是UNet网络，

是得到的键。

则是使用了恒等映射，使用的特征

。in,

is the feature obtained by formula (1.5),

is the UNet network,

is the obtained key.

Then the identity mapping is used, and the features used

.

，对其进行解码后，得到视觉部分的结果。Similarly, by using the self-attention formula, the final feature map of the visual part is obtained

, after decoding it, the result of the visual part is obtained.

；（1.8）

; (1.8)

；（1.9）

; (1.9)

对应的是公式（1.6）中的

，

对应的是公式（1.7）中的

，

是

的恒等映射，得到的

是视觉部分最终的特征；公式（1.8）中

是全连接函数，

是公式（1.8）中得到的输出，

指的是空间维度，

是单词的最大长度，这里是预定义的值，

是字符的类别数，同样是预定义的值，

是视觉部分的识别结果，

表示全体实数的集合。Among them, in formula (1.8)

Corresponding to the formula (1.6) in

,

Corresponds to the formula (1.7) in

,

Yes

The identity mapping of , we get

is the final feature of the visual part; in Equation (1.8)

is the fully connected function,

is the output obtained in Equation (1.8),

refers to the spatial dimension,

is the maximum length of a word, here is the predefined value,

is the number of categories of characters, again a predefined value,

is the recognition result of the visual part,

represents the set of all real numbers.

In this embodiment, the cross-entropy loss is introduced to guide the training formula of the visual part as follows:

；（1.10）

; (1.10)

指的就是公式（1.9）中

第

个时间片，

就是真实的标签，

对应的就是单词的长度。

就是对每一个预测出的字符，将其与真实值做交叉熵损失。in,

Refers to the formula (1.9) in

the first

time slice,

is the real label,

Corresponds to the length of the word.

It is to perform cross-entropy loss on each predicted character with the real value.

Further, the semantic feature extraction module adopts the attention mechanism module to align the feature vectors of the image, and then decode the aligned data to obtain the semantic features.

The alignment processing refers to aligning two-dimensional features into one-dimensional features.

Among them, the aligned data is decoded by using a long short-term memory network (Long Short-Term Memory, LSTM).

Among them, the long short-term memory network uses a fully connected layer to predict semantic features, and uses a second cross-entropy loss function for supervised training.

Exemplarily, due to the coupling of the semantic feature extraction module to the visual feature extraction module, the training of the semantic feature extraction module greatly depends on the visual feature extraction module. Therefore, in this embodiment, the semantic feature extraction module is independent, and the backbone network is independent. features to encode, thereby reducing the degree of coupling of the model.

，使得注意力机制不仅仅关注于有判别力的区域，而且关注图像之间文本的位置关系，公式化描述如下：The semantic feature extraction module first uses the attention mechanism to align the two-dimensional features obtained by the backbone network to make them into aligned one-dimensional features. In addition, this embodiment adds the introduction of position information into the attention mechanism.

, so that the attention mechanism not only pays attention to the discriminative regions, but also pays attention to the positional relationship of the text between the images, which is formulated as follows:

；（2.1）

; (2.1)

；（2.2）

; (2.2)

，

，

都是可训练参数，

是字符的顺序，包括[0,1,…N]，

做的是词嵌入操作，

是公式（1.4）中得到的特征向量；公式（2.2）中

指的是以

为底的指数函数，括号中的值是公式（2.1）得到的结果，也就是

的指数，

是预设的单词最大长度，全文其他的

都是同一个超参数，

得到的就是

时刻位置

的权重，通过将权重乘回特征序列

，可以得到已对齐的一维序列特征公式化描述如下：Among them, in formula (2.1)

,

,

are all trainable parameters.

is the sequence of characters, including [0,1,…N],

What it does is the word embedding operation,

is the eigenvector obtained in formula (1.4); in formula (2.2)

refers to

is the base exponential function, the value in parentheses is the result obtained by formula (2.1), which is

index,

is the preset maximum length of words, the full text is other

are the same hyperparameter,

what you get is

time position

the weights of , by multiplying the weights back into the feature sequence

, the aligned one-dimensional sequence features can be formulated as follows:

；（2.3）

; (2.3)

是公式（1.4）得到的特征向量，

是公式（2.2）得到的权重，

就是得到的对齐的一维序列。in,

is the eigenvector obtained from formula (1.4),

is the weight obtained by formula (2.2),

is the resulting aligned one-dimensional sequence.

After obtaining the aligned one-dimensional sequence features, use Long Short Term Memory (LSTM) to decode them. Due to the time factor, auto-regressive decoding is not used here, but the results are obtained. One-step decoding, the specific formulation is described as follows:

；（2.4）

; (2.4)

；（2.5）

; (2.5)

；（2.6）

; (2.6)

；（2.7）

; (2.7)

；（2.8）

; (2.8)

；（2.9）

; (2.9)

；（2.10）

; (2.10)

，

，

是LSTM对应的三个门控

对应的是

函数，所有的

与

都是用于学习的参数，

对应着上一时间片的隐藏层状态，

是公式（2.3）得到的结果，对应的是

时刻的输入，

是

函数，

对应的是点乘，

是LSTM的输出。in,

,

,

are the three gates corresponding to LSTM

corresponds to

function, all

and

are all parameters used for learning,

Corresponding to the hidden layer state of the previous time slice,

is the result obtained by formula (2.3), which corresponds to

time input,

Yes

function,

Corresponding to the point multiplication,

is the output of the LSTM.

After it is fully connected, the final recognition result is obtained:

；（2.11）

; (2.11)

是公式（2.10）中LSTM的输出，

是全连接层，

是语义模型最终的预测结果，

指的是空间维度，

是单词的最大长度，这里是预定义的值，

是字符的类别数，同样是预定义的值，

是解码后的视觉特征。in,

is the output of the LSTM in Equation (2.10),

is a fully connected layer,

is the final prediction result of the semantic model,

refers to the spatial dimension,

is the maximum length of a word, here is the predefined value,

is the number of categories of characters, again a predefined value,

is the decoded visual feature.

Through the decoding of LSTM, the decoded semantic features are obtained here, and the recognition results of the semantic parts are obtained after alignment and full connection. Similarly, for the semantic model, cross entropy is used to supervise the recognition results of this part. The formula is described as follows:

；（2.12）

; (2.12)

是公式（2.10）得到的

时刻的预测值，

对应的是

时刻的真实值。

是字符的长度，

指的就是求每一个预测的字符与真实字符之间的损失。in,

is obtained by formula (2.10)

the predicted value at the moment,

corresponds to

The true value of the moment.

is the length of characters,

It refers to the loss between each predicted character and the real character.

Further, the visual semantic feature fusion module is used to interact with the visual feature and the semantic feature to obtain the enhanced visual feature and the enhanced semantic feature; adopt the gate mechanism decision to fuse the enhanced visual feature and the enhanced semantic feature .

与语义特征

。为了充分利用两者的特征，本实施例采取一种新的视觉语义特征交互方式，具体的，同样遵循上面的self-attention的范式，不同的是这里是用于视觉特征与语义特征的融合，因此在交互时使用一个特征做查询

，另一个特征做索引

与键值

，

对应的是query也就是查询，key对应的是索引

，value对应的是键值

。图像化表示如图2，具体的公式化描述如下：Exemplarily, through the visual feature extraction module and the semantic feature extraction module, the visual features are obtained by decoding respectively.

with semantic features

. In order to make full use of the features of the two, this embodiment adopts a new visual-semantic feature interaction method. Specifically, it also follows the above self-attention paradigm. The difference is that it is used for the fusion of visual features and semantic features. So use a feature to query when interacting

, another feature to index

with key value

,

The corresponding query is the query, and the key corresponds to the index

, value corresponds to the key value

. The graphical representation is shown in Figure 2, and the specific formulation is described as follows:

；（3.1）

; (3.1)

；（3.2）

; (3.2)

对应的是公式（2.10）得到的语义特征，

对应的是公式（1.8）得到的视觉特征，

是特征的维度，

是一个超参数，

是

函数，

与

为视觉主导的融合向量与语义主导的融合向量。in,

Corresponding to the semantic features obtained by formula (2.10),

Corresponding to the visual features obtained by formula (1.8),

is the dimension of the feature,

is a hyperparameter,

Yes

function,

and

It is the visual-dominant fusion vector and the semantic-dominant fusion vector.

Through this step, the visual features and semantic features of the interaction are obtained. This method makes full use of the visual information and semantic information. Then, the gate mechanism is used to fuse the interactive features of the two. The specific formula is as follows:

；（3.3）

; (3.3)

；（3.4）

; (3.4)

是可训练参数，

与

为公式（3.1）和公式（3.2）得到的结果，

是将

与

进行级联。公式（3.4）中

是最终融合得到的向量，

是公式（3.3）得到的结果，

与

为公式（3.1）与公式（3.2）得到的结果，最后对

进行全连接，得到了最终结果

，公式化描述如下：Among them, in formula (3.3)

are trainable parameters,

and

The results obtained for Equation (3.1) and Equation (3.2),

will

and

to cascade. In formula (3.4)

is the final fusion vector,

is the result obtained by formula (3.3),

and

For the results obtained from formula (3.1) and formula (3.2), finally

Perform full connection and get the final result

, which is formulated as follows:

；（3.5）

; (3.5)

是全连接函数，

是公式（3.4）的输出，

是最终的预测结果。in,

is the fully connected function,

is the output of Equation (3.4),

is the final prediction result.

After the final result is obtained, the cross-entropy loss is used to supervise the training of the model. The formulation is described as follows:

；（3.6）

; (3.6)

指的是公式（3.5）中得到的结果，

也就对应着第

的时间片预测的结果，其实也就是第

个字符的预测结果。

就是第

个字符的真实值，这里的公式，同公式（1.10）和公式（2.11）一样，

是融合模块使用交叉熵损失来监督模型的训练。in,

refers to the result obtained in Equation (3.5),

which corresponds to the

The result of the time slice prediction is actually the first

predictions for characters.

is the first

The real value of characters, the formula here is the same as formula (1.10) and formula (2.11),

It is the fusion module that uses the cross-entropy loss to supervise the training of the model.

Further, the prediction module identifies the enhanced visual feature and the enhanced semantic feature to obtain the final text recognition result;

For enhanced visual features and enhanced semantic features, a fully connected layer is used to obtain the overall prediction results, and a third cross-entropy loss is used to supervise training.

Further, the deep learning model after the training, the training process includes:

constructing a training set; the training set is a natural scene image with known text recognition results;

Input the training set into the deep learning model, and train the model. When the total loss function value is the smallest, or when the set number of iterations is reached, the training is stopped, and the trained deep learning model is obtained;

Among them, the total loss function is the summation result of the first, second and third cross-entropy loss functions.

Among them, the total loss function is:

；（3.7）

; (3.7)

，

，

是用于平衡的超参数，

是视觉部分的损失，

是语义部分的损失，

是融合损失。文本识别损失函数用的都是交叉熵损失函数。The objective function consists of three parts, among which,

,

,

are the hyperparameters used for balancing,

is the loss of the visual part,

is the loss of the semantic part,

is the fusion loss. The text recognition loss function uses the cross entropy loss function.

Further, the construction of the training set includes:

Get natural scene images;

Augmenting natural scene images;

Normalize the size of the augmented natural scene image.

Exemplarily, the reason why it is necessary to normalize the size of the augmented natural scene image is because the text images in the natural scene have various shapes and aspect ratios, in order to obtain better generalization performance. In this embodiment, the width and height of the text image are respectively set to the default width and height.

Exemplarily, the reason why it is necessary to perform augmentation processing on natural scene images is due to the widespread problems of rotation, perspective distortion, and blurring in natural scenes. This embodiment uses a probability preset value to randomly add Gaussian noise to the original image. , rotation, perspective distortion and other image augmentation methods.

For the real text label of an image, a character dictionary containing all English characters, an array of 0-9, and an end character is used. Since the task is to classify sequences, this embodiment uses a length preset value to specify the maximum length of the text.

Embodiment 2

This embodiment provides a text recognition system for natural scene images;

Text recognition systems for natural scene images, including:

an acquisition module, which is configured to: acquire a natural scene image to be recognized;

A recognition module, which is configured to: use the trained deep learning model to perform text recognition on the natural scene image to be recognized, and obtain the recognized text;

Among them, the deep learning model firstly corrects the natural scene image to be recognized, and then extracts the feature vector of the image after correction; then extracts visual features and semantic features from the feature vectors of the image, respectively. The features are fused, and finally the fused features are used for text recognition.

It should be noted here that the above-mentioned acquisition module and identification module correspond to steps S101 to S102 in the first embodiment, and the examples and application scenarios implemented by the above-mentioned modules and the corresponding steps are the same, but are not limited to those disclosed in the above-mentioned first embodiment. content. It should be noted that the above modules may be executed in a computer system such as a set of computer-executable instructions as part of the system.

The description of each embodiment in the foregoing embodiments has its own emphasis. For the part that is not described in detail in a certain embodiment, reference may be made to the relevant description of other embodiments.

The proposed system can be implemented in other ways. For example, the system embodiments described above are only illustrative. For example, the division of the above modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into other A system, or some feature, can be ignored, or not implemented.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (3)

1. The text recognition method of the natural scene image is characterized by comprising the following steps:

acquiring a natural scene image to be identified;

performing text recognition on the natural scene image to be recognized by adopting the trained deep learning model to obtain a recognized text;

firstly, correcting a natural scene image to be recognized by a deep learning model, and then extracting a feature vector of the image from the corrected image; respectively extracting visual features and semantic features from the feature vectors of the image, performing feature fusion on the two features, and finally performing text recognition on the fused features;

the deep learning model comprises: the correction module is connected with the input end of a backbone network, and the output end of the backbone network is respectively connected with the input end of the visual feature extraction module and the input end of the semantic feature extraction module; the output end of the visual characteristic extraction module and the output end of the semantic characteristic extraction module are both connected with the input end of the visual semantic characteristic fusion module, the output end of the visual semantic characteristic fusion module is connected with the input end of the prediction module, and the output end of the prediction module outputs a text recognition result;

the backbone network is realized by adopting a ResNet convolution neural network and is used for extracting spatial features of a natural scene image, position information embedding is carried out on the spatial features by adopting position coding, and feature enhancement is carried out on the spatial features after the position information embedding by adopting a first Transformer neural network to obtain feature vectors of the image;

the visual feature extraction module is used for separating a visual part and a semantic part of the image feature vector by adopting a second transform neural network, and decoding the visual part by adopting a position attention mechanism module to obtain visual features; the position attention mechanism module is used for inquiring self-attention mechanism self-attention

Key, key

Sum value

Replacing with different elements; wherein the query

Is replaced by position code, key

Is replaced by the output value, value of the UNet network

Is replaced by

Identity mapping of (2);

；（1.4）

wherein,

is a position code, and the position code is,

it is referred to as an input image,

refer to a residual neural network that is,

refers to a Transformer network;

；（1.5）

wherein, in the formula (1.5)

Are all obtained from the formula (1.4)

The identity of the image to be scanned is mapped,

is the dimension of the feature that is,

is a hyper-parameter which is the parameter,

refer to

A function;

（1.7）

wherein,

is a characteristic obtained by the formula (1.5),

is a network of UNet's, and,

is the resulting bond;

the semantic feature extraction module adopts an attention mechanism module to align the feature vectors of the images, and then decodes the aligned data to obtain semantic features; wherein, the alignment processing refers to aligning the two-dimensional features into one-dimensional features; the aligned data is decoded and realized by adopting a long-term and short-term memory network.

2. The method for recognizing texts in images of natural scenes according to claim 1, wherein the visual semantic feature fusion module is configured to interact with the visual features and the semantic features to obtain enhanced visual features and enhanced semantic features; and adopting a door mechanism decision to fuse the enhanced visual features and the enhanced semantic features.

3. The text recognition system for a natural scene image using the text recognition method for a natural scene image according to claim 1, comprising:

an acquisition module configured to: acquiring a natural scene image to be identified;

an identification module configured to: performing text recognition on the natural scene image to be recognized by adopting the trained deep learning model to obtain a recognized text;

firstly, correcting a natural scene image to be recognized by a deep learning model, and then extracting a feature vector of the image from the corrected image; and respectively extracting visual features and semantic features from the feature vectors of the image, performing feature fusion on the two features, and finally performing text recognition on the fused features.

Images