KR20230030005A - Training method, apparatus and system of text recognition model framework - Google Patents

Abstract

The present invention provides a training method, apparatus and system for a text recognition model framework, and relates to the field of artificial intelligence technology, specifically to the field of computer vision and deep learning technology, and can be applied to smart city and smart finance scenes. there is. The method may include performing feature processing on a sample image based on a preset text detection model to obtain at least two pieces of feature information related to text information of the sample image; obtaining a fusion feature of the sample image by performing fusion processing on at least two pieces of feature information of the sample image based on a preset feature fusion model; and inputting fusion features into a feature fusion model and adjusting parameters of the text detection model and the feature fusion model based on the fusion feature model, respectively, to obtain a text recognition model framework; Including, the text detection model of the text recognition model framework and the feature fusion model have a relatively high correlation, thereby realizing the completeness and fullness of the training process and improving the accuracy and reliability of the text recognition model framework.

Description

Training method, apparatus and system of text recognition model framework

The present invention claims priority of a Chinese patent application filed with the Chinese Intellectual Property Office on July 28, 2021, with the application number CN202110858410.X, and the application title "Training Method, Apparatus and System for Text Recognition Model Framework". , all contents of which are hereby incorporated into this application by reference.

The present invention relates to the field of artificial intelligence technology, specifically to the field of computer vision and deep learning technology, and particularly to a method, apparatus and system for training a text recognition model framework, which can be applied to smart city and smart finance scenes. there is.

With the development of artificial intelligence technology, the recognition of text information in images develops from artificial recognition to automatic recognition. For example, a text recognition model framework assisting a text recognition model (also referred to as a structured interpretation framework model assisting text recognition model training) is pre-trained, and the text of the image to be recognized is based on the structured framework model. Train and create a text recognition model that recognizes information.

In the prior art, it is usually trained based on a text detection model and a feature fusion model to obtain a text recognition model framework, the text detection model and the feature fusion model are two independent models, and the feature fusion model is specifically text-recognized. Training is completed based on the offline recognition results of the detection model.

However, since the text detection model and the feature fusion model are independent of each other in the training process, the accuracy of the text recognition model framework obtained through training may cause a technical problem with excessively low accuracy.

The present invention provides a text recognition model framework training method and apparatus for improving the accuracy of the text recognition model framework.

According to a first aspect of the present invention, a method for training a text recognition model framework is provided, wherein the method performs feature processing on a sample image based on a preset text detection model to obtain information related to text information of the sample image. obtaining at least two pieces of feature information;

obtaining a fusion feature of the sample image by performing fusion processing on at least two pieces of feature information of the sample image based on a preset feature fusion model; and

obtaining a text recognition model framework by inputting the fusion features into the feature fusion model and adjusting parameters of the text detection model and the feature fusion model, respectively, based on the fusion feature model - the text recognition model frame The work includes a tuned text detection model and a tuned feature fusion model; includes

According to a second aspect of the present invention, there is provided a text recognition method, the method comprising:

obtaining an image to be recognized; and

Acquiring text information of the image to be recognized by inputting the image to be recognized into a pretrained text recognition model - The text recognition model is generated by training the image to be trained based on a pretrained text recognition model framework the text recognition model framework is obtained by training by the training method of the first aspect, and the image to be trained includes text information; includes

According to a third aspect of the present invention, an apparatus for training a text recognition model framework is provided, the apparatus comprising:

a processing unit configured to perform feature processing on a sample image based on a preset text detection model to obtain at least two pieces of feature information related to text information of the sample image;

a fusion unit configured to obtain a fusion feature of the sample image by performing fusion processing on at least two pieces of feature information of the sample image based on a preset feature fusion model; and

A training unit configured to input the fusion features into the feature fusion model and adjust parameters of the text detection model and the feature fusion model based on the fusion feature model, respectively, to obtain a text recognition model framework - the text recognition model - The framework includes a tuned text detection model and a tuned feature fusion model; includes

According to a fourth aspect of the present invention, a text recognition device is provided, the device comprising:

an acquiring unit acquiring an image to be recognized; and

A recognition unit for obtaining text information of the image to be recognized by inputting the image to be recognized into a pretrained text recognition model, wherein the text recognition model trains the image to be trained based on a pretrained text recognition model framework generated, the text recognition model framework is obtained by training by the training method of the first aspect, and the image to be trained includes text information; includes

According to a fifth aspect of the present invention, it includes an electronic device, wherein the electronic device,

at least one processor; and

a memory communicatively coupled to the at least one processor; including,

Instructions executable by the at least one processor are stored in the memory, and when the instructions are executed by the at least one processor, the at least one processor performs the method of the first aspect; Alternatively, the at least one processor performs the method of the second aspect.

According to a sixth aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions cause a computer to perform the method of the first aspect; Alternatively, the computer instructions cause a computer to perform the method of the second aspect.

According to a seventh aspect of the present invention, there is provided a computer program product including a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, The at least one processor executes the computer program so that the electronic device performs the method of the first aspect; Alternatively, the at least one processor executes the computer program so that the electronic device performs the method of the second aspect.

According to an eighth aspect of the present invention, a training system for a text recognition model framework is provided, the system comprising:

a text detection model for obtaining at least two pieces of feature information related to text information of the sample image by performing feature processing on the sample image; and

a feature fusion model for obtaining a fusion feature of the sample image by performing fusion processing on at least two pieces of feature information of the sample image; includes

The feature fusion model is further used to obtain a text recognition model framework by adjusting parameters of the text detection model and the feature fusion model respectively, wherein the text recognition model framework is adjusted text detection model and adjusted feature fusion model. include the model

According to a ninth aspect, embodiments of the present application provide a computer program, including program code, when a computer runs the computer program, the program code performs the method of the first aspect or the second aspect. do.

It should be understood that the content set forth in the Summary of the Invention section is not intended to identify key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention are readily understood through the following specification.

The drawings are used to better understand the present technical means and are not intended to limit the present invention.
1 is a schematic diagram according to a first embodiment of the present invention.
Fig. 2 is a schematic diagram according to a second embodiment of the present invention.
Fig. 3 is a scene schematic diagram of a method for training a text recognition model framework according to an embodiment of the present invention;
Fig. 4 is a schematic diagram according to a third embodiment of the present invention.
Fig. 5 is a schematic diagram according to a fourth embodiment of the present invention.
Fig. 6 is a schematic diagram according to a fifth embodiment of the present invention.
Fig. 7 is a schematic diagram according to a sixth embodiment of the present invention.
Fig. 8 is a block diagram of an electronic device implementing the training method and text recognition method of the text recognition model framework according to an embodiment of the present invention.
Fig. 9 is a schematic diagram according to a seventh embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in conjunction with the drawings. It contains various details of embodiments of the present invention for purposes of understanding, and the embodiments are to be regarded as illustrative only. Therefore, it should be recognized that those skilled in the art belonging to the present invention can make various changes and modifications to the embodiments without departing from the scope and spirit of the present invention. At the same time, for accuracy and conciseness, descriptions of well-known functions and structures are omitted in the following description.

Text recognition technology refers to recognition of text information in an image, and text recognition technology is widely applied to various fields such as education, finance, medical, transportation, and insurance.

For example, when text recognition technology is applied to the medical field, text information of a medical chart image is recognized based on the text recognition technology. Also, for example, when text recognition technology is applied to the insurance field, text information of an insurance certificate image is recognized based on the text recognition technology, etc., which are not enumerated here.

Following the reversal of deep learning technology in artificial intelligence technology, deep learning technology can be combined with other technologies. For example, by applying deep learning technology to text recognition technology, accuracy and reliability of text information recognition can be improved.

For example, a text recognition model that recognizes text information based on deep learning technology can be trained. However, training of a text recognition model should typically be based on a text recognition model framework. That is, usually, a text recognition model framework is first trained to obtain a text recognition model framework, and then a text recognition model is trained to obtain a text recognition model framework on the basis of the text recognition model framework.

In related art, it is usually trained by two independent models to obtain a text recognition model framework, and the two independent models are a text detection model and a feature fusion model, respectively, and to train the text recognition model framework, The feature fusion model is an offline recognition result based on a text detection model.

Specifically, the text detection model may be an optical character recognition (OCR) model, the feature fusion model may be a transfromer model, and the transfromer model is specifically trained based on offline recognition results of the optical character recognition model. complete, and acquire the text recognition model framework.

However, since the optical character recognition and the transformer model are independent of each other in the training process, the accuracy of the text recognition model framework obtained through training may result in technical challenges that are too low.

In order to avoid the above technical problem, the inventor of the present invention obtains the inventive concept of the present invention through creative labor. Based on the text detection model and the feature fusion model, fusion features are obtained, and the feature fusion model is trained on the text recognition model and the feature recognition model as a whole based on the fusion features to obtain a text recognition model framework.

Based on the above invention concept, the present invention provides a text recognition model framework training method, apparatus and system, applied to the field of computer vision and deep learning technology in the field of artificial intelligence technology, and applied to smart city and smart finance scenes. and improve the accuracy of the text recognition model framework.

Referring to Fig. 1, Fig. 1 is a schematic diagram according to a first embodiment of the present invention.

As shown in Fig. 1, the training method of the text recognition model framework provided by the embodiment of the present invention includes steps S101 to S103.

In operation S101, at least two pieces of feature information related to text information of the sample image are acquired by performing feature processing on the sample image based on a preset text detection model.

Exemplarily, the subject of this embodiment may be a training device (hereinafter referred to as a training device) of a text recognition model framework, and the training device may be a server (eg, a local server or a cloud server), It may be a terminal device, it may be a processor, it may be a chip, etc., and is not limited in this embodiment.

Sample images include text information. For example, for the medical field, the sample image may be an image of a medical treatment chart, and the sample image includes text information such as a patient's identity and text information of a case. Further, for example, in the field of insurance, the sample image may be an image of an insurance policy, and the sample image includes text information such as an insurer's identity and text information of insurance content.

It should be understood that the number of sample images for training the text recognition model framework may be set by the training device based on demand, historical records and tests, etc., and is not limited in this embodiment.

The text detection model may be a model that detects features related to text information in a sample image. For example, for the medical field, the text detection model can detect the text information of the patient's identity in the medical chart image.

Specifically, the text detection model may be optical character recognition.

In this embodiment, feature information is used to indicate features related to text information of a sample image, and at least two types of feature information include information related to text content, information related to text vision, and information of each character in a spatial relationship. You can, and I won't enumerate them here.

In operation S102, fusion processing is performed on at least two pieces of feature information of the sample image based on a preset feature fusion model to obtain a fusion feature of the sample image.

A feature fusion model refers to a model that performs fusion processing on a plurality of feature information. For example, a feature fusion model can be a transformer model.

The fusion processing may splice on a plurality of feature information, combine the plurality of feature information, and link the plurality of feature information, and the present embodiment is not limited, and the detailed processing process of the fusion processing may refer to related art. may be, and will not be further explained here.

In step S103, fusion features are input into the feature fusion model, and parameters of the text detection model and the feature fusion model are respectively adjusted based on the fusion feature model to obtain a text recognition model framework.

The text recognition model framework includes a tuned text detection model and a tuned feature fusion model.

In this embodiment, a text recognition model framework may be obtained by inputting fusion features to a feature fusion model, adjusting parameters of a text detection model based on the fusion features, and adjusting parameters of the feature fusion model.

It should be understood that training of a text recognition model framework is an iterative process. That is, it is the process of iteratively adjusting the parameters of the text detection model and the parameters of the feature fusion model, and when the number of iterations reaches a preset number, or the loss function for iterations is smaller than the preset loss threshold, training has already met the demand. By explaining what has been reached, we obtain the text recognition model framework.

As can be seen based on the above analysis, an embodiment of the present invention provides a method for training a text recognition model framework, wherein the method performs feature processing on sample images based on a preset text detection model, obtaining at least two pieces of feature information related to text information of an image; obtaining a fusion feature of the sample image by performing fusion processing on at least two pieces of feature information of the sample image based on a preset feature fusion model; inputting fusion features into a feature fusion model, and adjusting parameters of a text detection model and a feature fusion model based on the fusion feature model, respectively, to obtain a text recognition model framework - the text recognition model framework adjusts text detection -Including model and adjusted feature fusion model; In this embodiment, a technical feature is introduced to obtain a text recognition model framework by adjusting parameters of a text detection model and a feature fusion model based on fusion features, respectively, to detect text in the text recognition model framework. By having a relatively high correlation between the model and the feature fusion model, the completeness and comprehensiveness of the training process is implemented, and the text detection model and the feature fusion model are independent of each other in the related art, so that the text recognition model framework is trained. By avoiding the case of insufficient consideration of the entire dimension, the disadvantage of too low accuracy of the text recognition model framework is prevented, and the accuracy and reliability of the text recognition model framework are improved.

Referring to Fig. 2, Fig. 2 is a schematic diagram according to a second embodiment of the present invention.

For example, as shown in Fig. 2, the training method of the text recognition model framework provided by the embodiment of the present invention includes steps S201 to S204.

In step S201, location information of text lines in the sample image is determined based on the text detection model, and at least two pieces of feature information are determined according to the location information.

Regarding the same features of this embodiment and the first embodiment, they are not described in this embodiment.

As can be seen based on the above analysis, the training method of the text recognition model framework of this embodiment can be applied to different fields such as insurance and medical fields. An exemplary description of the present embodiment is given by taking as an example that the training method of the text recognition model framework of this embodiment is applied to the field of insurance as an example.

For example, as shown in FIG. 3, the sample image is an insurance certificate image, and the insurance certificate image is shown in FIG. Contains textual information.

In some embodiments, the sample image is transmitted to the training device through a scanning method, and location information of the text line of the sample image is determined by a text detection model of the training device.

In another embodiment, as shown in FIG. 3, the training device may be connected by an external device (eg, a storage device), and the sample image transmitted from the external device is received, and the text detection model of the training device is Determine the location information of the text line of the sample image by

A text line indicates a line where text information is located. The position information of the text row indicates information related to the position of the row where the text information is located, and may specifically be the coordinates of the row where the text information is located and a sample image.

For example, when the text detection model recognizes a sample image, box selection is performed for a text row of the sample image based on a preset rectangular box, and the coordinates of the rectangular box are determined in the sample image.

In this embodiment, positional information of a text row of a sample image is determined, at least two pieces of feature information are determined in the sample image based on the positional information, and the at least two pieces of feature information are obtained through a relatively high accuracy positioning method. By determining, it is possible to implement a technical effect of improving the accuracy and reliability of at least two pieces of feature information.

In some embodiments, the step of determining at least two pieces of feature information according to the location information may include obtaining a text area by performing a cropping operation on the sample image according to the location information, and determining at least two pieces of feature information in the text area. It includes obtaining information.

For example, in combination with the above embodiment, after determining the location information, a region selected by a rectangular box in the sample image may be cut out based on the location information, the region is a text area, and the text information of the text area may be cut out. Through the recognition method, at least two pieces of feature information are obtained.

In this embodiment, by cutting out the text area from the sample image based on the location information, almost all of the text information is included in the text area, omission of text information is prevented, and the cropping operation has relatively high accuracy. By doing so, the text area has a relatively high accuracy and reliability, and at least two pieces of characteristic information determined based on the text area have a relatively high comprehensiveness and reliability.

In some embodiments, acquiring at least two pieces of feature information from the text area includes extracting image features of the sample image from the text area, recognizing the image features, and acquiring the at least two pieces of feature information. do.

Image features can be understood in two large dimensions, and the two large dimensions are the content dimension and the appearance dimension, respectively. For example, in this embodiment, when the sample image is an image including text information, the image feature of the content level indicates a feature included in the image feature and related to the content of the text information, such as text content; Image features of the appearance dimension refer to features related to colors and patterns of text information included in image features.

Accordingly, in the present embodiment, at least two pieces of feature information may include two pieces of feature information determined respectively based on two large dimensions (ie, a content dimension and an appearance dimension). Of course, it can be seen by combining the analysis of the first embodiment above, that the two large dimensions are divided into smaller dimensions, and three or more types of feature information can be determined based on the smaller dimensions, which are limited in this embodiment. I never do that.

In this embodiment, since the text area has relatively high accuracy and fullness, the image feature extracted from the text area has relatively high accuracy and fullness, and when the image feature is recognized and feature information is obtained, multiple By analyzing in the dimension of , it is possible to obtain feature information of a plurality of dimensions, and implement a technical effect of improving the accuracy, comprehensiveness and reliability of feature information.

In some embodiments, the at least two pieces of feature information include text features and vision features.

Text features can be understood as feature information based on the content dimension, and vision features can be understood as feature information based on the appearance dimension.

In step S202, fusion processing is performed on text features and vision features based on a preset feature fusion model to obtain fusion features of the sample image.

The implementation principle for S202 may refer to the first embodiment, and is not further described herein.

In S203, a plurality of text feature blocks representing text features are constructed, and a plurality of vision feature blocks representing vision features are constructed.

For example, a plurality of text feature blocks having text features and mapping relationships may be constructed, and the plurality of text feature blocks may be used to represent text features.

Exemplarily, the number of text feature blocks may be determined, the text feature may be mapped to a plurality of text feature blocks, and the plurality of text feature blocks may represent text features, based on a method such as demand, history record, and test.

Specifically, the text feature block may be a 2*2 (pixel) feature block, and based on the semantic information of the text feature, the text feature is divided and stored as a plurality of 2*2 (pixel) feature blocks, so that a plurality of text feature blocks are stored. A text feature block of can be obtained.

Semantic information may be understood as information of text information related to field classification, information of text information related to positions between fields, or information of text information related to meaning.

Similarly, the principle and implementation of building a plurality of vision feature blocks representing vision features can be referred to the step of building a plurality of text feature blocks representing text features, and is not further described herein.

Step 204, the feature fusion model is adjusted according to the fusion feature and the plurality of text feature blocks, parameters of the text detection model and the feature fusion model, respectively; and adjusting parameters of the text detection model and the feature fusion model, respectively, according to the feature fusion model according to the fusion feature and the plurality of vision feature blocks; perform at least one of

In one example, a fusion feature may be combined with a plurality of text feature blocks to adjust the parameters of the text detection model and to adjust the parameters of the feature fusion model.

In some embodiments, adjusting the parameters of the text detection model by combining the fusion features with the plurality of text feature blocks, and adjusting the parameters of the feature fusion model includes a first step to a second step.

In the first step, the feature fusion model randomly covers some text features of the fusion features, and performs predictive complementation processing on some text features covered according to a plurality of text feature blocks to obtain some text features after prediction and complementation. do.

Based on the above analysis, it can be seen that the process of training the text recognition model framework is an iterative process, so in the process under training, some text features of the randomly covered fusion features in the current iteration and randomly covered fusions in the previous iteration Some text features of the feature are floating.

Illustratively, with each iteration, some text features of the randomly covered fusion features are completely different.

For example, in the first iteration, some text features of the random covered fusion features are text features 6% before the text features, and in the second iteration, some text features of the random covered fusion features are text features. It is a text feature between 6% and 12% in the future, inferring as it is, and not enumerating them one by one.

Also, for example, for the first iteration, some text features of the randomly covered fusion features are text features in 6% of the text features, and for the second iteration, some text features of the random covered fusion features are text features for 6% of the text features. When first repeated in a feature, 6% of text features other than covered fusion features.

Exemplarily, some text features of the randomly covered fusion feature with each iteration are not exactly the same.

For example, for the first repetition, some text features of the random covered fusion features are text features of 6% of the text features, and for the second repetition, some text features of the random covered fusion features are text features. 6% of the text features, and the same text features are present in 6% of text features that are repeated and covered for the first time and 6% of text features that are covered in the second iteration.

Based on the above analysis, since a plurality of text feature blocks can be used to represent text features, after covering some text features, supplementary prediction is performed on some files based on the plurality of text feature blocks, so that some of the predictions after complementation are performed. Text features can be obtained.

For example, if the text feature of the fusion feature is A, some covered text feature of text feature A is a1, and the feature of some other uncovered files is a2, then the training apparatus may set a plurality of text block features and some file features Based on feature a2, the content of some text features (that is, some text features after prediction and complementation) can be estimated.

In the second step, parameters of the text detection model and the feature fusion model are respectively adjusted according to some text features after prediction and complementation and features other than some text features covered by the fusion features.

Combined with the above embodiment, this step, according to the content of some text features estimated by the training device (namely, some text features after prediction and complementation), some file features are a2, adjust the parameters of the text detection model; It can be understood as adjusting the parameters of the fusion model.

In this embodiment, by covering some text features of fusion features, predictive complementation is performed on some of the text features covered based on a plurality of text feature blocks, and two models are obtained based on some text features obtained after predictive complementation. (i.e., text detection model and feature fusion model) parameters are adjusted respectively, and the correlation between text features and each part of text features (i.e., including text content and positional correlation) is fully considered. And, by improving the recognition discrimination ability of the two models, the technical effect of improving the accuracy and reliability of the text recognition model framework obtained by training is implemented.

In another embodiment, adjusting parameters of a text detection model by combining fusion features with a plurality of text feature blocks, and adjusting parameters of a feature fusion model includes first to second steps.

In the first step, the feature fusion model performs replacement processing on text features of the fusion feature according to at least some text feature blocks of the plurality of text feature blocks, so as to obtain replaced text features.

In the second step, according to the vision feature and the replaced text feature of the fusion feature, the parameters of the text detection model and the feature fusion model are respectively adjusted.

In this embodiment, the replacement principle for the text feature of the fusion feature may be full replacement or partial replacement, and is not limited in this embodiment.

As for the replacement principle of text features of fusion features, refer to the principle of cover processing for some text features of text features of fusion features in the above embodiments, which is not further described herein.

Similarly, in this embodiment, the text features of the fusion feature are replaced, and the parameters of the two models (namely, the text detection model and the feature fusion model) are respectively adjusted based on the replaced text features and the vision features of the fusion feature. Thus, by improving the ability to recognize and discriminate between the two models, it is possible to implement a technical effect of improving the accuracy and reliability of the text recognition model framework obtained through training.

In another example, a fusion feature and a plurality of vision feature blocks may be combined to adjust parameters of a text detection model and parameters of a feature fusion model.

In some embodiments, combining the fusion feature and the plurality of vision feature blocks, adjusting parameters of the text detection model, and adjusting parameters of the feature fusion model includes first steps to second steps.

In the first step, the feature fusion model covers some vision features of the fusion feature at random, performs prediction and complementation processing on some vision features covered according to a plurality of vision feature blocks, and obtains some vision features after prediction and complementation. .

In the second step, parameters of the text detection model and the feature fusion model are respectively adjusted according to some vision features after predictive complementation and features other than some vision features covered by the fusion features.

The implementation principle for this embodiment may refer to the implementation principle of combining a fusion feature and a plurality of text feature blocks in the above embodiment, which is not described herein further.

Similarly, in this embodiment, some vision features of fusion features are covered, and some vision features covered are predicted and supplemented based on a plurality of vision feature blocks, and two models are obtained based on some vision features after prediction and supplementation obtained. (i.e., text detection model and feature fusion model) can improve the recognition discrimination ability of the two models by adjusting the parameters respectively, and realize the technical effect of improving the accuracy and reliability of the text recognition model framework obtained by training do.

In another embodiment, combining the fusion feature and the plurality of vision feature blocks, adjusting parameters of the text detection model, and adjusting parameters of the feature fusion model includes first to second steps.

In a first step, the feature fusion model performs replacement processing on the vision features of the fusion features according to at least some vision feature blocks of the plurality of vision feature blocks, so as to obtain the replaced vision features.

In the second step, according to the text feature and the replaced vision feature of the fusion feature, parameters of the text detection model and the feature fusion model are respectively adjusted.

Regarding the implementation principle of this embodiment, reference may be made to the implementation principle of combining a fusion feature and a plurality of text feature blocks in the above embodiment, which is not further described herein.

Similarly, in this embodiment, the vision features of the fusion features are replaced, and based on the replaced vision features and the text features of the fusion features, the parameters of the two models (namely the text detection model and the feature fusion model) are respectively adjusted, Since the recognition discrimination ability of the two models can be improved, the technical effect of improving the accuracy and reliability of the text recognition model framework obtained by training is implemented.

In another example, in combination with the fusion feature, the plurality of text feature blocks, and the plurality of vision feature blocks, adjust parameters of the text detection model and adjust parameters of the feature fusion model.

For example, this example includes steps 1 to 3.

In a first step, the feature fusion model determines a first adjusting task result of adjusting the text detection model and the feature fusion model according to the fusion feature and the plurality of text feature blocks.

In a second step, the feature fusion model determines a second adjustment task result of adjusting the text detection model and the feature fusion model according to the fusion feature and the plurality of vision feature blocks.

In a third step, parameters of the text detection model and the feature fusion model are respectively adjusted according to the weighted average information of the result of the first adjustment task and the result of the second adjustment task.

In combination with the above embodiments, in some embodiments, taking covering some text features of fusion features as a first training task, obtaining a first training result; taking covering some vision features of the fusion feature as a second training task, obtaining a second training result; take covering text features of the fusion feature as a third training task, conduct training, and obtain a third training result; By performing weighted average processing on the first training result, the second training result, and the third training result, finally obtaining parameters for adjusting the text detection model and feature fusion model parameters, and finally adjusting the text detection model. The parameters of the text detection model are adjusted based on the parameters, and the parameters of the feature fusion model are adjusted based on the acquired parameters that adjust the last feature fusion model.

In another embodiment, covering some text features of fusion features is taken as a first training task, and a first training result is obtained; taking covering some vision features of the fusion feature as a second training task, obtaining a second training result; replace the vision feature of the fusion feature as a third training task, conduct training, and obtain a third training result; By performing weighted average processing on the first training result, the second training result, and the third training result, finally obtaining parameters for adjusting the text detection model and feature fusion model parameters, and finally adjusting the text detection model. The parameters of the text detection model are adjusted based on the parameters, and the parameters of the feature fusion model are adjusted based on the acquired parameters that adjust the last feature fusion model.

Regarding the method of combining training tasks in this embodiment, combining the three types of training tasks (it should be understood that the method of combining the three types of training tasks is for illustrative purposes only, and is regarded as a limitation on the combination method). That is, it may be a combination method other than the above combination method, which is not enumerated here), or may be a combination of two training tasks, and the implementation principle is three types of training tasks. Can refer to the implementation principle of combining, which is not further described herein.

In this embodiment, a text recognition model framework is obtained by training by a multi-task training method, training of fusion features and a plurality of text feature blocks is one of the training tasks, and training of fusion features and a plurality of vision feature blocks is performed. The parameters of the text detection model and the parameters of the feature fusion model are determined by setting the training as another training task, and finally determining parameters for adjusting the text detection model and parameters for adjusting the feature fusion model based on the training results of the two training tasks. It is possible to implement the adjustment for , and adjust the parameters of the text detection model and the parameter of the feature fusion model through a multi-task training method, thereby implementing a technical effect of improving the accuracy and reliability of the adjustment.

It should be noted that, in this embodiment, the parameters of the text detection model and the feature fusion model are respectively adjusted according to different methods to improve the flexibility and variety of parameter adjustment, thereby increasing the flexibility and flexibility of training the text recognition model framework. A variety of technical effects can be realized.

Fig. 4 is a schematic diagram according to a third embodiment of the present invention. For example, as shown in Fig. 4, the text recognition method in this embodiment includes steps S401 to S402.

In S401, an image to be recognized is acquired.

In S402, an image to be recognized is input to a pre-trained text recognition model to obtain text information of the image to be recognized.

The text recognition model is generated by training an image to be trained based on a pre-trained text recognition model framework, the text recognition model framework is obtained by training by the training method of the first aspect, and the image to be trained is text contains information

As can be seen based on the above analysis, since the text recognition model framework includes a text detection model and a feature fusion model, and has relatively high accuracy and reliability, the text recognition model can be trained based on the text recognition model framework. When obtained, a text recognition model has relatively high accuracy and reliability, and when recognizing an image to be recognized based on the text recognition model, a technical effect of improving the validity and reliability of recognition can be implemented.

Fig. 5 is a schematic diagram according to a fourth embodiment of the present invention. For example, as shown in Fig. 5, the training device 500 of the text recognition model framework of this embodiment,

a processing unit 501 for performing feature processing on a sample image based on a preset text detection model to obtain at least two pieces of feature information related to text information of the sample image;

a fusion unit 502 for obtaining a fusion feature of the sample image by performing fusion processing on at least two pieces of feature information of the sample image based on a preset feature fusion model; and

A training unit 503 for inputting fusion features into a feature fusion model, and adjusting parameters of the text detection model and feature fusion model respectively based on the fusion feature model, to obtain a text recognition model framework - the text recognition model framework comprises: - including a tuned text detection model and a tuned feature fusion model; includes

Fig. 6 is a schematic diagram according to a fifth embodiment of the present invention. For example, as shown in Fig. 6, the training device 600 of the text recognition model framework of this embodiment,

and a processing unit 601 for obtaining at least two pieces of feature information related to text information of the sample image by performing feature processing on the sample image based on a preset text detection model.

As can be seen in conjunction with FIG. 6, in some embodiments, processing unit 601:

a first determining subunit 6011 for determining positional information of text lines in the sample image based on the text detection model;

a second determining subunit 6012 for determining at least two pieces of feature information according to the location information; includes

In some embodiments, the second decision sub-unit 6012:

a cropping module that obtains a text area by performing a cropping operation on the sample image according to the location information;

an acquisition module acquiring at least two pieces of feature information in the text area; includes

In some embodiments, the acquisition module is used to extract image features of the sample image from the text area, recognize the image features, and acquire at least two pieces of feature information.

The fusion unit 602 is used to obtain a fusion feature of the sample image by performing fusion processing on at least two pieces of feature information of the sample image based on a preset feature fusion model.

The at least two types of feature information include text features and vision features.

The building unit 603 is used to build a plurality of text feature blocks representing text features, and to build a plurality of vision feature blocks representing vision features.

The training unit 604 is used to input the fusion features into the feature fusion model, adjust the parameters of the text detection model and the feature fusion model respectively based on the fusion feature model, to obtain a text recognition model framework, and the text recognition model The framework includes a tuned text detection model and a tuned feature fusion model.

As can be seen in conjunction with FIG. 6, in some embodiments, training unit 604:

a first cover subunit 60411 for randomly covering some text features of the fusion features;

a second prediction and complementation subunit 60412, configured to perform predictive complementation processing on some of the text features covered by a plurality of text feature blocks, so as to obtain some text features after prediction and complementation; and

a first adjustment subunit 60413 for adjusting parameters of the text detection model and the feature fusion model, respectively, according to some text features after predictive complementation and features other than some text features covered by the fusion features; includes

In another embodiment, the training unit 604:

a second cover subunit 60414 for randomly covering some vision features of the fusion features;

a second prediction supplementation subunit 60415, which performs prediction supplementation processing on some vision features covered by a plurality of vision feature blocks, and obtains some vision features after prediction and supplementation; and.

a second adjustment subunit 60416 for adjusting parameters of the text detection model and the feature fusion model, respectively, according to some vision features after predictive complementation and features other than some vision features covered by the fusion features; includes

As can be seen in conjunction with FIG. 6, in some embodiments, training unit 604:

a first replacement subunit 60417, configured to perform replacement processing on text features of the fusion feature according to at least some text feature blocks of the plurality of text feature blocks, so as to obtain replaced text features;

a third adjustment subunit 60418 for adjusting parameters of the text detection model and the feature fusion model, respectively, according to the vision feature and the replaced text feature of the fusion feature; more includes

In another embodiment, the training unit 604:

a second replacement subunit 60419 configured to perform replacement processing on the vision features of the fusion feature according to at least some vision feature blocks of the plurality of vision feature blocks, so as to obtain the replaced vision features;

a fourth adjustment subunit 60420 for adjusting parameters of the text detection model and the feature fusion model, respectively, according to the text feature and the replaced vision feature of the fusion feature; includes

As can be seen in conjunction with Figure 6, in some embodiments, when the feature fusion model adjusts the parameters of the text detection model and the feature fusion model respectively according to the fusion feature, the plurality of text feature blocks, and the plurality of vision feature blocks, The training unit 604,

a third determining subunit 60421, configured to determine a result of the first adjusting task of adjusting the text detection model and the feature fusion model according to the fusion feature and the plurality of text feature blocks;

a fourth determining subunit 60422, configured to determine a second adjusting task result of adjusting the text detection model and the feature fusion model according to the fusion feature and the plurality of vision feature blocks; and

a fifth adjustment subunit 60423 for adjusting parameters of the text detection model and the feature fusion model, respectively, according to the weighted average information of the first adjustment task result and the second adjustment task result; includes

Fig. 7 is a schematic diagram according to a sixth embodiment of the present invention. For example, as shown in FIG. 7, the text recognition device 700 of this embodiment,

an acquiring unit 701 for acquiring an image to be trained, the image to be trained including text information;

A recognition unit 702 for obtaining text information of the image to be recognized by inputting an image to be recognized into a pretrained text recognition model, wherein the text recognition model trains the image to be trained based on the pretrained text recognition model framework, generated, the text recognition model framework is obtained by training by the training method of the first aspect, and the image to be trained includes text information; includes

According to an embodiment of the present invention, the present invention provides an electronic device and a readable storage medium.

According to an embodiment of the present invention, the present invention provides a computer program product including a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, At least one processor executes a computer program so that the electronic device performs the means provided in any of the above embodiments.

As shown in Fig. 8, Fig. 8 is a schematic block diagram of an electronic device 800 implementing an embodiment of the present invention. Electronic devices display digital computers in various formats. For example, it may be a laptop computer, desktop computer, workstation, personal digital assistant (PAD), server, blade server, mainframe and other suitable computer. Electronic devices display various types of mobile devices. For example, it may be a personal information assistant (PAD), cellular phone, smart phone, wearable device and other similar computing devices. The components, their connections and relationships, and functions disclosed herein are illustrative only and are not intended to limit the implementation of the invention as described and/or claimed herein.

As shown in FIG. 8, the electronic device 800 includes a computing unit 801, a computer program stored in a read-only memory (ROM) 802 or a random access memory (RAM) in the storage unit 808 ( 803), each suitable operation and process is performed according to the loaded computer program. In the RAM 803, each program and data required for operation of the electronic device 800 may be stored. Computing unit 801 , ROM 802 and RAM 803 are connected to each other via a bus 804 . An input/output I/O interface 805 is also connected to bus 804.

A plurality of components of the device 800 are connected to the I/O interface 805, and the plurality of components include an input unit 806 such as a keyboard, a mouse, and the like; output units 807 such as various types of monitors, speakers, etc.; a storage unit 808 such as a magnetic disk, optical disk, or the like; and a communication unit 809 such as a network card, modem or wireless communication transceiver. The communication unit 809 allows the electronic device 800 to exchange information/data with other devices via a computer network such as the Internet and/or a respective telegraph network.

The computing unit 801 may be a general purpose/dedicated processing component with respective processing and calculation functions. Some examples of the computing unit 801 include a central processing unit (CPU), a graph processing unit (GPU), each dedicated artificial intelligence calculation chip, a computing unit that runs each machine learning model algorithm, a digital signal processing unit (DSP), any of suitable processors, controllers and microcontrollers, and the like. The computing unit 801 performs each method and process described above. For example, the text recognition model framework's training method and text recognition method are performed. For example, in one embodiment, the training method of the text recognition model framework and the text recognition method may be implemented as a computer software program and tangibly included in a machine-readable medium such as the storage unit 805 . In one embodiment, some or all of the computer programs may be loaded and/or installed into the electronic device 800 by the ROM 802 and/or the communication unit 809 . When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the text recognition model framework training method and text recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may perform the training method of the text recognition model framework, the text recognition method according to an embodiment of the present invention in any other suitable manner (eg, firmware). It is structured so that

Various embodiments of the systems and technologies described herein include digital electronic circuit systems, integrated circuit systems, programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), dedicated standard products (ASSPs), systems on a chip (SOCs), and composites. It may be realized in programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs, wherein the one or more computer programs are executed and/or interpreted in a programmable system including at least one programmable processor, The programmable processor may be a dedicated or general programmable processor, capable of receiving data and instructions from a storage system, at least one input device and at least one output device, and transmitting data and instructions from the storage system to the at least one output device. to an input device and to the at least one output device.

The program code for performing the method of the present invention may be written in one or any combination of programming languages. The program code is provided to a processor or controller of a general-purpose computer, dedicated computer, or other programmable data processing device to execute the functions/operations specified in the flowchart and/or block diagram when the program code is executed by the processor or controller. do. The program code runs wholly or partly on the machine, as an independent software package, which runs partly on the machine and partly or completely on a remote machine or server.

In the context of the present invention, a machine-readable medium may be a tangible medium and may contain or store a program for use by or provided for use by a command execution system, device or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices and devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more lines, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, CD-ROM, optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented in a computer, which may include a display device (e.g., a CRT cathode ray tube) or LCD (liquid crystal display) monitor that displays information to the user. ; and a keyboard and a pointing device (eg, a mouse or trackball), through which the user can provide input to the computer. Other types of devices may also be used for interaction with the user, for example, the feedback provided to the user may be any form of sensory feedback (eg, visual feedback, auditory feedback or tactile feedback), and any An input from the user may be received in the form of (sound input, voice input, or tactile input).

The systems and techniques described herein can be combined with a computing system that includes background components (eg, a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a data server). for example, on a user's computer having a graphical user interface or network browser, through which a user may interact with embodiments of the systems and techniques described herein), or such a background components, middleware components or any combination of front end components. Any form or medium of digital data communication (eg, a communication network) may connect the components of the system to each other. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and networks.

A computer system may include a client and a server. Clients and servers are usually remote from each other and interact with each other through a communication network. A relationship of client and server is created by computer programs running on corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also referred to as a cloud computing server, is a host product of a cloud computing service system, in a traditional physical host and VPS service ("Virtual Private Server", or "VPS"), difficult to manage, business scalability solve this weak flaw. The server may be a server of a distributed system or a server combining block chains.

According to another aspect of the embodiments of the present invention, the embodiments of the present invention provide a training system for a text recognition model framework, the system comprising:

a text detection model for acquiring at least two pieces of feature information related to text information of the sample image by performing feature processing on the sample image based on a preset text detection model;

a feature fusion model that obtains a fusion feature of the sample image by performing fusion processing on at least two pieces of feature information of the sample image based on a preset feature fusion model; includes

The feature fusion model is further used to obtain a text recognition model framework by adjusting parameters of the text detection model and the feature fusion model, respectively, and the text recognition model framework includes the adjusted text detection model and the adjusted feature fusion model. do.

As can be seen in conjunction with the above analysis, in some embodiments the text detection model may be optical character recognition and the feature fusion model may be a transfromer model.

As can be seen in conjunction with Figure 9, in some embodiments, the training system 900 of the text recognition model framework of an embodiment of the present invention:

optical character recognition (901) for detecting text information of the sample image, obtaining location information of text lines in the sample image, and transmitting the location information to the region feature extractor (902);

a region feature extractor 902 that obtains a text region by performing a cropping operation on the sample image according to the location information, and transmits the text region to the character recognizer 903 and the vision recognizer 904, respectively;

a character recognizer 903 that determines text features of the text area and transmits the text features to the transformer model 905; and

a vision recognizer 904 that determines the vision features of the text area and transmits the vision features to the transformer model 905; includes

The transfromer model 905 performs fusion processing on text features and vision features, acquires fusion features, and adjusts parameters of the optical character recognition 901 and the parameters of the transfromer model 905 based on the fusion features, so that the text Acquire a recognition model framework.

The text recognition model framework includes a tuned text detection model and a tuned feature fusion model.

It should be understood that each component in the above embodiment may be integrated into one or formed independently, and is not limited in this embodiment.

For example, optical character recognition, region feature extractor, character recognizer and vision recognizer are components independent of each other; Further, for example, the area feature extractor is a component of integrated optical character recognition, and is independent of the character recognizer and vision recognizer, respectively, and the character recognizer and vision recognizer are two independent components, which are not individually enumerated here.

The implementation principle for each of the above features may refer to the description of the above method embodiment, and is not described herein further.

According to another aspect of the embodiments of the present application, the embodiments of the present application provide a computer program, including program code, and when a computer runs the computer program, the program code performs any of the above methods.

It should be understood that steps may be rearranged, added or deleted by the above plural forms of flow. For example, each step described in the present invention may be executed in parallel or sequentially, or may be executed in a different order. As long as the technical means disclosed in the present application can implement only the desired results, the present application is not limited thereto.

The above specific implementation manner does not limit the protection scope of the present invention. Those skilled in the art to which the present invention pertains should understand that various modifications, combinations, subcombinations, and substitutions may be made according to design requirements and other factors. All modifications, equivalent replacements and improvements made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims (27)

In the training method of the text recognition model framework,
obtaining at least two pieces of feature information related to text information of the sample image by performing feature processing on the sample image based on a preset text detection model;
obtaining a fusion feature of the sample image by performing fusion processing on at least two pieces of feature information of the sample image based on a preset feature fusion model; and
obtaining a text recognition model framework by inputting the fusion features into the feature fusion model and adjusting parameters of the text detection model and the feature fusion model, respectively, based on the fusion feature model - the text recognition model frame The work includes a tuned text detection model and a tuned feature fusion model; including,
Training method of a text recognition model framework, characterized in that. According to claim 1,
The step of obtaining at least two pieces of feature information related to the text of the sample image by performing feature processing on the sample image based on a preset text detection model,
Determining location information of a text row in the sample image based on the text detection model, and determining the at least two pieces of feature information according to the location information.
Training method of a text recognition model framework, characterized in that. According to claim 2,
The step of determining the at least two pieces of characteristic information according to the location information,
Performing a cropping operation on the sample image according to the location information to obtain a text area, and acquiring the at least two pieces of feature information in the text area.
Training method of a text recognition model framework, characterized in that. According to claim 3,
Obtaining the at least two pieces of feature information in the text area includes:
Extracting image features of the sample image from the text area, recognizing the image features, and obtaining the at least two pieces of feature information.
Training method of a text recognition model framework, characterized in that. According to any one of claims 2 to 4,
The at least two pieces of characteristic information include text characteristics and vision characteristics; After determining the at least two pieces of feature information according to the location information,
Constructing a plurality of text feature blocks representing the text feature, and constructing a plurality of vision feature blocks representing the vision feature;
The step of inputting the fusion feature into the feature fusion model and adjusting the parameters of the text detection model and the feature fusion model, respectively, based on the fusion feature model, wherein the feature fusion model determines the fusion feature and the plurality of texts. adjusting parameters of the text detection model and the feature fusion model, respectively, according to the feature block; and adjusting parameters of the text detection model and the feature fusion model, respectively, according to the feature fusion model according to the fusion feature and the plurality of vision feature blocks. Including at least one step of
Training method of a text recognition model framework, characterized in that. According to claim 5,
Adjusting, by the feature fusion model, parameters of the text detection model and the feature fusion model, respectively, according to the fusion feature and the plurality of text feature blocks,
The feature fusion model randomly covers some text features of the fusion features, and performs predictive complementation processing on the covered some text features according to the plurality of text feature blocks to obtain some text features after prediction and complementation. step; and
adjusting parameters of the text detection model and the feature fusion model, respectively, according to some text features after prediction and complementation and features other than the some text features covered by the fusion feature; including,
Training method of a text recognition model framework, characterized in that. According to claim 5,
The step of adjusting parameters of the text detection model and the feature fusion model, respectively, according to the fusion feature and the plurality of vision feature blocks, by the feature fusion model,
The feature fusion model randomly covers some vision features of the fusion features, and predicts and supplements the covered some vision features according to the plurality of vision feature blocks to obtain some vision features after prediction and complementation. ; and
adjusting parameters of the text detection model and the feature fusion model, respectively, according to some vision features after prediction and complementation and features other than the some vision features covered by the fusion feature; including,
Training method of a text recognition model framework, characterized in that. According to claim 5,
Adjusting, by the feature fusion model, parameters of the text detection model and the feature fusion model, respectively, according to the fusion feature and the plurality of text feature blocks,
subjecting, by the feature fusion model, text features of the fusion feature to replacement processing according to at least some of the text feature blocks of the plurality of text feature blocks, to obtain replaced text features; and
adjusting parameters of the text detection model and the feature fusion model, respectively, according to the vision feature of the fusion feature and the replaced text feature; including,
Training method of a text recognition model framework, characterized in that. According to claim 5,
The step of adjusting parameters of the text detection model and the feature fusion model, respectively, according to the fusion feature and the plurality of vision feature blocks, by the feature fusion model,
subjecting the feature fusion model to replacement processing for a vision feature of the fusion feature according to at least some of the vision feature blocks of the plurality of vision feature blocks, so as to obtain a replaced vision feature; and
adjusting parameters of the text detection model and the feature fusion model, respectively, according to the text feature of the fusion feature and the replaced vision feature; including,
Training method of a text recognition model framework, characterized in that. According to any one of claims 5 to 9,
When the feature fusion model adjusts parameters of the text detection model and the feature fusion model, respectively, according to the fusion feature, the plurality of text feature blocks, and the plurality of vision feature blocks, the text detection model and the feature fusion model The step of adjusting each parameter of
determining, by the feature fusion model, a first adjustment task result of adjusting the text detection model and the feature fusion model according to the fusion feature and the plurality of text feature blocks;
determining, by the feature fusion model, a second adjustment task result of adjusting the text detection model and the feature fusion model according to the fusion feature and the plurality of vision feature blocks; and
adjusting parameters of the text detection model and the feature fusion model, respectively, according to the weighted average information of the result of the first adjustment task and the result of the second adjustment task; including,
Training method of a text recognition model framework, characterized in that. In the text recognition method,
obtaining an image to be recognized;
Acquiring text information of the image to be recognized by inputting the image to be recognized into a pretrained text recognition model - The text recognition model is generated by training the image to be trained based on a pretrained text recognition model framework The text recognition model framework is obtained by training by the training method of any one of claims 1 to 10, and the image to be trained includes text information; including,
Text recognition method characterized in that. In the training apparatus of the text recognition model framework,
a processing unit configured to perform feature processing on a sample image based on a preset text detection model to obtain at least two pieces of feature information related to text information of the sample image;
a fusion unit configured to obtain a fusion feature of the sample image by performing fusion processing on at least two pieces of feature information of the sample image based on a preset feature fusion model; and
A training unit configured to input the fusion features into the feature fusion model and adjust parameters of the text detection model and the feature fusion model based on the fusion feature model, respectively, to obtain a text recognition model framework - the text recognition model - The framework includes a tuned text detection model and a tuned feature fusion model; including,
Training device of the text recognition model framework, characterized in that. According to claim 12,
The processing unit is
a first decision subunit that performs feature processing on sample images based on a preset text detection model;
a second determining subunit to acquire at least two pieces of feature information related to the text of the sample image; including,
Training device of the text recognition model framework, characterized in that. According to claim 13,
The second decision subunit,
a cropping module that obtains a text area by performing a cropping operation on the sample image according to the location information; and
an acquiring module acquiring the at least two pieces of characteristic information in the text area; including,
Training device of the text recognition model framework, characterized in that. According to claim 14,
The acquisition module is used to extract image features of the sample image from the text area, recognize the image features, and obtain the at least two pieces of feature information.
Training device of the text recognition model framework, characterized in that. According to any one of claims 13 to 15,
The at least two pieces of feature information include a text feature and a vision feature;
a building unit configured to construct a plurality of text feature blocks representing the text feature, and constructing a plurality of vision feature blocks representing the vision feature;
The training unit may: adjust parameters of the text detection model and the feature fusion model, respectively, according to the feature fusion model according to the fusion feature and the plurality of text feature blocks; And, the feature fusion model is used for at least one of adjusting parameters of the text detection model and the feature fusion model, respectively, according to the fusion feature and the plurality of vision feature blocks,
Training device of the text recognition model framework, characterized in that. According to claim 16,
The training unit is
a first cover subunit randomly covering some text features of the fusion features;
a first prediction and complementation subunit configured to perform prediction and supplementation processing on some of the covered text features according to the plurality of text feature blocks, and obtain some text features after prediction and complementation; and
a first adjustment subunit configured to adjust parameters of the text detection model and the feature fusion model, respectively, according to some text features after prediction and complementation and features other than the some text features covered by the fusion feature; including,
Training device of the text recognition model framework, characterized in that. According to claim 16,
The training unit is
a second cover subunit randomly covering some vision features of the fusion features;
a second prediction and complementation subunit configured to perform prediction and complementation processing on some of the covered vision features according to the plurality of vision feature blocks, and obtain some vision features after prediction and complementation; and
a second adjustment subunit configured to adjust parameters of the text detection model and the feature fusion model, respectively, according to some vision features after prediction and complementation and features other than the some vision features covered by the fusion features; including,
Training device of the text recognition model framework, characterized in that. According to claim 16,
The training unit is
a first replacing sub-unit configured to perform replacement processing on text features of the fusion feature according to at least some text feature blocks of the plurality of text feature blocks, so as to obtain replaced text features; and
a third adjustment subunit configured to adjust parameters of the text detection model and the feature fusion model, respectively, according to the vision feature of the fusion feature and the replaced text feature; including,
Training device of the text recognition model framework, characterized in that. According to claim 16,
The training unit is
a second replacing sub-unit configured to perform replacement processing on vision features of the fusion feature according to at least some vision feature blocks of the plurality of vision feature blocks, so as to obtain replaced vision features; and
a fourth adjustment subunit configured to adjust parameters of the text detection model and the feature fusion model, respectively, according to the text feature of the fusion feature and the replaced vision feature; including,
Training device of the text recognition model framework, characterized in that. The method of any one of claims 16 to 20,
When the feature fusion model adjusts parameters of the text detection model and the feature fusion model respectively according to the fusion feature, the plurality of text feature blocks, and the plurality of vision feature blocks, the training unit:
a third determining subunit configured to determine a first adjusting task result of adjusting the text detection model and the feature fusion model according to the fusion feature and the plurality of text feature blocks;
a fourth decision subunit configured to determine a result of a second adjustment task for adjusting the text detection model and the feature fusion model according to the fusion feature and the plurality of vision feature blocks; and
a fifth adjustment subunit configured to adjust parameters of the text detection model and the feature fusion model, respectively, according to the weighted average information of the result of the first adjustment task and the result of the second adjustment task; including,
Training device of the text recognition model framework, characterized in that. In the text recognition device,
an acquiring unit acquiring an image to be recognized;
A recognition unit for obtaining text information of the image to be recognized by inputting the image to be recognized into a pretrained text recognition model, wherein the text recognition model trains the image to be trained based on a pretrained text recognition model framework generated, the text recognition model framework is obtained by training by the training method of any one of claims 1 to 10, and the image to be trained includes text information; including,
Text recognition device characterized in that. In electronic devices,
at least one processor; and
a memory communicatively coupled to the at least one processor; including,
Instructions executable by the at least one processor are stored in the memory, and when the instructions are executed by the at least one processor, the at least one processor performs the method of any one of claims 1 to 10. perform; Or, the at least one processor performs the method of claim 11,
Electronic device characterized in that. A non-transitory computer-readable storage medium in which computer instructions are stored,
The computer instructions cause a computer to perform the method of any one of claims 1 to 10; Alternatively, the computer instructions cause the computer to perform the method of claim 11,
A non-transitory computer-readable storage medium, characterized in that. A computer program product comprising a computer program,
When the computer program is executed by a processor, the method of any one of claims 1 to 10 is implemented; Alternatively, when the computer program is executed by a processor, the method of claim 11 is implemented.
A computer program product, characterized in that. In the training system of the text recognition model framework,
a text detection model for obtaining at least two pieces of feature information related to text information of the sample image by performing feature processing on the sample image;
a feature fusion model for obtaining a fusion feature of the sample image by performing fusion processing on at least two pieces of feature information of the sample image; including,
The feature fusion model is further used to obtain a text recognition model framework by adjusting parameters of the text detection model and the feature fusion model respectively, and the text recognition model framework adjusts the text detection model and the adjusted feature fusion model. including the model,
Training system of the text recognition model framework, characterized in that. In the computer program containing the program code,
When a computer executes the computer program, the program code performs the method of any one of claims 1 to 10; Or, the program code performs the method of claim 11,
A computer program characterized in that.

Images