CN110889469B - Image processing method and device, electronic equipment and storage medium - Google Patents
Image processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN110889469B CN110889469B CN201911337804.XA CN201911337804A CN110889469B CN 110889469 B CN110889469 B CN 110889469B CN 201911337804 A CN201911337804 A CN 201911337804A CN 110889469 B CN110889469 B CN 110889469B
- Authority
- CN
- China
- Prior art keywords
- text
- network
- image
- text image
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003860 storage Methods 0.000 title claims abstract description 29
- 238000003672 processing method Methods 0.000 title claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 126
- 238000000034 method Methods 0.000 claims abstract description 58
- 238000012549 training Methods 0.000 claims abstract description 56
- 238000004590 computer program Methods 0.000 claims description 15
- 238000005070 sampling Methods 0.000 claims description 8
- 230000008439 repair process Effects 0.000 claims description 6
- 230000007547 defect Effects 0.000 claims 4
- 230000004913 activation Effects 0.000 description 22
- 238000010586 diagram Methods 0.000 description 20
- 230000006870 function Effects 0.000 description 15
- 230000008569 process Effects 0.000 description 13
- 238000004891 communication Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 8
- 238000002372 labelling Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 238000003491 array Methods 0.000 description 3
- 239000004973 liquid crystal related substance Substances 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4046—Scaling the whole image or part thereof using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
Abstract
The present disclosure relates to an image processing method and apparatus, an electronic device, and a storage medium, the method including: acquiring an original text image; and carrying out image quality enhancement processing on the original text image through a generating network to obtain an enhanced text image, wherein the generating network is obtained by carrying out auxiliary training based on a first text recognition network. The embodiment of the disclosure can improve the image quality of the original text image, thereby improving the text recognition precision.
Description
Technical Field
The disclosure relates to the field of computer technology, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium.
Background
With the development of related technology, text recognition has been widely used. Text recognition can be applied in many important fields, for example: reading understanding, invoice recognition, license plate recognition, visual question and answer and other fields. However, the accuracy of text recognition is greatly related to image quality, and text recognition accuracy is low for images with poor image quality, such as images with low resolution, broken images, stained images, corroded images, images with scratches, and the like.
Disclosure of Invention
The disclosure provides an image processing method and device, electronic equipment and a storage medium.
According to an aspect of the present disclosure, there is provided an image processing method including: acquiring an original text image; and carrying out image quality enhancement processing on the original text image through a generating network to obtain an enhanced text image, wherein the generating network is obtained by carrying out auxiliary training based on a first text recognition network.
In one possible implementation, the image quality enhancement process includes a text repair process.
In one possible implementation, the generation network includes at least one first convolution layer with a step length greater than 1, at least one second convolution layer with a step length of 1, and at least one deconvolution layer; the step of performing quality enhancement processing on the original text image through a generation network to obtain an enhanced text image comprises the following steps: downsampling the original text image through the at least one first convolution layer to obtain a downsampling result; convolving the downsampling result through the at least one second convolution layer to obtain a convolution result; and carrying out up-sampling processing on the convolution result through the at least one deconvolution layer to obtain the enhanced text image.
In one possible implementation, the image processing includes super-resolution processing, wherein the resolution of the enhanced text image is higher than the resolution of the original text image.
In one possible implementation, the generation network includes at least one second convolution layer and at least one deconvolution layer with a step size of 1; the step of performing quality enhancement processing on the original text image through a generation network to obtain an enhanced text image comprises the following steps: carrying out convolution processing on the original text image through the at least one second convolution layer to obtain a convolution result; and carrying out up-sampling processing on the convolution result through the at least one deconvolution layer to obtain the enhanced text image.
In one possible implementation, the training sample of the generated network includes the original text image and corresponding labeled text information; the method further comprises the steps of: performing text recognition processing on the enhanced text image by using the first text recognition network to obtain a first text recognition result; and adjusting network parameters of the generated network based on the difference between the marked text information and the first text recognition result.
In a possible implementation manner, the training sample further comprises a reference text image corresponding to the original text image; the method further comprises the steps of: and adjusting network parameters of the generated network according to the difference between the reference text image and the enhanced text image.
In one possible implementation, the generating network is obtained by performing auxiliary training based on the first text recognition network and the discrimination network.
In one possible implementation, the method further includes: and carrying out text recognition processing on the enhanced text image by using a second text recognition network to obtain a second text recognition result.
According to an aspect of the present disclosure, there is provided an image processing apparatus including: the acquisition module is used for acquiring an original text image; the image processing module is used for carrying out image quality enhancement processing on the original text image through a generating network to obtain an enhanced text image, wherein the generating network is obtained by carrying out auxiliary training based on a first text recognition network.
In one possible implementation, the image quality enhancement process includes a text repair process.
In one possible implementation, the generation network includes at least one first convolution layer with a step length greater than 1, at least one second convolution layer with a step length of 1, and at least one deconvolution layer; the image processing module comprises: the downsampling submodule is used for downsampling the original text image through the at least one first convolution layer to obtain a downsampling result; the convolution submodule is used for carrying out convolution processing on the downsampling result through the at least one second convolution layer to obtain a convolution result; and the upsampling sub-module is used for upsampling the convolution result through the at least one deconvolution layer to obtain the enhanced text image.
In one possible implementation, the image processing includes super-resolution processing, wherein the resolution of the enhanced text image is higher than the resolution of the original text image.
In one possible implementation, the generation network includes at least one second convolution layer and at least one deconvolution layer with a step size of 1; the image processing module comprises: the convolution sub-module is used for carrying out convolution processing on the original text image through the at least one second convolution layer to obtain a convolution result; and the upsampling sub-module is used for upsampling the convolution result through the at least one deconvolution layer to obtain the enhanced text image.
In one possible implementation, the training sample of the generated network includes the original text image and corresponding labeled text information; the apparatus further comprises: the first text recognition module is used for carrying out text recognition processing on the enhanced text image by utilizing the first text recognition network to obtain a first text recognition result; and the first parameter adjustment module is used for adjusting the network parameters of the generated network based on the difference between the marked text information and the first text recognition result.
In a possible implementation manner, the training sample further comprises a reference text image corresponding to the original text image; the apparatus further comprises: and the second parameter adjustment module is used for adjusting network parameters of the generating network according to the difference between the reference text image and the enhanced text image.
In one possible implementation, the generating network is obtained by performing auxiliary training based on the first text recognition network and the discrimination network.
In one possible implementation, the apparatus further includes: and the second text recognition module is used for carrying out text recognition processing on the enhanced text image by using a second text recognition network to obtain a second text recognition result.
According to an aspect of the present disclosure, there is provided an electronic apparatus including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above method.
According to an aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described method.
In the embodiment of the disclosure, an original text image is acquired, and the image quality enhancement processing is performed on the original text image through a generating network obtained by performing auxiliary training based on a first text recognition network to obtain an enhanced text image, so that the image quality of the original text image can be improved, and the text recognition precision can be improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the technical aspects of the disclosure.
FIG. 1 shows a flow chart of an image processing method of an embodiment of the present disclosure;
FIG. 2 illustrates a schematic diagram of a generation network in accordance with an embodiment of the present disclosure;
FIG. 3 illustrates a schematic diagram of a generation network in accordance with an embodiment of the present disclosure;
fig. 4 shows a block diagram of an image processing apparatus of an embodiment of the present disclosure;
FIG. 5 illustrates a block diagram of an electronic device of an embodiment of the present disclosure;
Fig. 6 shows a block diagram of an electronic device of an embodiment of the disclosure.
Detailed Description
Various exemplary embodiments, features and aspects of the disclosure will be described in detail below with reference to the drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. Although various aspects of the embodiments are illustrated in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
The word "exemplary" is used herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.
The term "and/or" is herein merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.
Furthermore, numerous specific details are set forth in the following detailed description in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order not to obscure the present disclosure.
Fig. 1 shows a flowchart of an image processing method of an embodiment of the present disclosure. The image processing method may be performed by a terminal device or other processing device, wherein the terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle mounted device, a wearable device, etc. The other processing device may be a server or cloud server, etc. In some possible implementations, the image processing method may be implemented by way of a processor invoking computer readable instructions stored in a memory. It should be appreciated that the image processing method may be applied to both an inference phase for image quality enhancement processing of an image using a generation network and a training phase for training the generation network, which is not specifically limited in the embodiments of the present disclosure. As shown in fig. 1, the method may include:
step S11, an original text image is acquired.
And step S12, performing image quality enhancement processing on the original text image through a generating network to obtain an enhanced text image, wherein the generating network is obtained by performing auxiliary training based on the first text recognition network.
After the original text image is acquired, the image quality enhancement processing is carried out on the original text image through a generating network which is obtained by carrying out auxiliary training based on the first text recognition network, so that the enhanced text image is obtained, the image quality of the original text image can be improved, and the text recognition precision can be improved.
The generating network and the first text recognition network may be a neural network, or may be other network models that can implement corresponding processing, which is not specifically limited in this disclosure.
In one possible implementation, the image quality enhancement process includes a text repair process.
In the case that the text information in the original text image is missing (for example, the original text image is a damaged image, a polluted image, a corroded image, a scratched image and the like), the original text image can be subjected to text restoration processing through a generating network, so that an enhanced text image with the integrity of the text information being higher than that of the original text image is obtained, and the image quality of the original text image is improved.
In one possible implementation, the generation network includes at least one first convolution layer with a step length greater than 1, at least one second convolution layer with a step length of 1, and at least one deconvolution layer; performing image quality enhancement processing on the original text image through a generating network to obtain an enhanced text image, wherein the method comprises the following steps: performing downsampling processing on the original text image through at least one first convolution layer to obtain a downsampling result; convolving the downsampling result through at least one second convolution layer to obtain a convolution result; and up-sampling the convolution result through at least one deconvolution layer to obtain an enhanced text image.
Fig. 2 shows a schematic diagram of a structure of a generation network according to an embodiment of the present disclosure. As shown in fig. 2, the generation network may include a first convolution layer (Conv layer) 201 with a step length greater than 1, an activation layer (ReLU layer) 202, a plurality of residual modules 203, a deconvolution layer (DeConv layer) 204, an activation layer (ReLU layer) 205, a deconvolution layer (DeConv layer) 206, an activation layer (ReLU layer) 207, and a second convolution layer (Conv layer) 208 with a step length of 1.
The generating network performs a downsampling process on the original text image 209 through the first convolution layer 201, reduces the resolution of the original text image, and obtains a downsampling result, where the downsampling result includes text feature information extracted from the original text image, for example, semantic information. After the activation layer 202 performs activation processing on the downsampling result, the activated downsampling result is input into a residual module 203, residual of the activated downsampling result is determined by the residual module 203, summation processing is performed on the residual and the activated downsampling result to obtain first data, and residual calculation and summation calculation of the obtained residual and input data of the residual module are sequentially performed according to the first data and other residual modules to obtain residual processing data. The deconvolution layer 204 performs up-sampling (i.e., deconvolution) on the residual processing data to obtain a first deconvolution result, and then inputs the first deconvolution result to the activation layer 205. After performing activation processing on the first deconvolution result, the activation layer 205 inputs the activated first deconvolution result into the deconvolution layer 206. The deconvolution layer 206 upsamples (i.e., deconvolutes) the activated deconvolution result to obtain a second deconvolution result, and inputs the second deconvolution result to the activation layer 207. The activation layer 207 performs an activation operation on the second deconvolution result to obtain an activated second deconvolution result. The activated second deconvolution result is convolved by the second convolution layer 208 to obtain the enhanced text image 210. The text information integrity of the enhanced text image is higher than that of the original text image, and therefore, the image quality of the enhanced text image is higher than that of the original text image. The first convolution layer may implement a downsampling process for input data to extract text feature information from an original text image of low image quality; the deconvolution layer may implement an upsampling process for the input data to construct an enhanced text image that has the same resolution as the original text image and a higher image quality than the original text image. The number of first convolution layers, second convolution layers, and deconvolution layers, and the locations in the generation network, may vary depending on the actual situation, which is not specifically limited by the present disclosure.
In one possible implementation, the image processing includes super-resolution processing in which the resolution of the enhanced text image is higher than the resolution of the original text image.
In the case where the resolution in the original text image is low (e.g., the original text image is a low resolution image), the original text image may be super-resolution processed through the generation network to obtain an enhanced text image (e.g., a super-resolution image) having a higher resolution than the original text image, so as to enhance the image quality of the original text image.
In one possible implementation, the generation network comprises at least one second convolution layer and at least one deconvolution layer with a step size of 1; performing image quality enhancement processing on the original text image through a generating network to obtain an enhanced text image, wherein the method comprises the following steps: carrying out convolution processing on the original text image through at least one second convolution layer to obtain a convolution result; and up-sampling the convolution result through at least one deconvolution layer to obtain an enhanced text image.
Fig. 3 shows a schematic diagram of a structure of a generation network according to an embodiment of the present disclosure. As shown in fig. 3, the generation network may include a second convolution layer (Conv layer) 301 with a step size of 1, an activation layer (ReLU layer) 302, a plurality of residual modules 303, a deconvolution layer (DeConv layer) 304, an activation layer (ReLU layer) 305, a deconvolution layer (DeConv layer) 306, an activation layer (ReLU layer) 307, and a second convolution layer (Conv layer) 308 with a step size of 1.
The generating network performs convolution processing on the original text image 309 through the second convolution layer 301 to obtain a first convolution result, where the first convolution result includes text feature information, for example, semantic information, extracted from the original text image. After the activation layer 302 performs activation processing on the first convolution result, the activated first convolution result is input into a residual module 303, a residual of the activated first convolution result is determined by the residual module 303, summation processing is performed on the residual and the activated first convolution result, first data is obtained, and residual calculation and summation calculation of the obtained residual and input data of the residual module are sequentially performed according to the first data and other residual modules, so that residual processing data is obtained. The deconvolution layer 304 performs up-sampling (i.e., deconvolution) on the residual processing data to obtain a first deconvolution result, and then inputs the first deconvolution result to the activation layer 305. After performing activation processing on the first deconvolution processing result, the activation layer 305 inputs the activated first deconvolution processing result into the deconvolution layer 306. The deconvolution layer 306 upsamples (i.e., deconvolutes) the activated first deconvolution result to obtain a second deconvolution result, and inputs the second deconvolution result to the activation layer 307. After performing the activation operation on the second deconvolution result, the activation layer 307 obtains the activated second deconvolution result. The activated second deconvolution result is convolved by a second convolution layer 308 to obtain an enhanced text image 310 (e.g., a super-resolution image). The resolution of the enhanced text image is higher than that of the original text image, and thus, the image quality of the enhanced text image is higher than that of the original text image. The deconvolution layer may implement an upsampling process for the input data to construct an enhanced text image having a higher resolution than the original text image and a higher image quality than the original text image. The number of second convolution layers and deconvolution layers and the locations in the generation network may vary depending on the actual situation, which is not specifically limited by the present disclosure.
In one possible implementation, the method further includes: and carrying out text recognition processing on the enhanced text image by using a second text recognition network to obtain a second text recognition result.
Because the generating network carries out image quality enhancement processing on the original text image, the obtained enhanced text image has higher image quality, and the enhanced text image can be subjected to text recognition processing through the second text recognition network to obtain a second text recognition result, thereby improving the text recognition precision. The first text recognition network and the second text recognition network may be the same or different, and this disclosure is not limited in detail.
In one possible implementation, the training sample of the generated network includes an original text image and corresponding annotated text information; the method further comprises the steps of: performing text recognition processing on the enhanced text image by using a first text recognition network to obtain a first text recognition result; and adjusting network parameters of the generated network based on the difference between the marked text information and the first text recognition result.
In an embodiment of the present disclosure, a training sample of the generating network may be created in advance, the training sample may include an original text image and labeling text information corresponding to the original text image (the labeling text information corresponding to the original text image is text information included in the original text image), and the generating network may be trained based on the first text recognition network and the training sample. The first text recognition network may be a pre-trained network, or may be co-trained with the generation network, which is not specifically limited in this disclosure.
In the embodiment of the present disclosure, after the generating network generates the enhanced text image corresponding to the original text image, the enhanced text image may be input to the first text recognition network to perform text recognition processing, so as to obtain first text information included in the enhanced text image, that is, a first text recognition result. Wherein the first text recognition network may be a convolutional neural network for text recognition.
As shown in fig. 2, the first text recognition network performs text recognition processing on the enhanced text image 210 to obtain a first text recognition result. As shown in fig. 3, the first text recognition network performs text recognition processing on the enhanced text image 310 to obtain a first text recognition result.
In one possible implementation, the first text recognition network includes a text rectifier sub-network and a text recognition sub-network; performing text recognition processing on the enhanced text image by using a first text recognition network to obtain a first text recognition result, wherein the text recognition result comprises: rectifying texts in the enhanced text image through a text rectifying sub-network to obtain a rectified text sequence; and carrying out text recognition processing on the text sequence through a text recognition sub-network to obtain a first text recognition result.
The text rectification subnetwork can rectify text in the enhanced text image, for example: and performing operations such as character correction, sequencing, position and font size adjustment on the text in the enhanced text image so as to adjust the text in the enhanced text image into a standard format and obtain a rectified text sequence.
The text recognition sub-network can perform text recognition processing on the rectified text sequence obtained by the text rectification sub-network to obtain first text information included in the enhanced text image, and a first text recognition result is obtained. As an example, as shown in fig. 2 and 3, the first text recognition sub-network may be composed of an encoder and a decoder, wherein the encoder may be configured to extract text feature information in the rectified enhanced text image and convert the text feature information into a feature sequence, and the decoder is configured to predict a character sequence according to the feature sequence, so as to obtain the first text information included in the enhanced text image.
After the text in the enhanced text image is rectified based on the text rectification sub-network, text recognition processing is performed on the rectified text sequence through the text recognition sub-network, so that the text recognition precision can be improved.
After the enhanced text image is subjected to text recognition processing through the first text recognition network to obtain a first text recognition result, according to the labeling text information corresponding to the original text image and the difference between the first text recognition results obtained through the first text recognition network, a first recognition loss of the first text recognition network for the original text image can be determined, for example: the first recognition loss may be determined by the following equation (one):
wherein t represents the length of the character sequence of the text information, I SR Representing enhanced text image, y t T-th character, p (y) t |I SR ) T character, l representing output of first text recognition result obtained by first text recognition network recognition TR Representing a first recognition penalty of the first text recognition network for the original text image.
In the case that a plurality of original text images are included in the training sample, a first generation penalty of the generation network may be determined from a first recognition penalty of the first text recognition network for each original text image, for example: the average of the first recognition losses of all the original text images may be determined as a first generation loss of the generation network, which may be determined by the following equation (two):
Wherein G represents a generation network, and n represents an originalThe number of the text image, N represents the total number of the original text images,representing the nth original text image, +.>Representing an enhanced text image corresponding to the nth original text image, y n Marking text information corresponding to the nth original text image,>representing a first recognition penalty of the first text recognition network for the nth original text image.
After determining a first generation penalty for the generation network based on a first recognition penalty of the first text recognition network for the original text image, the generation network may be back-trained based on the first generation penalty, for example: and adjusting network parameters of the generating network according to the first generating loss until the first generating loss meets the training requirement of the generating network.
In the embodiment of the disclosure, the first text recognition network may be trained in advance before the generation network is trained, or the first text recognition network may be trained while the generation network is trained.
In one possible implementation, the training sample further includes a reference text image corresponding to the original text image, and the method further includes: aiming at a reference text image corresponding to the original text image, performing text recognition processing on the reference text image through a first text recognition network to obtain a second text recognition result; and adjusting network parameters of the first text recognition network according to the labeling text information corresponding to the original text image and the second text recognition result.
Training a first text recognition network in advance according to a training sample, and after training of the first text recognition network is completed, supervising and generating training of the network according to the first text recognition network obtained after training.
The training process for the first text recognition network specifically includes: and carrying out text recognition processing on the reference text image corresponding to the original text image through the first recognition network to obtain second text information contained in the reference text image, namely obtaining a second text recognition result, determining second text recognition loss of the first text recognition network according to the difference between the labeling text information corresponding to the original text image and the second text information contained in the reference text image, and further adjusting network parameters of the first text recognition network according to the second text recognition loss until the second text recognition loss of the first text recognition network meets the training requirement of the first text recognition network, and stopping training of the first text recognition network to obtain the trained first text recognition network.
After training of the first text recognition network is completed, text recognition processing can be carried out on the enhanced text image generated by the generation network through the first text recognition network, a first text recognition result is obtained, and network parameters of the generation network are adjusted according to the first text recognition result and the difference of the marked text information corresponding to the original text image.
Before the generating network is trained, the first text recognition network is trained in advance, and the training of the generating network is supervised through the trained first text recognition network, so that the convergence speed of the generating network can be increased, and the generating network can generate enhanced text images with higher image quality.
In one possible implementation, the method further includes: the first text recognition network is trained based on the first recognition penalty.
The first text recognition network may be trained simultaneously in the course of training the generation network. After determining the first recognition loss of the first text recognition network for the original text image, network parameters of the first text recognition network can be adjusted according to the first recognition loss until the first recognition loss meets the training requirement of the recognition network.
In one possible implementation, the method further includes: and adjusting network parameters of the generated network according to the difference between the reference text image and the enhanced text image.
The reference text image corresponding to the original text image is consistent with the content included in the original text image, and the image quality of the reference text image is higher than that of the original text image. From the difference between the reference text image corresponding to the original text image and the enhanced text image, a second generation penalty of the generation network may be determined, for example: the second generated loss may be determined using a loss function such as a 0-1 loss function, a square loss function, an absolute loss function, an exponential loss function, etc., and the present disclosure is not particularly limited as to the function for determining the second generated loss of the generation network.
After determining a second generation penalty for the generation network based on the difference between the reference text image corresponding to the original text image and the enhanced text image, the generation network may be back-propagated based on the second generation penalty to train, for example: and adjusting network parameters of the generating network according to the second generating loss until the second generating loss meets the training requirement of the generating network.
In one possible implementation, network parameters of the generation network are comprehensively adjusted according to the first generation loss and the second generation loss.
After determining a first generation penalty of the generation network based on a first recognition penalty of the first text recognition network on the original text image, the comprehensive training generation network may be counter-propagated in conjunction with a second generation penalty of the generation network, such as: and comprehensively adjusting network parameters of the generating network according to the first generating loss and the second generating loss until the first generating loss and the second generating loss meet the training requirement of the generating network.
In one possible implementation, the comprehensively adjusting the network parameters of the generating network according to the first generating loss and the second generating loss includes: performing weighted summation operation on the first generation loss and the second generation loss to obtain a first total loss of the generation network; and adjusting network parameters of the generated network according to the first total loss. The first total loss of the generated network may be determined by the following equation (three):
Wherein, the liquid crystal display device comprises a liquid crystal display device,representing a reference text image corresponding to the nth original text image,>and (3) representing a weighted sum of the first generation loss and the second generation loss corresponding to the nth original text image.
The method comprises the steps of performing image quality enhancement processing on an original text image through a generation network to obtain an enhanced text image, determining second generation loss of the generation network according to a reference text image corresponding to the original text image and the original text image, determining first generation loss of the generation network according to a first text recognition result obtained by performing text recognition processing on the enhanced text image through a first text recognition network and labeling text information corresponding to the original text image, and monitoring training of the generation network through back propagation of the first generation loss and the second generation loss, so that convergence speed of the generation network is increased, the generation network can generate the enhanced text image with higher image quality, and text recognition accuracy of the original text image is improved.
In one possible implementation, the generation network is based on a first text recognition network and a discrimination network for training assistance.
The enhanced text image generated by the generation network and the reference text image corresponding to the original text image are input into the discrimination network, as shown in fig. 2, the enhanced text image 210 and the reference text image 211 are input into the discrimination network, as shown in fig. 3, and the enhanced text image 310 and the reference text image 311 are input into the discrimination network. The judging network judges whether the input image is a reference text image or an enhanced text image, and a judging result of the enhanced text image and a judging result of the reference text image are obtained. And determining countermeasures loss according to the discrimination result of the enhanced text image and the discrimination result of the reference text image, and adjusting network parameters of the generating network and the discrimination network according to the countermeasures loss, namely performing countermeasures training on the generating network and the discrimination network.
The generating network and the discriminating network can be subjected to countermeasure training through the following formula (IV):
wherein D represents a discrimination network, I HR Representing a reference text image corresponding to the original text image, I LR Representing the original text image of the document,representing a loss of discrimination in the case of a probability distribution of the reference text image subject to the training samples,d represents a discriminant loss of the original text image subject to the output of the generation network θD (I HR ) Representing the probability that the discrimination network determines the reference text image as the reference text image, D θD (G θG (I LR ) A probability that the discrimination network determines the enhanced text image as the reference text image.
In one possible implementation, the method further includes: obtaining a second overall loss of the generated network according to the first generation loss, the second generation loss and the countering loss; and adjusting network parameters of the generating network and the judging network according to the second total loss.
After determining the countermeasures loss, a weighted summation operation can be performed on the first generating loss, the second generating loss and the countermeasures loss of the generating network to obtain a second total loss of the generating network, and network parameters of the generating network are adjusted according to the second total loss.
And carrying out image quality enhancement processing on the original text image through the generation network after parameter adjustment to obtain an enhanced text image, respectively judging the enhanced text image and a reference text image corresponding to the original text image through the judging network, determining the countermeasure loss of the countermeasure training according to the judging result, and adjusting the parameters of the judging network according to the countermeasure loss. And circularly executing the process of carrying out parameter adjustment on the generating network and the judging network according to the countermeasures loss so as to optimize the generating network and the judging network until the countermeasures loss meets the countermeasures training requirement. In optimizing the generation network, it is desirable that V (D, G) be minimized, i.e., the generation network can generate enhanced text images that approach infinitely to the reference text image corresponding to the original text image. When the discrimination network is optimized, the expected V (D, G) is maximized, that is, the discrimination network can determine whether the input image is the reference text image corresponding to the original text image or the enhanced text image generated by the generation network. In the training process, the generating network tries to confuse the reference text image corresponding to the enhanced text image and the original text image, and the distinguishing network tries to distinguish the reference text image corresponding to the enhanced text image and the original text image, and the two are mutually opposed. In the optimization process, the two networks respectively promote the generating and distinguishing capabilities, and finally the optimal network parameters are obtained.
In the embodiment of the disclosure, an original text image is acquired, and the image quality enhancement processing is performed on the original text image through a generating network obtained by performing auxiliary training based on a first text recognition network to obtain an enhanced text image, so that the image quality of the original text image can be improved, and the text recognition precision can be improved.
It will be appreciated that the above-mentioned method embodiments of the present disclosure may be combined with each other to form a combined embodiment without departing from the principle logic, and are limited to the description of the present disclosure. It will be appreciated by those skilled in the art that in the above-described methods of the embodiments, the particular order of execution of the steps should be determined by their function and possible inherent logic.
In addition, the disclosure further provides an image processing apparatus, an electronic device, a computer readable storage medium, and a program, where the foregoing may be used to implement any one of the image processing methods provided in the disclosure, and corresponding technical schemes and descriptions and corresponding descriptions referring to method parts are not repeated.
Fig. 4 shows a block diagram of an image processing apparatus of an embodiment of the present disclosure. As shown in fig. 4, the apparatus 40 includes:
An acquisition module 41 for acquiring an original text image;
the image processing module 42 is configured to perform image quality enhancement processing on the original text image through a generating network, where the generating network is obtained by performing auxiliary training based on the first text recognition network.
In one possible implementation, the image quality enhancement process includes a text repair process.
In one possible implementation, the generation network includes at least one first convolution layer with a step length greater than 1, at least one second convolution layer with a step length of 1, and at least one deconvolution layer;
the image processing module 42 includes:
the downsampling submodule is used for downsampling the original text image through at least one first convolution layer to obtain a downsampling result;
the convolution submodule is used for carrying out convolution processing on the downsampling result through at least one second convolution layer to obtain a convolution result;
and the upsampling sub-module is used for upsampling the convolution result through at least one deconvolution layer to obtain an enhanced text image.
In one possible implementation, the image processing includes super-resolution processing in which the resolution of the enhanced text image is higher than the resolution of the original text image.
In one possible implementation, the generation network comprises at least one second convolution layer and at least one deconvolution layer with a step size of 1;
the image processing module 42 includes:
the convolution sub-module is used for carrying out convolution processing on the original text image through at least one second convolution layer to obtain a convolution result;
and the upsampling sub-module is used for upsampling the convolution result through at least one deconvolution layer to obtain an enhanced text image.
In one possible implementation, the training sample of the generated network includes an original text image and corresponding annotated text information;
the apparatus 40 further comprises:
the first text recognition module is used for carrying out text recognition processing on the enhanced text image by utilizing a first text recognition network to obtain a first text recognition result;
the first parameter adjustment module is used for adjusting network parameters of the generated network based on the difference between the marked text information and the first text recognition result.
In one possible implementation, the training sample further includes a reference text image corresponding to the original text image;
the apparatus 40 further comprises:
and the second parameter adjustment module is used for adjusting network parameters of the generated network according to the difference between the reference text image and the enhanced text image.
In one possible implementation, the generation network is based on a first text recognition network and a discrimination network for training assistance.
In one possible implementation, the apparatus 40 further includes:
and the second text recognition module is used for carrying out text recognition processing on the enhanced text image by utilizing a second text recognition network to obtain a second text recognition result.
In some embodiments, functions or modules included in an apparatus provided by the embodiments of the present disclosure may be used to perform a method described in the foregoing method embodiments, and specific implementations thereof may refer to descriptions of the foregoing method embodiments, which are not repeated herein for brevity.
The disclosed embodiments also provide a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described method. The computer readable storage medium may be a non-volatile computer readable storage medium.
The embodiment of the disclosure also provides an electronic device, which comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above method.
The disclosed embodiments also provide a computer program product comprising computer readable code which, when run on a device, causes a processor in the device to execute instructions for implementing the image processing method as provided in any of the embodiments above.
The disclosed embodiments also provide another computer program product for storing computer readable instructions that, when executed, cause a computer to perform the operations of the image processing method provided in any of the above embodiments.
The electronic device may be provided as a terminal, server or other form of device.
Fig. 5 shows a block diagram of an electronic device of an embodiment of the present disclosure. For example, electronic device 500 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, exercise device, personal digital assistant, or the like.
Referring to fig. 5, an electronic device 500 may include one or more of the following components: a processing component 502, a memory 504, a power supply component 506, a multimedia component 508, an audio component 510, an input/output (I/O) interface 512, a sensor component 514, and a communication component 516.
The processing component 502 generally controls overall operation of the electronic device 500, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the methods described above. Further, the processing component 502 can include one or more modules that facilitate interactions between the processing component 502 and other components. For example, the processing component 502 can include a multimedia module to facilitate interaction between the multimedia component 508 and the processing component 502.
The memory 504 is configured to store various types of data to support operations at the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 504 may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
The power supply component 506 provides power to the various components of the electronic device 500. The power components 506 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 500.
The multimedia component 508 includes a screen between the electronic device 500 and the user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or slide action, but also the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front-facing camera and/or a rear-facing camera. When the electronic device 500 is in an operational mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
The audio component 510 is configured to output and/or input audio signals. For example, the audio component 510 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 500 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 further comprises a speaker for outputting audio signals.
The I/O interface 512 provides an interface between the processing component 502 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.
The sensor assembly 514 includes one or more sensors for providing status assessment of various aspects of the electronic device 500. For example, the sensor assembly 514 may detect an on/off state of the electronic device 500, a relative positioning of components such as a display and keypad of the electronic device 500, a change in position of the electronic device 500 or a component of the electronic device 500, the presence or absence of a user's contact with the electronic device 500, an orientation or acceleration/deceleration of the electronic device 500, and a change in temperature of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 516 is configured to facilitate communication between the electronic device 500 and other devices, either wired or wireless. The electronic device 500 may access a wireless network based on a communication standard, such as WiFi,2G, or 3G, or a combination thereof. In one exemplary embodiment, the communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the electronic device 500 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for executing the methods described above.
In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as memory 504 including computer program instructions executable by processor 520 of electronic device 500 to perform the above-described method.
Fig. 6 shows a block diagram of an electronic device of an embodiment of the disclosure. For example, the electronic device 600 may be provided as a server. Referring to fig. 6, the electronic device 600 includes a processing component 622 that further includes one or more processors and memory resources represented by a memory 632 for storing instructions, such as application programs, executable by the processing component 622. The application programs stored in memory 632 may include one or more modules each corresponding to a set of instructions. Further, the processing component 622 is configured to execute instructions to perform the methods described above.
The electronic device 600 may also include a power component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input-output (I/O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server, mac OS XTM, unixTM, linuxTM, freeBSDTM, or the like.
In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as memory 632 that includes computer program instructions executable by processing component 622 of electronic device 1900 to perform the above-described methods.
The present disclosure may be a system, method, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement aspects of the present disclosure.
The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.
The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.
Computer program instructions for performing the operations of the present disclosure can be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, c++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present disclosure are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information of computer readable program instructions, which can execute the computer readable program instructions.
Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The computer program product may be realized in particular by means of hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied as a computer storage medium, and in another alternative embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), or the like.
The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the improvement of technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (14)
1. An image processing method, comprising:
acquiring an original text image;
performing image quality enhancement processing on the original text image through a generating network to obtain an enhanced text image, wherein the generating network is obtained by performing auxiliary training based on a first text recognition network;
Wherein the image quality enhancement processing includes: text repair processing, wherein the text information integrity of the enhanced text image is higher than that of the original text image;
the generating training samples of the network includes: the original text image, the marked text information corresponding to the original text image and the reference text image, wherein the text information in the original text image is in a defect, the defect comprises at least one of damage, polluted damage, corroded and scratched, the reference text image is consistent with the content included in the original text image, and the image quality of the reference text image is higher than that of the original text image;
the method further comprises the steps of:
performing text recognition processing on the enhanced text image by using the first text recognition network to obtain a first text recognition result;
the generating network performs auxiliary training based on the first text recognition network, and comprises the following steps:
determining a first recognition loss of the first text recognition network to the original text image according to the difference between the marked text information and the first text recognition result;
Determining a first generation loss of the generation network according to the first recognition loss of the first text recognition network on each original text image;
determining a second generation penalty for the generation network based on a difference between the reference text image and the enhanced text image;
and comprehensively adjusting network parameters of the generated network according to the first generation loss and the second generation loss.
2. The method of claim 1, wherein the generation network comprises at least one first convolution layer with a step size greater than 1, at least one second convolution layer with a step size of 1, and at least one deconvolution layer;
the image quality enhancement processing is carried out on the original text image through a generating network to obtain an enhanced text image, and the method comprises the following steps:
downsampling the original text image through the at least one first convolution layer to obtain a downsampling result;
convolving the downsampling result through the at least one second convolution layer to obtain a convolution result;
and carrying out up-sampling processing on the convolution result through the at least one deconvolution layer to obtain the enhanced text image.
3. The method according to claim 1 or 2, wherein the image processing comprises super resolution processing, wherein the resolution of the enhanced text image is higher than the resolution of the original text image.
4. A method according to claim 3, characterized in that the generation network comprises at least one second convolution layer and at least one deconvolution layer with a step size of 1;
the image quality enhancement processing is carried out on the original text image through a generating network to obtain an enhanced text image, and the method comprises the following steps:
carrying out convolution processing on the original text image through the at least one second convolution layer to obtain a convolution result;
and carrying out up-sampling processing on the convolution result through the at least one deconvolution layer to obtain the enhanced text image.
5. The method according to claim 1 or 2, wherein the generating network is based on the first text recognition network and a discriminant network for training assistance.
6. The method according to claim 1 or 2, characterized in that the method further comprises:
and carrying out text recognition processing on the enhanced text image by using a second text recognition network to obtain a second text recognition result.
7. An image processing apparatus, comprising:
the acquisition module is used for acquiring an original text image;
the image processing module is used for carrying out image quality enhancement processing on the original text image through a generating network to obtain an enhanced text image, wherein the generating network is obtained by carrying out auxiliary training based on a first text recognition network;
wherein the image quality enhancement processing includes: text repair processing, wherein the text information integrity of the enhanced text image is higher than that of the original text image;
the generating training samples of the network includes: the original text image, the marked text information corresponding to the original text image and the reference text image, wherein the text information in the original text image is in a defect, the defect comprises at least one of damage, polluted damage, corroded and scratched, the reference text image is consistent with the content included in the original text image, and the image quality of the reference text image is higher than that of the original text image;
the apparatus further comprises:
the first text recognition module is used for carrying out text recognition processing on the enhanced text image by utilizing the first text recognition network to obtain a first text recognition result;
The parameter adjustment module is used for determining a first recognition loss of the first text recognition network to the original text image according to the difference between the marked text information and the first text recognition result; determining a first generation loss of the generation network according to the first recognition loss of the first text recognition network on each original text image; determining a second generation penalty for the generation network based on a difference between the reference text image and the enhanced text image; and comprehensively adjusting network parameters of the generated network according to the first generation loss and the second generation loss.
8. The apparatus of claim 7, wherein the generation network comprises at least one first convolution layer with a step size greater than 1, at least one second convolution layer with a step size of 1, and at least one deconvolution layer;
the image processing module comprises:
the downsampling submodule is used for downsampling the original text image through the at least one first convolution layer to obtain a downsampling result;
the convolution submodule is used for carrying out convolution processing on the downsampling result through the at least one second convolution layer to obtain a convolution result;
And the upsampling sub-module is used for upsampling the convolution result through the at least one deconvolution layer to obtain the enhanced text image.
9. The apparatus according to claim 7 or 8, wherein the image processing comprises super-resolution processing in which the resolution of the enhanced text image is higher than the resolution of the original text image.
10. The apparatus of claim 9, wherein the generation network comprises at least one second convolution layer and at least one deconvolution layer with a step size of 1;
the image processing module comprises:
the convolution sub-module is used for carrying out convolution processing on the original text image through the at least one second convolution layer to obtain a convolution result;
and the upsampling sub-module is used for upsampling the convolution result through the at least one deconvolution layer to obtain the enhanced text image.
11. The apparatus of claim 7 or 8, wherein the generation network is based on training assistance from the first text recognition network and a discrimination network.
12. The apparatus according to claim 7 or 8, characterized in that the apparatus further comprises:
And the second text recognition module is used for carrying out text recognition processing on the enhanced text image by using a second text recognition network to obtain a second text recognition result.
13. An electronic device, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to invoke the instructions stored in the memory to perform the method of any of claims 1 to 6.
14. A computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method of any of claims 1 to 6.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2019108886189 | 2019-09-19 | ||
| CN201910888618.9A CN110633755A (en) | 2019-09-19 | 2019-09-19 | Network training method, image processing method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110889469A CN110889469A (en) | 2020-03-17 |
| CN110889469B true CN110889469B (en) | 2023-07-21 |
Family
ID=68971852
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910888618.9A Pending CN110633755A (en) | 2019-09-19 | 2019-09-19 | Network training method, image processing method and device and electronic equipment |
| CN201911337804.XA Active CN110889469B (en) | 2019-09-19 | 2019-12-23 | Image processing method and device, electronic equipment and storage medium |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910888618.9A Pending CN110633755A (en) | 2019-09-19 | 2019-09-19 | Network training method, image processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN110633755A (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111223040B (en) * | 2020-01-09 | 2023-04-25 | 北京市商汤科技开发有限公司 | Network training method and device, and image generation method and device |
| CN111310764B (en) * | 2020-01-20 | 2024-03-26 | 上海商汤智能科技有限公司 | Network training method, image processing device, electronic equipment and storage medium |
| CN111402156B (en) * | 2020-03-11 | 2021-08-03 | 腾讯科技（深圳）有限公司 | Restoration method and device for smear image, storage medium and terminal equipment |
| CN111461070B (en) * | 2020-04-29 | 2023-12-08 | Oppo广东移动通信有限公司 | Text recognition method, device, electronic equipment and storage medium |
| CN111652093B (en) * | 2020-05-21 | 2023-10-24 | 中国工商银行股份有限公司 | Text image processing method and device |
| CN112419159A (en) * | 2020-12-07 | 2021-02-26 | 上海互联网软件集团有限公司 | Character image super-resolution reconstruction system and method |
| CN112906699A (en) * | 2020-12-23 | 2021-06-04 | 深圳市信义科技有限公司 | Method for detecting and identifying enlarged number of license plate |
| CN113538235B (en) * | 2021-06-30 | 2024-01-09 | 北京百度网讯科技有限公司 | Training method and device for image processing model, electronic equipment and storage medium |
| CN113537151B (en) * | 2021-08-12 | 2023-10-17 | 北京达佳互联信息技术有限公司 | Training method and device for image processing model, image processing method and device |
| CN113591798B (en) * | 2021-08-23 | 2023-11-03 | 京东科技控股股份有限公司 | Method and device for reconstructing text of document, electronic equipment and computer storage medium |
| CN116095509B (en) * | 2021-11-05 | 2024-04-12 | 荣耀终端有限公司 | Method, device, electronic equipment and storage medium for generating video frame |
| CN116091871B (en) * | 2023-03-07 | 2023-08-25 | 哈尔滨工业大学(深圳)(哈尔滨工业大学深圳科技创新研究院) | Physical countermeasure sample generation method and device for target detection model |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107330444A (en) * | 2017-05-27 | 2017-11-07 | 苏州科技大学 | A kind of image autotext mask method based on generation confrontation network |
| CN107679533A (en) * | 2017-09-27 | 2018-02-09 | 北京小米移动软件有限公司 | Character recognition method and device |
| CN109657229A (en) * | 2018-10-31 | 2019-04-19 | 北京奇艺世纪科技有限公司 | A kind of intention assessment model generating method, intension recognizing method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10776903B2 (en) * | 2017-07-17 | 2020-09-15 | Open Text Corporation | Systems and methods for image modification and image based content capture and extraction in neural networks |
-
2019
- 2019-09-19 CN CN201910888618.9A patent/CN110633755A/en active Pending
- 2019-12-23 CN CN201911337804.XA patent/CN110889469B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107330444A (en) * | 2017-05-27 | 2017-11-07 | 苏州科技大学 | A kind of image autotext mask method based on generation confrontation network |
| CN107679533A (en) * | 2017-09-27 | 2018-02-09 | 北京小米移动软件有限公司 | Character recognition method and device |
| CN109657229A (en) * | 2018-10-31 | 2019-04-19 | 北京奇艺世纪科技有限公司 | A kind of intention assessment model generating method, intension recognizing method and device |
Non-Patent Citations (1)
| Title |
|---|
| 双判别器生成对抗网络图像的超分辨率重建方法;袁飘逸等;《激光与光电子学进展》;20190515(第23期);第148-157页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110889469A (en) | 2020-03-17 |
| CN110633755A (en) | 2019-12-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110889469B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN110348537B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN110378976B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN110287874B (en) | Target tracking method and device, electronic equipment and storage medium | |
| CN111445493B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN111783756B (en) | Text recognition method and device, electronic equipment and storage medium | |
| CN111753822A (en) | Text recognition method and device, electronic equipment and storage medium | |
| CN109615006B (en) | Character recognition method and device, electronic equipment and storage medium | |
| CN111931844B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN112465843A (en) | Image segmentation method and device, electronic equipment and storage medium | |
| CN109145970B (en) | Image-based question and answer processing method and device, electronic equipment and storage medium | |
| CN110781813B (en) | Image recognition method and device, electronic equipment and storage medium | |
| CN111539410B (en) | Character recognition method and device, electronic equipment and storage medium | |
| CN111242303B (en) | Network training method and device, and image processing method and device | |
| CN111340048B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN113065591B (en) | Target detection method and device, electronic equipment and storage medium | |
| CN109685041B (en) | Image analysis method and device, electronic equipment and storage medium | |
| CN111583142B (en) | Image noise reduction method and device, electronic equipment and storage medium | |
| CN110633715B (en) | Image processing method, network training method and device and electronic equipment | |
| CN111192218B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN109447258B (en) | Neural network model optimization method and device, electronic device and storage medium | |
| CN113313115B (en) | License plate attribute identification method and device, electronic equipment and storage medium | |
| CN111311588B (en) | Repositioning method and device, electronic equipment and storage medium | |
| CN110826463B (en) | Face recognition method and device, electronic equipment and storage medium | |
| CN109635926B (en) | Attention feature acquisition method and device for neural network and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |