CN114495101A - Text detection method, text detection network training method and device - Google Patents

Abstract

The present disclosure provides a text detection method and a training method of a text detection network, which relate to the technical field of image processing, and in particular, to the technical field of artificial intelligence. The specific implementation scheme is: determine the sequence feature of the image to be detected; determine the decoded sequence vector based on the sequence feature and the instance feature corresponding to the text instance; determine the type of the to-be-detected image based on the decoded sequence vector ; In response to the type of the image to be detected that the image to be detected includes text, determine the position information of the text in the image to be detected based on the decoded sequence vector and the vector corresponding to the sequence feature.

Description

Text detection method, text detection network training method and device

technical field

The present disclosure relates to the technical field of image processing, and in particular, to an artificial intelligence text detection method, a text detection network training method and device.

Background technique

Text detection refers to determining the position of text in an image and finding its bounding box. It is a pre-step for many visual tasks, such as text recognition and scene understanding, etc. It can be widely used in business scenarios such as ID card recognition and bill recognition, thereby greatly saving manual input time and improving the efficiency in various application scenarios.

SUMMARY OF THE INVENTION

The present disclosure provides a text detection method, a text detection network training method and a device.

According to an aspect of the present disclosure, there is provided a text detection method, comprising:

Determine the sequence characteristics of the image to be detected;

Determine the decoded sequence vector based on the sequence feature and the instance feature corresponding to the text instance;

Determine the type of the image to be detected based on the decoded sequence vector;

In response to the type of the image to be detected being that the image to be detected includes text, the location information of the text in the image to be detected is determined based on the decoded sequence vector and the vector corresponding to the sequence feature.

According to another aspect of the present disclosure, a method for training a text detection network is provided, the text detection network includes an encoding sub-network, a decoding sub-network and an output sub-network;

Determine the sequence sample features of the sample images in the training sample set based on the coding sub-network;

Using the sequence sample feature and the instance sample feature corresponding to the text instance sample as the input of the cross-layer attention layer of the decoding sub-network, and determining the output of the cross-layer attention layer as the decoded sample sequence vector;

The decoded sample sequence vector is used as the input of the output sub-network, and the prediction type of the sample image is determined according to the output of the output sub-network;

In response to the prediction type of the sample image being that the sample image includes text, the decoded sample sequence vector and the sample sequence feature are used as the input of the output sub-network, and according to the output of the output sub-network, determining the predicted position information of the text in the sample image;

Match the prediction type of the sample image with the annotation type of the sample image, and the predicted position information and the annotation position information of the text in the sample image, and adjust the parameters of the text detection network based on the matching result.

A third aspect of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, The instructions are executed by the at least one processor to enable the at least one processor to perform the above-described text detection method or text detection network training method.

A fourth aspect of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the above-described text detection method or text detection network training method.

A fifth aspect of the present disclosure provides a computer program product, comprising a computer program/instruction, the computer program/instruction, when executed by a processor, implements the above-described text detection method or text detection network training method.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings are used for better understanding of the present solution, and do not constitute a limitation to the present disclosure. in:

FIG. 1 shows an optional schematic flowchart of a text detection method provided by an embodiment of the present application;

FIG. 2 shows an optional schematic flowchart of a training method for a text detection network provided by an embodiment of the present application;

FIG. 3 shows another optional schematic flowchart of the text detection method provided by the embodiment of the present application;

FIG. 4 shows a data schematic diagram of the text detection method provided by the embodiment of the present application;

FIG. 5 shows an optional structural schematic diagram of the text detection device provided by the embodiment of the present application;

FIG. 6 shows an optional structural schematic diagram of a text detection network training device provided by an embodiment of the present application;

7 shows a schematic block diagram of an example electronic device that may be used to implement embodiments of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

Text detection technology refers to determining the position of text in an image and finding its bounding box. It is a pre-step for many visual tasks, such as text recognition and scene understanding, etc. It can be widely used in business scenarios such as ID card recognition and bill recognition, thereby greatly saving manual input time and improving the efficiency in various application scenarios. Compared with general target detection, text detection in natural scenes has its particularity. As the main visual target, the text, such as font, size, color, direction, shape, etc., presents various characteristics, which is more difficult than general target detection. big. Text detection techniques have developed rapidly in recent years, and these techniques have achieved good results on conventional text detection datasets, but their results are frustrating on challenging natural scene datasets containing arbitrary shapes.

Existing methods are mainly divided into regression-based methods and segmentation-based methods. Regression-based methods often require additional shape modeling to solve the problem of curved text, and still cannot effectively solve text of any shape. Segmentation-based methods can naturally solve the problem. The problem of arbitrary shape, but often requires post-processing rules to distinguish different text entities, and can not achieve the best effect.

Among them, the former (regression-based method) locates the target text by regressing the corresponding bounding box, and the latter usually uses a fully convolutional network to classify and predict the image pixel by pixel, divide the image into text and non-text areas, and then pass Specific post-processing operations convert the pixel-level output into bounding box form. Among them, segmentation-based text detection algorithms mainly use Mask-RCNN as the basic neural network to generate segmentation maps. Although segmentation-based methods have achieved high accuracy in conventional horizontal text detection, they usually require complex post-processing steps. to generate the corresponding bounding boxes, which consumes a lot of memory and time during the inference phase, as the resulting regions are refined and labeled. At the same time, due to the segmentation idea, the detection effect is very poor for overlapping text boxes. The regression-based detection method usually predicts the bounding box directly. Common algorithms such as EAST and CPTN are used. Due to the simple post-processing process, its inference speed is significantly better than the segmentation-based algorithm. However, in complex natural scenes, due to the font size For problems with large changes and serious scene interference, the detection effect is not very good.

To sum up, the existing text detection methods have the following shortcomings:

1) Most of the existing methods can achieve satisfactory detection accuracy for horizontal text boxes. However, when faced with characters of arbitrary shape in natural scenes, there are still a lot of missed detections and false detections due to the limited modeling ability.

2) Most of the existing text detection models are based on the CNN architecture, which requires a lot of artificial prior assumptions and complicated post-processing processes, such as Anchor design and NMS processes, and the overall process is complex.

The present disclosure provides a text detection method and a text detection network training method, so as to at least solve the defects of text detection in the prior art.

FIG. 1 shows an optional schematic flowchart of a text detection method provided by an embodiment of the present application, which will be described according to each step.

Step S101, determining the sequence features of the images to be detected.

In some embodiments, the text detection device (hereinafter referred to as the first device) converts the image matrix corresponding to the image to be detected into a one-dimensional vector, and determines that the feature corresponding to the one-dimensional vector is the sequence feature of the image to be detected.

During specific implementation, the first device may determine the sequence feature of the image to be detected based on the encoding sub-network included in the text detection network; that is, input the image to be detected into the encoding sub-network to obtain the image to be detected sequence features.

Step S102: Determine a decoded sequence vector based on the sequence feature and the instance feature corresponding to the text instance.

In some embodiments, before determining the decoded sequence vector, the first device obtains an embedded value corresponding to the text instance, where the embedded value is an instance feature corresponding to the text instance; wherein the text Instances include strings and/or phrases, and different text instances correspond to different embedded values.

During specific implementation, the first device may obtain the embedded value corresponding to the text instance based on the self-attention layer of the decoding sub-network included in the text detection network; that is, the text instance is used as the self-attention The input of the self-attention layer is the embedded value corresponding to the text instance.

In some embodiments, the first device may obtain the decoded sequence vector based on a cross-layer attention layer included in the decoding sub-network; that is, the sequence feature and the instance corresponding to the text instance The feature is used as the input of the cross-layer attention layer, and the output of the cross-layer attention layer is the decoded sequence vector.

Step S103, based on the decoded sequence vector, determine the type of the image to be detected.

In some embodiments, the first device may determine the type of the image to be detected based on the output sub-network included in the text detection network; that is, the decoded sequence vector is used as the output sub-network included in the output sub-network. The input of the fully connected layer, and the output of the fully connected layer is the type of the image to be detected.

The type of the image to be detected may include that the image to be detected includes text or the image to be detected does not include text.

Step S104, in response to the type of the image to be detected that the image to be detected includes text, determine the position of the text in the image to be detected based on the decoded sequence vector and the vector corresponding to the sequence feature information.

In some embodiments, in response to the type of the image to be detected being that the image to be detected includes text, the first device determines the text based on the decoded sequence vector and a vector corresponding to the sequence feature location information in the image to be detected.

In some embodiments, the first device multiplies the decoded sequence vector and the vector corresponding to the sequence feature to obtain a product result; based on the product result, it is determined that the text is in the sequence feature The position information of the text; based on the position information of the text in the sequence feature, determine the position information of the text in the to-be-detected image. Optionally, a Tensor multiplication operation can be performed on the decoded sequence vector and the vector corresponding to the sequence feature; when performing a Tensor multiplication operation, the decoded sequence vector and the The vectors corresponding to the sequence features have the same length.

In some optional embodiments, after the first apparatus determines the position information of the text in the image to be detected based on the decoded sequence vector and the vector corresponding to the sequence feature, it may also The position information of the text in the image to be detected determines a connected domain in the image to be detected; a text bounding box is determined based on the boundary of the connected domain; wherein, the text bounding box is used to identify the text in the image to be detected.

In this way, the text detection method provided by the embodiments of the present disclosure can directly predict the position of text of any shape at the text instance level in the image, and can adapt to the detection task of text of various shapes in complex scenes. In particular, in the case of text recognition combined with Optical Character Recognition (OCR), the text recognition capability of OCR in natural scenes can be improved.

FIG. 2 shows an optional schematic flowchart of a training method for a text detection network provided by an embodiment of the present application, which will be described according to each step.

Step S201 , based on the coding sub-network, determine the sequence sample features of the sample images in the training sample set.

In some embodiments, the text detection network includes: an encoding sub-network, a decoding sub-network, and an output sub-network; wherein the decoding sub-network includes a self-attention layer and a cross-layer attention layer; the output sub-network The network includes fully connected layers.

In some embodiments, the training device of the text detection network (hereinafter referred to as the second device) converts the image matrix corresponding to the sample image into a one-dimensional vector representing the sample image based on the encoding sub-network; the sample image The corresponding feature of the one-dimensional vector is the sequence sample feature of the sample image.

Step S202, using the sequence sample feature and the instance sample feature corresponding to the text instance sample as the input of the cross-layer attention layer of the decoding sub-network, and determining the output of the cross-layer attention layer as the decoded sample sequence vector.

In some embodiments, before the second device determines the decoded sample sequence vector, the text instance samples may also be input into a self-attention layer of the decoding sub-network, based on the self-attention layer of the self-attention layer. Output and obtain the embedded value corresponding to the text instance sample, where the embedded value is the instance sample feature corresponding to the text instance sample; wherein, the text instance sample includes character strings and/or phrases, and different text instance samples correspond to The embedded values are different; optionally, the instance sample feature may be a one-dimensional vector.

In some embodiments, the second device inputs the sequence sample feature and the instance sample feature to the cross-layer attention layer; and determines that the output of the cross-layer attention layer is the decoded sample sequence vector.

Step S203: The decoded sample sequence vector is used as the input of the output sub-network, and the prediction type of the sample image is determined according to the output of the output sub-network.

In some embodiments, the second device uses the decoded sample sequence vector as an input of a fully connected layer included in the output sub-network, and determines the prediction type of the sample image based on the output of the fully connected layer .

The prediction type may include: the sample image includes text, or the sample image does not include text.

Step S204, in response to the prediction type of the sample image being that the sample image includes text, use the decoded sample sequence vector and the sample sequence feature as the input of the output sub-network, according to the output sub-network output, determine the predicted position information of the text in the sample image.

In some embodiments, in response to the prediction type of the sample image being that the sample image includes text, the second device multiplies the decoded sample sequence vector by a vector corresponding to the sample sequence feature to obtain product result; based on the product result, determine the predicted position information of the text in the sample sequence feature; based on the predicted position information of the text in the sample sequence feature, determine the text in the sample image predicted location information.

In some optional embodiments, the decoded sample sequence vector and the vector corresponding to the sample sequence feature may perform a Tensor multiplication operation; when performing a Tensor multiplication operation, the decoding may be performed by means of 0-filling, copying, etc. The length of the latter sample sequence vector is the same as that of the vector corresponding to the sample sequence feature.

In specific implementation, after the Tensor multiplication operation is performed between the vector corresponding to the sample sequence feature and the decoded sample sequence vector, in the result (represented by a vector), the text is not compared with the parameter corresponding to the result. The text corresponds to whether the corresponding parameter in the result is large or small, and the predicted position information of the text in the sample sequence feature can be determined by the result of multiplication; since the sample sequence feature is the sample image after transformation The obtained one-dimensional vector; by means of matrix transformation, the predicted position information of the text in the sample image can be determined based on the predicted position information of the text in the sample sequence feature.

Step S205: Match the prediction type of the sample image with the annotation type of the sample image, and the predicted position information and annotation position information of the text in the sample image, and adjust the parameters of the text detection network based on the matching result.

In some embodiments, if the prediction type and the annotation type are the same, and the loss value between the predicted location information and the annotation location information is less than a preset threshold, the second device determines not to adjust the Describe the parameters of the text detection network.

In other embodiments, if the prediction type and the labeling type are different, or the loss value between the predicted location information and the labeling location information is greater than or equal to the preset threshold, based on the The difference between the prediction type of the sample image and the annotation type of the sample image, and/or the difference between the predicted position information and the annotation position information of the text in the sample image, adjust the parameters of the text detection network.

In specific implementation, the prediction type and the labeling type, or the predicted location information and the labeling location information can be matched by a bipartite graph matching algorithm, and the type loss and location information loss (or mask loss) are calculated respectively. .

Wherein, the value may include at least one of a binary classification cross-entropy loss value and a dice loss value.

In this way, through the training method of the text detection network provided by the embodiments of the present disclosure, a neural network that can directly predict the position of text of any shape at the text instance level in the image can be obtained, which provides powerful detection tasks for text of various shapes in complex scenes. support.

FIG. 3 shows another optional schematic flowchart of the text detection method provided by the embodiment of the present application, which will be described according to each step; FIG. 4 is a schematic data diagram of the text detection method provided by the embodiment of the present application.

The text detection method provided by the embodiment of the present disclosure includes that a natural scene image (image to be detected) of any shape text first passes through the coding sub-network of the text detection network to extract the sequence features of the to-be-detected image, and then passes through the decoding sub-network and The text vectors corresponding to different learnable text instances (Object Queries) are used to pay attention to different text instance information in the image to be detected. Further, the text detection network outputs the position information of different instance-level text instances; based on the position information , a simple connected domain analysis can be performed to obtain the border of the text instance, which provides strong support for subsequent text recognition and improves the accuracy of text recognition.

The following is the text detection network and the process of text detection based on the completion of the training (such as through steps S201 to S205):

Step S301, input the image to be detected into the text detection network.

Step S302, the coding sub-network included in the text detection network determines the sequence feature of the image to be detected.

In some embodiments, the encoding sub-network (or encoding Encode module) included in the text detection network may be a network structure based on a convolutional neural network (CNN), a Transform (a machine learning model), or a mixture of CNN and Transform. , the purpose of the encoding sub-network is to extract the sequence features of the image to be detected. The sequence includes a vector obtained by arranging the images in rows (if the image to be detected can be represented as an h*w matrix, the sequence corresponding to the image to be detected is a one-dimensional vector of 1*hw).

In some embodiments, the encoding sub-network converts the image matrix corresponding to the image to be detected into a one-dimensional vector; and the feature corresponding to the one-dimensional vector is determined as the sequence feature of the image to be detected.

Step S303, the decoding sub-network included in the text detection network determines the decoded sequence vector.

In some embodiments, the decoding sub-network (or called decoding Decoder module) included in the text detection network is constructed based on self-attention and cross-layer attention mechanism, and the purpose is to decode the sequence features extracted by the encoding sub-network operate. Specifically, the decoding sub-network includes a self-attention layer and a cross-layer attention layer; the self-attention layer is used to convert at least one learnable text instance (Object Queries) into at least one text feature, wherein, The at least one learnable text instance includes strings and/or phrases of different lengths; the cross-layer attention layer is used to output a decoded sequence vector.

In some embodiments, the at least one text instance is input to a self-attention layer of the decoding sub-network, and an embedding value corresponding to the at least one text instance is obtained based on an output of the self-attention layer, the The embedded value is the instance feature corresponding to the text instance; the instance feature corresponding to the sequence sample and the text instance is used as the input of the cross-layer attention layer of the decoding sub-network, and the output of the cross-layer attention layer is used. Determined as the decoded sequence vector.

Step S304, based on the decoded sequence vector, determine the type of the image to be detected and/or the position information of the text in the image to be detected.

In some embodiments, the output sub-network included in the text detection network includes a fully connected layer; the fully connected layer is configured to determine the type of the image to be detected based on the decoded sequence vector; the text detection network It is also used to determine the position information of the text in the image to be detected.

During specific implementation, the output sub-network may multiply the decoded sequence vector and the sequence feature through a Tensor multiplication operation, and determine the position information of the text in the sequence feature based on the product result; The position information of the text in the sequence feature determines the position information of the text in the to-be-detected image.

As shown in Figure 4, after the image to be detected is input into the encoding sub-network, the sequence features are obtained; after the text instance is input into the decoding sub-network, its output and the sequence features are jointly used as the output of the decoding sub-network. The output sub-network is multiplied with the sequence feature to obtain the position information of the text in the image to be detected.

In this way, with the text detection method provided by the embodiments of the present application, a text detection method is proposed to solve the problem that the existing text detection methods cannot effectively handle the detection of characters with arbitrary shapes. The detection problem is transformed into the problem of mask classification (that is, instance segmentation, distinguishing different instances of different classes). The text detection method provided by the embodiment of the present application solves the problem of arbitrary shape and distinguishes different text entities simultaneously by predicting instance-level text instances, without complex Post-processing and manual rules. Specifically, the text detection method provided by the embodiment of the present application uses the coding sub-network included in the text detection network to perform feature extraction on the input image to be detected, obtains sequence features, and uses the decoding sub-network included in the text detection network to output the image to be detected. The position information of at least one text, and finally a simple connected domain analysis is performed on the corresponding position to obtain the bounding box of the text instance. Compared with the previous text detection methods based on regression and segmentation, the text detection method provided by the embodiment of the present application simplifies the complexity of arbitrary shape modeling by directly predicting the position information of instance-level text in the image to be detected, and does not require complicated manual work. Post-processing can effectively improve the detection effect of arbitrary text in complex natural scenes in a data-driven way.

FIG. 5 shows an optional structural schematic diagram of the text detection apparatus provided by the embodiment of the present application, which will be described according to each part.

In some embodiments, the text detection apparatus 500 includes an encoding unit 501 , a decoding unit 502 , an image type determination unit 503 and an output unit 504 .

an encoding unit 501, configured to determine the sequence feature of the image to be detected;

A decoding unit 502, configured to determine a decoded sequence vector based on the sequence feature and the instance feature corresponding to the text instance;

The image type determination unit 503 is configured to determine the type of the image to be detected based on the decoded sequence vector;

The output unit 504 is configured to determine, based on the decoded sequence vector and the vector corresponding to the sequence feature, that the text is in the to-be-detected image in response to the type of the to-be-detected image being that the to-be-detected image includes text. Detect location information in an image.

The encoding unit 501 is specifically configured to convert the image matrix corresponding to the image to be detected into a one-dimensional vector; and determine that the feature corresponding to the one-dimensional vector is the sequence feature of the image to be detected.

The decoding unit 502 is further configured to obtain an embedded value corresponding to the text instance, where the embedded value is an instance feature corresponding to the text instance; wherein the text instance includes a character string and/or a phrase, and different Text instances correspond to different embedded values.

The output unit 504 is specifically configured to multiply the decoded sequence vector and the vector corresponding to the sequence feature to obtain a product result; based on the product result, determine the position of the text in the sequence feature information; based on the position information of the text in the sequence feature, determine the position information of the text in the to-be-detected image.

In some optional embodiments, the text detection apparatus 500 may further include: a bounding box determination unit 505 .

The bounding box determining unit 505 is configured to, after determining the position information of the text in the image to be detected based on the decoded sequence vector and the vector corresponding to the sequence feature, based on the image to be detected Chinese The location information of the present determines a connected domain in the image to be detected; a text bounding box is determined based on the boundary of the connected domain; wherein, the text bounding box is used to identify the text in the image to be detected.

FIG. 6 is a schematic diagram showing an optional structure of the text detection network training apparatus provided by the embodiment of the present application, which will be described according to each part.

In some embodiments, the training apparatus 600 of the text detection network includes a first determination unit 601 , a second determination unit 602 , a third determination unit 603 , a response unit 604 and an adjustment unit 605 .

The first determining unit 601 is configured to determine the sequence sample features of the sample images in the training sample set based on the coding sub-network;

The second determining unit 602 uses the sequence sample feature and the instance sample feature corresponding to the text instance sample as the input of the cross-layer attention layer of the decoding sub-network, and determines the output of the cross-layer attention layer. is the decoded sample sequence vector;

The third determining unit 603 uses the decoded sample sequence vector as the input of the output sub-network, and determines the prediction type of the sample image according to the output of the output sub-network;

The response unit 604 is configured to, in response to the prediction type of the sample image being that the sample image includes text, use the decoded sample sequence vector and the sample sequence feature as the input of the output sub-network, according to The output of the output sub-network determines the predicted position information of the text in the sample image;

The adjustment unit 605 is configured to match the prediction type of the sample image and the annotation type of the sample image, as well as the predicted position information and annotation position information of the text in the sample image, and adjust the text detection network based on the matching result. parameter.

The first determining unit 601 is specifically configured to convert the image matrix corresponding to the sample image into a one-dimensional vector representing the sample image through an encoding sub-network; the feature corresponding to the one-dimensional vector of the sample image is the Sequence sample features for sample images.

The second determining unit 602 is further configured to input the text instance samples to the self-attention layer of the decoding sub-network, and obtain the embedded value corresponding to the text instance samples based on the output of the self-attention layer , the embedded value is an instance sample feature corresponding to the text instance sample; wherein, the text instance sample includes character strings and/or phrases, and different text instance samples correspond to different embedded values.

The response unit 604 is specifically configured to use the decoded sample sequence vector as the input of the fully connected layer included in the output sub-network, and determine the prediction type of the sample image based on the output of the fully connected layer.

The third determining unit 603 is specifically configured to multiply the decoded sample sequence vector and the vector corresponding to the sample sequence feature to obtain a product result; based on the product result, determine that the text is in the sample sequence. The predicted position information in the sequence feature; based on the predicted position information of the text in the sample sequence feature, the predicted position information of the text in the sample image is determined.

The adjustment unit 605 is specifically configured to determine not to adjust the text if the prediction type and the annotation type are the same, and the loss value between the predicted location information and the annotation location information is less than a preset threshold Detect the parameters of the network; if the prediction type and the annotation type are different, or the loss value between the predicted location information and the annotation location information is greater than or equal to the preset threshold, then the sample image The difference between the prediction type of the sample image and the annotation type of the sample image, and/or the difference between the predicted position information and the annotation position information of the text in the sample image, adjust the parameters of the text detection network.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

7 shows a schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 7 , the device 800 includes a computing unit 801 that can be executed according to a computer program stored in a read only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803 Various appropriate actions and handling. In the RAM 803, various programs and data necessary for the operation of the device 800 can also be stored. The computing unit 801 , the ROM 802 , and the RAM 803 are connected to each other through a bus 804 . An input/output (I/O) interface 805 is also connected to bus 804 .

Various components in the device 800 are connected to the I/O interface 805, including: an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc. ; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 809 allows the device 800 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

Computing unit 801 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 801 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as a text detection method or a training method of a text detection network. For example, in some embodiments, a text detection method or a training method of a text detection network may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 808 . In some embodiments, part or all of the computer program may be loaded and/or installed on device 800 via ROM 802 and/or communication unit 809 . When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the text detection method or the training method of the text detection network described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform a text detection method or a training method of a text detection network by any other suitable means (eg, by means of firmware).

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a distributed system server, or a server combined with blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in different orders. As long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, there is no limitation herein.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included within the protection scope of the present disclosure.

Claims (25)

1. A text detection method, comprising: Determine the sequence characteristics of the image to be detected; Determine the decoded sequence vector based on the sequence feature and the instance feature corresponding to the text instance; Determine the type of the image to be detected based on the decoded sequence vector; In response to the type of the image to be detected being that the image to be detected includes text, the location information of the text in the image to be detected is determined based on the decoded sequence vector and the vector corresponding to the sequence feature. 2. The method according to claim 1, wherein said determining the sequence features of the images to be detected comprises: Convert the image matrix corresponding to the image to be detected into a one-dimensional vector; It is determined that the feature corresponding to the one-dimensional vector is the sequence feature of the image to be detected. 3. The method according to claim 1, wherein, before said determining the decoded sequence vector, the method further comprises: obtaining an embedded value corresponding to the text instance, where the embedded value is an instance feature corresponding to the text instance; The text instances include character strings and/or phrases, and different text instances correspond to different embedded values. 4. The method according to claim 1, wherein the determining the position information of the text in the to-be-detected image based on the decoded sequence vector and the vector corresponding to the sequence feature comprises: Multiplying the decoded sequence vector with the vector corresponding to the sequence feature to obtain a product result; Based on the product result, determine the position information of the text in the sequence feature; Based on the position information of the text in the sequence feature, the position information of the text in the image to be detected is determined. 5. The method according to claim 1, wherein after the position information of the text in the to-be-detected image is determined based on the decoded sequence vector and the vector corresponding to the sequence feature, the method include: Determine a connected domain in the to-be-detected image based on the position information of the text in the to-be-detected image; determining a text bounding box based on the boundaries of the connected domain; Wherein, the text bounding box is used to identify the text in the image to be detected. 6. A training method for a text detection network, wherein the text detection network comprises an encoding sub-network, a decoding sub-network and an output sub-network; Determine the sequence sample features of the sample images in the training sample set based on the coding sub-network; Using the sequence sample feature and the instance sample feature corresponding to the text instance sample as the input of the cross-layer attention layer of the decoding sub-network, and determining the output of the cross-layer attention layer as the decoded sample sequence vector; The decoded sample sequence vector is used as the input of the output sub-network, and the prediction type of the sample image is determined according to the output of the output sub-network; In response to the prediction type of the sample image being that the sample image includes text, the decoded sample sequence vector and the sample sequence feature are used as the input of the output sub-network, and according to the output of the output sub-network, determining the predicted position information of the text in the sample image; The prediction type of the sample image and the annotation type of the sample image are matched, as well as the predicted position information and the annotation position information of the text in the sample image, and the parameters of the text detection network are adjusted based on the matching result. 7. The method according to claim 6, wherein the determining the sequence sample features of the sample images in the training sample set based on the coding sub-network comprises: The encoding sub-network converts the image matrix corresponding to the sample image into a one-dimensional vector representing the sample image; The feature corresponding to the one-dimensional vector of the sample image is the sequence sample feature of the sample image. 8. The method according to claim 6, wherein, before said determining the decoded sample sequence vector, the method further comprises: Inputting the text instance sample to the self-attention layer of the decoding sub-network, and obtaining an embedded value corresponding to the text instance sample based on the output of the self-attention layer, where the embedded value is the text instance sample Corresponding instance sample features; The text instance samples include character strings and/or phrases, and different text instance samples correspond to different embedding values. 9. The method according to claim 6, wherein the decoded sample sequence vector is used as the input of the output sub-network, and the prediction type of the sample image is determined according to the output of the output sub-network ,include: The decoded sample sequence vector is used as the input of the fully connected layer included in the output sub-network, and the prediction type of the sample image is determined based on the output of the fully connected layer. 10. The method according to claim 6, wherein the decoded sample sequence vector and the sample sequence feature are used as the input of the output sub-network, and according to the output of the output sub-network, the The predicted position information of the text in the sample image, including: Multiplying the decoded sample sequence vector with the vector corresponding to the sample sequence feature to obtain a product result; Based on the product result, determine the predicted position information of the text in the sample sequence feature; Based on the predicted position information of the text in the sample sequence feature, the predicted position information of the text in the sample image is determined. 11. The method according to claim 6, wherein the prediction type of the sample image and the annotation type of the sample image are matched, as well as the predicted position information and the annotation position information of the text in the sample image, and adjusted based on the matching result. The parameters of the text detection network include: If the prediction type and the annotation type are the same, and the loss value between the predicted location information and the annotation location information is less than a preset threshold, it is determined not to adjust the parameters of the text detection network; If the prediction type and the annotation type are different, or, the loss value between the predicted position information and the annotation position information is greater than or equal to the preset threshold, then based on the prediction type of the sample image and the The parameters of the text detection network are adjusted according to the difference between the annotation types of the sample images, and/or the difference between the predicted position information and the annotation position information of the text in the sample images. 12. A text detection device, comprising: an encoding unit, used to determine the sequence feature of the image to be detected; a decoding unit for determining a decoded sequence vector based on the sequence feature and the instance feature corresponding to the text instance; an image type determination unit, configured to determine the type of the image to be detected based on the decoded sequence vector; an output unit, configured to determine that the text is in the image to be detected based on the decoded sequence vector and the vector corresponding to the sequence feature in response to the type of the image to be detected being that the image to be detected includes text location information. 13. The apparatus according to claim 12, wherein the encoding unit is specifically used for: Convert the image matrix corresponding to the image to be detected into a one-dimensional vector; It is determined that the feature corresponding to the one-dimensional vector is the sequence feature of the image to be detected. 14. The apparatus of claim 12, wherein the decoding unit is further configured to: Before determining the decoded sequence vector, obtain an embedded value corresponding to the text instance, where the embedded value is an instance feature corresponding to the text instance; The text instances include character strings and/or phrases, and different text instances correspond to different embedded values. 15. The apparatus of claim 12, wherein the output unit is specifically configured to: Multiplying the decoded sequence vector with the vector corresponding to the sequence feature to obtain a product result; Based on the product result, determine the position information of the text in the sequence feature; Based on the position information of the text in the sequence feature, the position information of the text in the image to be detected is determined. 16. The apparatus of claim 12, wherein the apparatus further comprises: a bounding box determination unit, configured to determine the position information of the text in the image to be detected based on the decoded sequence vector and the vector corresponding to the sequence feature, based on the position of the text in the image to be detected information to determine a connected domain in the image to be detected; determine a text bounding box based on the boundary of the connected domain; Wherein, the text bounding box is used to identify the text in the image to be detected. 17. A training device for a text detection network, comprising: a first determining unit, configured to determine the sequence sample features of the sample images in the training sample set based on the coding sub-network; The second determining unit is configured to use the sequence sample feature and the instance sample feature corresponding to the text instance sample as the input of the cross-layer attention layer of the decoding sub-network, and determine the output of the cross-layer attention layer as The decoded sample sequence vector; a third determining unit, configured to use the decoded sample sequence vector as the input of the output sub-network, and determine the prediction type of the sample image according to the output of the output sub-network; a response unit, configured to use the decoded sample sequence vector and the sample sequence feature as the input of the output sub-network in response to the prediction type of the sample image being that the sample image includes text, and according to the output The output of the sub-network determines the predicted position information of the text in the sample image; An adjustment unit, configured to match the prediction type of the sample image and the annotation type of the sample image, as well as the predicted position information and annotation position information of the text in the sample image, and adjust the parameters of the text detection network based on the matching result. 18. The apparatus according to claim 17, wherein the first determining unit is specifically configured to: Convert the image matrix corresponding to the sample image into a one-dimensional vector representing the sample image through the coding sub-network; The feature corresponding to the one-dimensional vector of the sample image is the sequence sample feature of the sample image. 19. The apparatus of claim 17, wherein the second determining unit is further configured to: Inputting the text instance sample to the self-attention layer of the decoding sub-network, and obtaining an embedded value corresponding to the text instance sample based on the output of the self-attention layer, where the embedded value is the text instance sample Corresponding instance sample features; The text instance samples include character strings and/or phrases, and different text instance samples correspond to different embedding values. 20. The apparatus according to claim 17, wherein the response unit is specifically used for: The decoded sample sequence vector is used as the input of the fully connected layer included in the output sub-network, and the prediction type of the sample image is determined based on the output of the fully connected layer. 21. The apparatus according to claim 17, wherein the third determining unit is specifically configured to: Multiplying the decoded sample sequence vector with the vector corresponding to the sample sequence feature to obtain a product result; Based on the product result, determine the predicted position information of the text in the sample sequence feature; Based on the predicted position information of the text in the sample sequence feature, the predicted position information of the text in the sample image is determined. 22. The device according to claim 17, wherein the adjustment unit is specifically used for: If the prediction type and the annotation type are the same, and the loss value between the predicted location information and the annotation location information is less than a preset threshold, it is determined not to adjust the parameters of the text detection network; If the prediction type and the annotation type are different, or the loss value between the predicted location information and the annotation location information is greater than or equal to the preset threshold, the The parameters of the text detection network are adjusted according to the difference between the annotation types of the sample images, and/or the difference between the predicted position information and the annotation position information of the text in the sample images. 23. An electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any of claims 1-5 Methods; Alternatively, to enable the at least one processor to perform the method of any of claims 6-11. 24. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-5; Alternatively, the computer instructions are for causing the computer to perform the method of any of claims 6-11. 25. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-5; Alternatively, the computer program, when executed by a processor, implements the method according to any of claims 6-11.

Images