TW202219822A - Character detection method, electronic equipment and computer-readable storage medium - Google Patents

Abstract

The present disclosure relates to a character detection method, electronic equipment and computer-readable storage medium. The method includes: respectively predicting multiple boundary lines of a first character sequence in an image to be processed to obtain prediction parameters of multiple boundary lines of the first character sequence, wherein the boundary line of the first character sequence represents the dividing line between the area where the first character sequence is located and the area where the first character sequence is not located; determine the position information of the vertices of the bounding box of the first character sequence according to the prediction parameters of the multiple boundary lines of the first character sequence; determine the position information of the bounding box of the first character sequence according to the position information of the vertices of the bounding box of the first character sequence. The embodiments of the present disclosure can improve the accuracy of character detection.

Description

Character detection method, electronic device and computer-readable storage medium

The present invention relates to the technical field of computer vision, and in particular, to a character detection method, an electronic device and a computer-readable storage medium.

Character detection in natural scenes is an important research field in computer vision, and has been applied to various application scenarios, such as real-time text translation, document recognition, license plate recognition, etc. In related technologies, characters are rigid in practical application scenarios. However, during the imaging process, due to the distortion and distortion of the camera's viewing angle, the characters on the rigid plane appear as irregular arbitrary quadrilateral shapes. The correct character shape is corrected in the character recognition link of , so as to correctly identify the character content.

The present invention provides a technical solution for character detection.

An embodiment of the present invention provides a character detection method, including: Predicting the multiple boundary lines of the first character sequence in the image to be processed respectively, to obtain the prediction parameters of the multiple boundary lines of the first character sequence, wherein the boundary line of the first character sequence represents the the boundary between the area where the first character sequence is located and the area not where the first character sequence is located; determining the position information of the vertices of the bounding box of the first character sequence according to the prediction parameters of the plurality of boundary lines of the first character sequence; The position information of the bounding box of the first character sequence is determined according to the position information of the vertices of the bounding box of the first character sequence. In this way, by predicting the prediction parameters of multiple boundary lines of the first character sequence in the image to be processed, the position information of the vertices of the bounding box of the first character sequence is determined, and according to the bounding box of the first character sequence The position information of the vertices of the character sequence is determined, and the position information of the bounding box of the first character sequence is determined, so that the polygon (such as quadrilateral) bounding box of the character sequence is decomposed into multiple (such as four) independent boundary lines. Independent boundary lines are independently detected, so that the detection of each boundary line will not be disturbed by two different vertices, thereby improving the accuracy of character detection.

In some embodiments of the present invention, the multiple boundary lines of the first character sequence in the image to be processed are predicted respectively, and the prediction parameters of the multiple boundary lines of the first character sequence are obtained, including: Based on the to-be-processed image, for the first feature point related to the first character sequence, predict the parameters corresponding to the first feature point of a plurality of boundary lines of the first character sequence respectively; The prediction parameters of the plurality of boundary lines of the first character sequence are determined according to the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature points. In this way, based on the image to be processed, for the first feature points related to the first character sequence, the parameters of the multiple boundary lines of the first character sequence corresponding to the first feature points are respectively predicted, and according to the first character sequence The plurality of boundary lines correspond to the parameters of the first feature point, and the prediction parameters of the plurality of boundary lines of the first character sequence are determined, so that the boundary of the first character sequence is based on the feature points related to the first character sequence. The parameters of the line are predicted, thereby helping to improve the efficiency of obtaining the predicted parameters of the boundary line, and helping to improve the accuracy of the obtained predicted parameters.

In some embodiments of the present invention, the method further includes: Predict the probability that the position of the primitive in the image to be processed belongs to the character; The first feature point is determined according to the probability that the position of the graphic element in the image to be processed belongs to the character element. In this way, by predicting the probability that the position of the primitive in the image to be processed belongs to the character, and according to the probability that the position of the primitive in the image to be processed belongs to the character, the first feature point can be determined, thereby accurately determining A first feature point associated with the first sequence of characters. Predicting the parameters of the boundary line of the first character sequence based on the first feature point thus determined is helpful to further improve the efficiency of obtaining the prediction parameters of the boundary line and further improve the accuracy of the obtained prediction parameters sex.

In some embodiments of the present invention, the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature point include: The distance parameters and angle parameters of the plurality of boundary lines of the first character sequence in the polar coordinate system corresponding to the first feature point, wherein the polar coordinate system corresponding to the first feature point is represented by the first feature point. A polar coordinate system with a point as a pole. In this way, by mapping the straight line equation of the boundary line in the Cartesian coordinate system to the polar coordinate system, the distance parameter and angle parameter with clear physical meaning and independent of each other in the image are obtained, which reduces the parameter quantity and correlation, and Useful for online learning.

In some embodiments of the present invention, the prediction of the plurality of boundary lines of the first character sequence is determined according to the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature point parameters, including: The distance parameters and angle parameters of the plurality of boundary lines of the first character sequence under the polar coordinate system corresponding to the first feature point are mapped to the Cartesian coordinate system to obtain the plurality of boundaries of the first character sequence The line corresponds to the parameter of the first feature point in the Cartesian coordinate system; The prediction parameters of the plurality of boundary lines of the first character sequence are determined according to the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature point in the Cartesian coordinate system. In this way, by mapping the distance parameters and angle parameters of the multiple boundary lines of the first character sequence in the polar coordinate system corresponding to the first feature point to the Cartesian coordinate system, the multiple boundary lines of the first character sequence are obtained in the Cartesian coordinate system. The parameters corresponding to the first feature point in the Cartesian coordinate system, and according to the parameters corresponding to the first feature point in the Cartesian coordinate system of the multiple boundary lines of the first character sequence, multiple boundaries of the first character sequence are determined The predicted parameters of the boundary line can be obtained by regression based on the parameters in different polar coordinate systems.

In some embodiments of the present invention, the plurality of boundary lines of the first sequence of characters include an upper boundary, a right boundary, a lower boundary and a left boundary of the first sequence of characters. In this way, since in most cases, the shape of the character sequence is a quadrilateral, according to this implementation, it is helpful to obtain more accurate position information of the bounding box of the character sequence in most cases.

In some embodiments of the present invention, based on the to-be-processed image, for the first feature points related to the first character sequence, it is respectively predicted that a plurality of boundary lines of the first character sequence corresponds to the Describe the parameters of the first feature point, including: Input the image to be processed into a pre-trained neural network, and through the neural network, for the first feature points related to the first character sequence, respectively predict the correspondence of multiple boundary lines of the first character sequence parameters at the first feature point. In this way, for the first feature points related to the first character sequence, the pre-trained neural network respectively predicts the parameters corresponding to the first feature points of the multiple boundary lines of the first character sequence, thereby improving the prediction parameters. speed and improve the accuracy of the predicted parameters.

In some embodiments of the present invention, the method further includes: The probability that the location of the primitive in the to-be-processed image belongs to the character is predicted through the neural network. In this way, by predicting the probability that the position of the primitive in the image to be processed belongs to the character through the pre-trained neural network, the speed of predicting the probability that the position of the primitive belongs to the character can be improved, and the predicted probability can be improved. accuracy.

In some embodiments of the present invention, before inputting the to-be-processed image into a pre-trained neural network, the method further includes: Input the training image into the neural network, and through the neural network, for the second feature points related to the second character sequence in the training image, respectively predict multiple items of the second character sequence. The boundary line corresponds to the predicted value of the parameter of the second feature point; According to the plurality of boundary lines of the second character sequence corresponding to the predicted value of the parameter of the second feature point, and the plurality of boundary lines of the second character sequence corresponding to the parameter of the second feature point The true value of , to train the neural network. In this way, by disassembling the polygon (such as quadrilateral) bounding box of the character sequence into multiple (such as four) independent boundary lines, each independent boundary line is detected separately, so that the neural network will not be affected due to regression vertices. The network brings training disturbance, thereby improving the learning efficiency and detection effect of the neural network, and the neural network trained according to this implementation can learn the ability to accurately predict the parameters of the boundary line of the character sequence.

In some embodiments of the present invention, the parameters of the plurality of boundary lines of the second character sequence corresponding to the second feature point include: the plurality of boundary lines of the second character sequence are in the second The distance parameter and the angle parameter under the polar coordinate system corresponding to the feature point, wherein, the polar coordinate system corresponding to the second feature point represents a polar coordinate system with the second feature point as a pole; The plurality of boundary lines according to the second character sequence corresponds to the predicted value of the parameter of the second feature point, and the plurality of boundary lines of the second character sequence corresponds to the second feature point The ground-truth values of the parameters that train the neural network, including: The plurality of boundary lines of the second character sequence correspond to the predicted value of the distance parameter of the second feature point, and the plurality of boundary lines of the second character sequence correspond to the predicted value of the distance parameter of the second feature point. the true value of the distance parameter to train the neural network; and/or, According to the plurality of boundary lines of the second character sequence corresponding to the predicted value of the angle parameter of the second feature point, and the plurality of boundary lines of the second character sequence corresponding to the second feature point The true value of the angle parameter to train the neural network. In this way, by mapping the straight line equation under the Cartesian coordinate system to the polar coordinate system, the amount of parameters and the correlation are reduced, and the actual physical meaning of the parameters is given, which is conducive to network learning, and learning to detect characters by training a neural network. Each boundary line of the sequence corresponds to the distance and angle of the feature points, so that the detection of the boundary lines does not interfere with each other, thereby improving the learning efficiency and detection effect of the neural network.

In some embodiments of the present invention, the plurality of boundary lines according to the second character sequence corresponds to the predicted value of the distance parameter of the second feature point, and the plurality of the second character sequence The boundary line corresponds to the true value of the distance parameter of the second feature point, and the training of the neural network includes: For any boundary line among the plurality of boundary lines of the second character sequence, according to the boundary line corresponding to the actual value and the predicted value of the distance parameter of the second feature point, the smaller value and the larger value The ratio of values to train the neural network. In this way, for any one of the multiple boundary lines of the second character sequence, according to the boundary line corresponding to the ratio of the smaller value to the larger value of the distance parameter of the second feature point and the predicted value , training the neural network, which can normalize the distance parameters of different sizes in different application scenarios, which can help to perform multi-scale character detection, that is, it is helpful to achieve different scales in character detection. higher accuracy.

混淆而對神經網路的學習帶來干擾，從而有助於提高神經網路的學習效率和檢測效果。 In some embodiments of the present invention, the plurality of boundary lines according to the second character sequence corresponds to the predicted value of the angle parameter of the second feature point, and the plurality of the second character sequence The boundary line corresponds to the true value of the angle parameter of the second feature point, and training the neural network includes: for any boundary line in the plurality of boundary lines of the second character sequence, determining the boundary The line corresponds to the absolute value of the difference between the true value of the angle parameter of the second feature point and the predicted value; the neural network is trained according to the sine value of the half angle of the absolute value. In this way, for any one of the multiple boundary lines of the second character sequence, the absolute value of the difference between the true value and the predicted value of the angle parameter of the boundary line corresponding to the second feature point is determined, and according to the absolute value The sine of half the angle of , trains the neural network so that it will not be

Confusion interferes with the learning of the neural network, thereby helping to improve the learning efficiency and detection effect of the neural network.

In some embodiments of the present invention, the second feature points include feature points in an effective area corresponding to the second character area. In this way, when calculating the loss function of the neural network, by only supervising the feature points in the effective area corresponding to the second character area, and not supervising the feature points outside the effective area corresponding to the second character area, it is helpful to reduce the network cost. road burden.

In some embodiments of the present invention, the method further includes: Predicting the probability that the position of the primitive in the training image belongs to the character through the neural network; The neural network is trained according to the probability that the position of the graphic element in the training image belongs to the character element, and the labeling data that the position of the graphic element in the training image belongs to the character element. In this way, the neural network can learn the ability to predict the probability that the location of the primitive belongs to the character.

In some embodiments of the present invention, according to the probability that the position of the primitive in the training image belongs to the character, and the labeling data of the position of the primitive in the training image belonging to the character, the training The neural network described above, including: The neural network is trained according to the probability that the position of the graphic element in the effective area corresponding to the second character sequence belongs to the character element, and the labeling data of the position of the graphic element in the effective area belonging to the character element. In this way, by training the neural network according to the probability that the position of the graphic element in the effective area corresponding to the second character sequence belongs to the character, and the labeling data of the position of the graphic element in the effective area belonging to the character, the neural network can be trained. It can learn the ability of character segmentation, and can improve the efficiency of neural network learning character segmentation.

In some embodiments of the present invention, the method further includes: obtaining location information of the real bounding box of the second character sequence; According to the position information of the real bounding box and a preset ratio, the real bounding box is reduced to obtain an effective area corresponding to the second character sequence. In this way, the effective area corresponding to the second character sequence is obtained, and the neural network is trained based on the feature points in the effective area corresponding to the second character sequence, which helps to reduce the network load.

In some embodiments of the present invention, according to the position information of the real bounding box and a preset ratio, reducing the real bounding box to obtain an effective area corresponding to the second character sequence, including: According to the position information of the real bounding box, the anchor point of the real bounding box is determined, wherein the anchor point of the real bounding box is the intersection of the diagonal lines of the real bounding box; According to the position information of the real bounding box, the position information of the anchor point of the real bounding box, and the preset ratio, the real bounding box is reduced to obtain the effective area corresponding to the second character sequence, wherein the first The ratio of a distance to a second distance is equal to the preset ratio, the first distance represents the distance between the first vertex of the effective area and the anchor point, and the second distance represents the distance in the real bounding box The distance between the vertex corresponding to the first vertex and the anchor point, where the first vertex represents any vertex of the effective area. In this way, the effective area corresponding to the second character sequence is obtained, and the neural network is trained based on the feature points in the effective area corresponding to the second character sequence, which helps to improve the learning efficiency and prediction accuracy of the neural network.

For descriptions of the effects of the following apparatuses, electronic devices, etc., reference may be made to the descriptions of the above-mentioned methods, which will not be repeated here.

The embodiment of the present invention also provides a character detection device, including: The first prediction module is configured to respectively predict multiple boundary lines of the first character sequence in the image to be processed, and obtain prediction parameters of the multiple boundary lines of the first character sequence, wherein the first The boundary line of the character sequence represents the dividing line between the area where the first character sequence is located and the area not where the first character sequence is located; a first determining module configured to determine position information of vertices of the bounding box of the first character sequence according to prediction parameters of a plurality of boundary lines of the first character sequence; The second determining module is configured to determine the position information of the bounding box of the first character sequence according to the position information of the vertices of the bounding box of the first character sequence.

In some embodiments of the present invention, the first prediction module is configured to predict, based on the to-be-processed image, the first feature points of the first character sequence, respectively, for the first character sequence. A plurality of boundary lines correspond to parameters of the first feature point; The prediction parameters of the plurality of boundary lines of the first character sequence are determined according to the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature points.

In some embodiments of the present invention, the apparatus further comprises: The second prediction module is configured to predict the probability that the position of the primitive in the to-be-processed image belongs to the character; The third determination module is configured to determine the first feature point according to the probability that the position of the graphic element in the image to be processed belongs to the character element.

In some embodiments of the present invention, the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature point include: The distance parameters and angle parameters of the plurality of boundary lines of the first character sequence in the polar coordinate system corresponding to the first feature point, wherein the polar coordinate system corresponding to the first feature point is represented by the first feature point. A polar coordinate system with a point as a pole.

In some embodiments of the present invention, the first prediction module is configured to map distance parameters and angle parameters of a plurality of boundary lines of the first character sequence in a polar coordinate system corresponding to the first feature point To the Cartesian coordinate system, obtain a plurality of boundary lines of the first character sequence corresponding to the parameters of the first feature point under the Cartesian coordinate system; The prediction parameters of the plurality of boundary lines of the first character sequence are determined according to the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature point in the Cartesian coordinate system.

In some embodiments of the present invention, the plurality of boundary lines of the first sequence of characters include an upper boundary, a right boundary, a lower boundary and a left boundary of the first sequence of characters.

In some embodiments of the present invention, the first prediction module is configured to input the to-be-processed image into a pre-trained neural network, via the neural network, for the first prediction associated with the first sequence of characters. feature points, respectively predicting that multiple boundary lines of the first character sequence correspond to parameters of the first feature points.

In some embodiments of the present invention, the apparatus further comprises: The third prediction module is configured to predict the probability that the location of the primitive in the image to be processed belongs to the character through the neural network.

In some embodiments of the present invention, the apparatus further comprises: The fourth prediction module is configured to input the training image into the neural network, and through the neural network, respectively predict the second feature points related to the second character sequence in the training image. A plurality of boundary lines of the second character sequence corresponds to the predicted value of the parameter of the second feature point; The first training module is configured to correspond to the predicted value of the parameter of the second feature point according to the multiple boundary lines of the second character sequence, and the multiple boundary lines of the second character sequence correspond to The true value of the parameter of the second feature point is used to train the neural network.

In some embodiments of the present invention, the parameters of the plurality of boundary lines of the second character sequence corresponding to the second feature point include: the plurality of boundary lines of the second character sequence are in the second The distance parameter and the angle parameter under the polar coordinate system corresponding to the feature point, wherein, the polar coordinate system corresponding to the second feature point represents a polar coordinate system with the second feature point as a pole; The first training module is configured to correspond to the predicted value of the distance parameter of the second feature point according to the plurality of boundary lines of the second character sequence, and the plurality of boundary lines of the second character sequence training the neural network corresponding to the true value of the distance parameter of the second feature point; and/or, According to the plurality of boundary lines of the second character sequence corresponding to the predicted value of the angle parameter of the second feature point, and the plurality of boundary lines of the second character sequence corresponding to the second feature point The true value of the angle parameter to train the neural network.

In some embodiments of the present invention, the first training module is configured to, for any one of a plurality of boundary lines of the second character sequence, correspond to the second feature according to the boundary line The ratio of the smaller value to the larger value of the true value and the predicted value of the distance parameter of the point, the neural network is trained.

In some embodiments of the present invention, the first training module is configured to, for any one of a plurality of boundary lines of the second character sequence, determine that the boundary line corresponds to the second feature The absolute value of the difference between the true value of the point's angle parameter and the predicted value; The neural network is trained according to the sine of the half angle of the absolute value.

In some embodiments of the present invention, the second feature points include feature points in an effective area corresponding to the second character area.

In some embodiments of the present invention, the apparatus further comprises: a fifth prediction module, configured to predict the probability that the position of the primitive in the training image belongs to the character through the neural network; The second training module is configured to train the neural network according to the probability that the position of the graphic element in the training image belongs to the character, and the labeling data that the position of the graphic element in the training image belongs to the character road.

In some embodiments of the present invention, the second training module is configured to be based on the probability that the location of the graphic element in the valid area corresponding to the second character sequence belongs to the character, and the graph in the valid area The location of the element belongs to the labeling data of the character, and the neural network is trained.

In some embodiments of the present invention, the apparatus further comprises: an acquisition module, configured to acquire the position information of the real bounding box of the second character sequence; The shrinking module is configured to shrink the real bounding box according to the position information of the real bounding box and a preset ratio to obtain an effective area corresponding to the second character sequence.

In some embodiments of the present invention, the reduction module is configured to determine the anchor point of the real bounding box according to the position information of the real bounding box, wherein the anchor point of the real bounding box is the real bounding box the intersection of the diagonals of the bounding box; According to the position information of the real bounding box, the position information of the anchor point of the real bounding box, and the preset ratio, the real bounding box is reduced to obtain the effective area corresponding to the second character sequence, wherein the first The ratio of a distance to a second distance is equal to the preset ratio, the first distance represents the distance between the first vertex of the effective area and the anchor point, and the second distance represents the distance in the real bounding box The distance between the vertex corresponding to the first vertex and the anchor point, where the first vertex represents any vertex of the effective area.

An embodiment of the present invention further provides an electronic device, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the memory stored in the memory The executable instruction is used to execute the character detection method described in any of the above embodiments.

An embodiment of the present invention further provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the character detection method described in any of the foregoing embodiments is implemented.

An embodiment of the present invention further provides a computer program, where the computer program includes computer-readable code, and when the computer-readable code is executed in an electronic device, the processor of the electronic device executes any of the above-mentioned embodiments. The character detection method described above.

In the embodiment of the present invention, the prediction parameters of the plurality of boundary lines of the first character sequence are obtained by respectively predicting the plurality of boundary lines of the first character sequence in the image to be processed. the prediction parameters of the boundary lines, determine the position information of the vertices of the bounding box of the first character sequence, and determine the position information of the bounding box of the first character sequence according to the position information of the vertices of the bounding box of the first character sequence , so that the polygon (such as quadrilateral) bounding box of the character sequence is decomposed into multiple (such as four) independent boundary lines, and each independent boundary line is detected separately, so that the detection of each boundary line will not is interfered by two different vertices, which in turn can improve the accuracy of character detection.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.

Other features and aspects of the present invention will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Various exemplary embodiments, features and aspects of the present invention will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote components that have the same or similar functions. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" in this article is only a relationship to describe related objects, which means that there can be three relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. three conditions. In addition, the term "at least one" herein refers to any combination of any one of a plurality or at least two of a plurality, for example, including at least one of A, B, and C, and may mean including those composed of A, B, and C. Any one or more elements selected in the collection.

In addition, in order to better illustrate the present invention, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present invention may be practiced without certain specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present invention.

In the related art, a rectangular frame or a rotated rectangular frame is mostly used to detect characters, but these rectangular frames or rotated rectangular frames cannot accurately locate the character boundary, which affects subsequent character recognition. In addition, the related art also proposes a character detection method in which a bounding box of a character is formed by regressing four vertices of a quadrilateral. However, a vertex is actually formed by the intersection of two adjacent edges, and the regression of each vertex affects both edges, so each edge is disturbed by two different vertices, which affects the accuracy of the character detection results.

Based on the above problems, embodiments of the present invention provide a character detection method, device, electronic device, storage medium, and program. By decomposing a polygon (eg, quadrilateral) bounding box of a character into multiple (eg, four) independent boundaries Each independent boundary line is independently detected, so that the detection of each boundary line will not be disturbed by two different vertices, so that the accuracy of character detection can be improved.

The character detection method provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

FIG. 1 shows a flowchart of a character detection method provided by an embodiment of the present invention. The execution subject of the character detection method may be a character detection device. In some embodiments of the present invention, the character detection method may be performed by a terminal device or a server or other processing device. The terminal device may be User Equipment (UE), mobile device, user terminal, terminal, cellular phone, wireless phone, Personal Digital Assistant (PDA), handheld device, computing device, vehicle-mounted device Or wearable devices, etc. In some embodiments of the present invention, the character detection method may be implemented by the processor calling computer-readable instructions stored in the memory. As shown in FIG. 1 , the character detection method includes steps S11 to S13.

In step S11 , the multiple boundary lines of the first character sequence in the image to be processed are respectively predicted, and the prediction parameters of the multiple boundary lines of the first character sequence are obtained.

Wherein, the boundary line of the first character sequence represents the boundary between the area where the first character sequence is located and the area not where the first character sequence is located.

In this embodiment of the present invention, the character detection may indicate the position of detecting the character and/or the character sequence in the image, for example, may indicate the position of the bounding box of detecting the character and/or character sequence in the image . In this embodiment of the present invention, the to-be-processed image may represent an image that needs to be subjected to character detection. The first sequence of characters represents any sequence of characters in the image to be processed. The image to be processed may include one or more sequences of characters. The first sequence of characters may include one or more characters, and the characters may include at least one of words, letters, digits, punctuation, operator symbols, and the like. In some embodiments of the present invention, in the image to be processed, if the distance between any two characters is less than or equal to a preset first distance threshold, it is determined that the two characters belong to the same character sequence. In other embodiments of the present invention, when the writing direction in the image to be processed is the horizontal direction, if any two characters belong to the same line of text, and the distance between the two characters is less than or equal to The preset second distance threshold, it is determined that the two characters belong to the same character sequence; in the case where the writing direction in the image to be processed is the vertical direction, if any two characters belong to the same column of text, and If the distance between the two characters is less than or equal to the preset third distance threshold, it is determined that the two characters belong to the same character sequence. The writing direction may represent the positional relationship between two adjacent characters. For example, if the positional relationship between two adjacent characters is the left-right relationship, the writing direction is the horizontal direction; if the positional relationship between the two adjacent characters is the up-down relationship, the writing direction is the vertical direction .

In the embodiment of the present invention, the boundary line of the first character sequence represents the boundary between the area where the first character sequence is located and the area where the non-first character sequence is located, wherein the area where the non-first character sequence is located may be The background area (that is, the area where the non-characters are located) and/or the area where other sequences of characters are located. The boundary line of the first character sequence may be a straight line or a curved line, which is not limited herein. The prediction parameter of any one of the boundary lines of the first sequence of characters may represent the parameters of the predicted boundary line. When the boundary line of the first character sequence is a straight line, the prediction parameter of any boundary line of the first character sequence may represent the prediction parameter of the line equation corresponding to the boundary line. Based on the prediction parameters of the line equation corresponding to the boundary line, the position of the boundary line can be determined.

In this embodiment of the present invention, when the boundary line of the first character sequence is a straight line, the number of the boundary lines of the first character sequence is at least three, and multiple boundary lines of the first character sequence may enclose The bounding box of the first character sequence. The bounding box of the first character sequence may be a polygon, and accordingly, the number of boundary lines of the first character sequence may correspond to the number of sides of the bounding box of the first character sequence. For example, if the bounding box of the first character sequence is a quadrilateral, the number of boundary lines of the first character sequence is four. Of course, the bounding box of the first character sequence may also be a pentagon, a triangle, etc., which is not limited herein.

In some embodiments of the present invention, the plurality of boundary lines of the first sequence of characters includes an upper boundary, a right boundary, a lower boundary and a left boundary of the first sequence of characters. In this embodiment, the bounding box of the first character sequence is a quadrilateral, and the number of boundary lines of the first character sequence is four. Wherein, the upper boundary line of the first character sequence may indicate that the direction of the characters in the first character sequence is used as a reference to divide the area where the first character sequence is located and the non-first character sequence above the first character sequence. The dividing line of the area where the character sequence is located; the right boundary line of the first character sequence can indicate that the direction of the characters in the first character sequence is used as a reference to divide the area where the first character sequence is located and the first character. The boundary of the area on the right side of the sequence that is not the first character sequence; the lower boundary line of the first character sequence can be used to divide the first character sequence with the direction of the characters in the first character sequence as a reference The dividing line between the area where the first character sequence is located and the area where the non-first character sequence is located below the first character sequence; the left boundary line of the first character sequence can indicate that the direction of the characters in the first character sequence is used as a reference. A boundary line that divides the area where the first character sequence is located and the area on the left of the first character sequence that is not where the first character sequence is located. Since in most cases, the shape of the character sequence is a quadrilateral, according to this embodiment, it is helpful to obtain more accurate position information of the bounding box of the character sequence in most cases.

In this embodiment, predicting multiple boundary lines of the first character sequence in the image to be processed respectively, to obtain prediction parameters of multiple boundary lines of the first character sequence, may include: the first character sequence in the image to be processed. The upper boundary line of the character sequence is predicted to obtain the prediction parameters of the straight line equation corresponding to the upper boundary line of the first character sequence; the right boundary line of the first character sequence in the image to be processed is predicted to obtain the first character sequence The prediction parameters of the straight line equation corresponding to the right boundary line of the image to be processed; predict the lower boundary line of the first character sequence in the image to be processed, and obtain the prediction parameters of the straight line equation corresponding to the lower boundary line of the first character sequence; the image to be processed The left boundary line of the first character sequence is predicted, and the prediction parameter of the straight line equation corresponding to the left boundary line of the first character sequence is obtained.

In step S12, the position information of the vertices of the bounding box of the first character sequence is determined according to the prediction parameters of the plurality of boundary lines of the first character sequence.

In the embodiment of the present invention, according to the prediction parameters of the multiple boundary lines of the first character sequence, the intersection of the multiple boundary lines of the first character sequence can be obtained, and the multiple boundary lines of the first character sequence can be The position information of the intersection point of , as the position information of the vertices of the bounding box of the first character sequence. For example, the multiple boundary lines of the first character sequence include an upper boundary line, a right boundary line, a lower boundary line and a left boundary line of the first character sequence; according to the prediction parameters of the straight line equation corresponding to the upper boundary line of the first character sequence The prediction parameters of the straight line equation corresponding to the right boundary line of the first character sequence can obtain the intersection of the upper boundary line of the first character sequence and the right boundary line of the first character sequence, and the upper boundary line of the first character sequence can be obtained. The position information of the intersection of the boundary line and the right boundary line of the first character sequence is used as the position information of the upper right corner vertex of the bounding box of the first character sequence; according to the prediction parameters of the straight line equation corresponding to the right boundary line of the first character sequence and The prediction parameters of the straight line equation corresponding to the lower boundary line of the first character sequence can be obtained by obtaining the intersection of the right boundary line of the first character sequence and the lower boundary line of the first character sequence, and the right side of the first character sequence can be The position information of the intersection of the boundary line and the lower boundary line of the first character sequence is used as the position information of the lower right corner vertex of the bounding box of the first character sequence; according to the prediction parameter of the straight line equation corresponding to the lower boundary line of the first character sequence The prediction parameters of the straight line equation corresponding to the left boundary line of the first character sequence can obtain the intersection of the lower boundary line of the first character sequence and the left boundary line of the first character sequence, and the lower boundary line of the first character sequence can be obtained. The position information of the intersection of the boundary line and the left boundary line of the first character sequence is used as the position information of the lower left corner vertex of the bounding box of the first character sequence; according to the prediction parameters of the line equation corresponding to the left boundary line of the first character sequence and The prediction parameters of the straight line equation corresponding to the upper boundary line of the first character sequence can obtain the intersection of the left boundary line of the first character sequence and the upper boundary line of the first character sequence, and the left boundary line of the first character sequence can be obtained. The position information of the intersection of the boundary line and the upper boundary line of the first character sequence is used as the position information of the upper left corner vertex of the bounding box of the first character sequence. In this embodiment of the present invention, the position information of the vertices of the bounding box of the first character sequence may be represented by the coordinates of the vertices of the bounding box of the first character sequence. For example, the location information of the vertices of the bounding box of the first character sequence may include the coordinates of the upper left vertex, the upper right vertex, the lower right vertex and the lower left vertex of the bounding box of the first character sequence.

In step S13, the position information of the bounding box of the first character sequence is determined according to the position information of the vertices of the bounding box of the first character sequence.

In this embodiment of the present invention, the position information of the vertices of the bounding box of the first character sequence may be used as the position information of the bounding box of the first character sequence. For example, the location information of the bounding box of the first sequence of characters may include coordinates of respective vertices of the bounding box of the first sequence of characters. Of course, when the bounding box of the first character sequence is a rectangle, the coordinates of any vertex of the bounding box of the first character sequence and the lengths of two sides connected to the vertex can also be used to represent the first character. The position information of the bounding box of the meta-sequence is not limited here.

FIG. 2 is a schematic diagram of a system architecture to which a character detection method according to an embodiment of the present invention can be applied; as shown in FIG. In order to support an exemplary application, the image acquisition terminal 201 and the location determination terminal 203 establish a communication connection through the network 202, and the image acquisition terminal 201 reports the image to be processed to the location determination terminal 203 through the network 202, and the location determination terminal 203 reports the image to be processed. 203 In response to the received image to be processed, the multiple boundary lines of the first character sequence in the image to be processed are respectively predicted to obtain prediction parameters of multiple boundary lines of the first character sequence, and based on the first The prediction parameters of multiple boundary lines of the character sequence determine the position information of the vertices of the bounding box of the first character sequence; according to the position information of the vertices of the bounding box of the first character sequence, the boundary of the first character sequence is determined The location information of the box. Finally, the location determination terminal 203 uploads the determined location information to the network 202 , and sends the determined location information to the image acquisition terminal 201 through the network 202 .

As an example, the image acquisition terminal 201 may include an image acquisition device, and the location determination terminal 203 may include a visual processing device or a remote server with visual information processing capability. The network 202 can be wired or wireless. Wherein, when it is determined that the location terminal 203 is a visual processing device, the image acquisition terminal 201 can be connected to the visual processing device through a wired connection, such as data communication through a bus; when it is determined that the location terminal 203 is a remote server , the image acquisition terminal 201 can exchange data with the remote server through the wireless network.

Alternatively, in some scenarios, the image acquisition terminal 201 may be a vision processing device with an image acquisition module, and is specifically implemented as a host with a camera. At this time, the character detection method according to the embodiment of the present invention may be executed by the image acquisition terminal 201 , and the above-mentioned system architecture may not include the network 202 and the location determination terminal 203 .

In the embodiment of the present invention, the prediction parameters of the plurality of boundary lines of the first character sequence are obtained by respectively predicting the plurality of boundary lines of the first character sequence in the image to be processed. the prediction parameters of the boundary lines, determine the position information of the vertices of the bounding box of the first character sequence, and determine the position information of the bounding box of the first character sequence according to the position information of the vertices of the bounding box of the first character sequence , so that the polygon (such as quadrilateral) bounding box of the character sequence is decomposed into multiple (such as four) independent boundary lines, and each independent boundary line is detected separately, so that the detection of each boundary line will not is interfered by two different vertices, which in turn can improve the accuracy of character detection.

In some embodiments of the present invention, the multiple boundary lines of the first character sequence in the image to be processed are respectively predicted to obtain the prediction parameters of the multiple boundary lines of the first character sequence, including: based on the image to be processed , for the first feature point relevant to the first character sequence, respectively predict that the multiple boundary lines of the first character sequence correspond to the parameters of the first feature point; A parameter of a feature point that determines prediction parameters of a plurality of boundary lines of the first character sequence. In this embodiment, the first feature points represent feature points associated with the first sequence of characters. Among them, the feature points can represent the points where the gray value of the image changes drastically and/or the points with large curvature on the edge of the image (that is, the intersection of two edges). The number of the first feature points may be multiple, of course, may also be one, which is not limited here. For example, when the number of first feature points is multiple and the number of boundary lines of the first character sequence is 4, for any first feature point, each boundary line of the first character sequence is predicted separately Corresponding to the parameters of the first feature point, and for any boundary line, the prediction parameters of the boundary line are determined according to the parameters of the boundary line corresponding to each of the first feature points. For example, the parameters of the boundary line corresponding to each of the first feature points can be regressed to obtain the predicted parameters of the boundary line. In this embodiment, based on the image to be processed, for the first feature points related to the first character sequence, the parameters corresponding to the first feature points of the plurality of boundary lines of the first character sequence are respectively predicted, and according to the first feature point The plurality of boundary lines of a sequence of characters correspond to parameters of the first feature point, and the prediction parameters of the plurality of boundary lines of the first sequence of characters are determined, whereby the first character The parameters of the boundary line of the meta-sequence are predicted, thereby helping to improve the efficiency of obtaining the prediction parameters of the boundary line, and helping to improve the accuracy of the obtained prediction parameters. Of course, in other embodiments of the present invention, it is also possible to determine the number of the first character sequence based on all the primitive points related to the first character sequence (not limited to the first feature points related to the first character sequence). The prediction parameters of the boundary lines are not limited here.

As an example of this embodiment, the method further includes: predicting the probability that the position of the graphic element in the image to be processed belongs to the character element; according to the probability that the position of the graphic element in the image to be processed belongs to the character element Feature points. In this example, the probability that the location of each primitive in the image to be processed belongs to a character can be predicted. According to the probability that the position of each graphic element in the image to be processed belongs to a character element, the area occupied by each sequence of characters in the image to be processed can be preliminarily determined. For any first character sequence, the first feature point may be determined according to the preliminarily determined feature points in the area occupied by the first character sequence. For example, all or part of the feature points in the area occupied by the initially determined first character sequence may be determined as the first feature points. In this example, by predicting the probability that the position of the primitive in the image to be processed belongs to the character, and determining the first feature point according to the probability that the position of the primitive in the image to be processed belongs to the character, it is possible to A first feature point associated with the first sequence of characters is accurately determined. Predicting the parameters of the boundary line of the first character sequence based on a feature point thus determined helps to further improve the efficiency and accuracy of obtaining the prediction parameters of the boundary line.

In the examples provided by other embodiments of the present invention, the feature points in the to-be-processed image can also be used as the first feature points, respectively, without predicting the probability of characters. For example, in the case that there is only one first character sequence in the image to be processed, and the first character sequence occupies or almost occupies the image to be processed, the feature points in the image to be processed can be respectively used as the first character sequence. a feature point.

As an example of this embodiment, the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature point include: distances of the plurality of boundary lines of the first character sequence in the polar coordinate system corresponding to the first feature point parameters and angle parameters, wherein the polar coordinate system corresponding to the first feature point represents a polar coordinate system with the first feature point as a pole. In this example, the polar coordinate system corresponding to the first feature point may use the axis whose pole points to the positive direction of the x-axis as the polar axis. Of course, those skilled in the art can flexibly set the polar axis according to actual application scenario requirements, which is not limited here. In this example, the distance parameter of any boundary line of the first character sequence in the polar coordinate system corresponding to the first feature point can represent the distance between the first feature point and the boundary line in the polar coordinate system corresponding to the first feature point. The minimum distance between, that is, it can represent the length of the vertical line segment from the first feature point to the boundary line in the polar coordinate system corresponding to the first feature point; any boundary line of the first character sequence is in the first feature point. The angle parameter in the polar coordinate system can represent the angle between the vector from the first feature point to the vertical point on the boundary line in the polar coordinate system corresponding to the first feature point and the polar axis of the polar coordinate system corresponding to the first feature point, The vertical point on the boundary line represents the intersection point of the vertical line segment from the first feature point to the boundary line and the boundary line.

公式（1）； 其中，

、

和

表示直線方程的參數。 In one example, the equation of a straight line in the Cartesian coordinate system (the Cartesian coordinate system or the oblique coordinate system) can be expressed by formula (1):

Formula (1); where,

,

and

The parameters representing the equation of the line.

時，公式1所示的直線方程中的參數及參數之間的相關性存在冗餘。另外，笛卡爾座標系下的直線方程的參數在圖像中沒有明確的物理意義，導致不利於網路學習。 However, when

When , the parameters in the equation of the straight line shown in Equation 1 and the correlations between the parameters are redundant. In addition, the parameters of the straight line equation in the Cartesian coordinate system have no clear physical meaning in the image, which is not conducive to network learning.

公式（2）； 其中，

可以表示第一字元序列的任意一條邊界線在第一特徵點對應的極座標系下的距離參數，

可以表示第一字元序列的任意一條邊界線在第一特徵點對應的極座標系下的角度參數。 In this example, the equation of the line in the Cartesian coordinate system can be converted to the polar coordinate system, and the formula (2) can be obtained:

Formula (2); where,

can represent the distance parameter of any boundary line of the first character sequence in the polar coordinate system corresponding to the first feature point,

It can represent the angle parameter of any boundary line of the first character sequence in the polar coordinate system corresponding to the first feature point.

公式（3）。 Correspondingly, the parameters of the straight line equation can be expressed by formula (3):

Formula (3).

，角度參數為

；第一字元序列的右邊界線在該第一特徵點對應的極座標系下的距離參數為

，角度參數為

；第一字元序列的下邊界線在該第一特徵點對應的極座標系下的距離參數為

，角度參數為

；第一字元序列的左邊界線在該第一特徵點對應的極座標系下的距離參數為

，角度參數為

。 FIG. 3 is a schematic diagram showing the distance parameters and angle parameters of the four boundary lines of the first character sequence in a polar coordinate system corresponding to a certain first feature point. As shown in Figure 3, the distance parameter of the upper boundary line of the first character sequence in the polar coordinate system corresponding to the first feature point is:

, the angle parameter is

; The distance parameter of the right boundary line of the first character sequence under the polar coordinate system corresponding to the first feature point is

, the angle parameter is

; the distance parameter of the lower boundary line of the first character sequence under the polar coordinate system corresponding to the first feature point is

, the angle parameter is

; The distance parameter of the left boundary line of the first character sequence under the polar coordinate system corresponding to the first feature point is

, the angle parameter is

.

In this example, by mapping the straight line equation of the boundary line in the Cartesian coordinate system to the polar coordinate system, the distance parameters and angle parameters that have clear physical meaning and are independent of each other in the image are obtained, reducing the amount of parameters and related Sexuality, and is conducive to online learning.

Wherein, according to the parameters of the multiple boundary lines of the first character sequence corresponding to the first feature point, determining the prediction parameters of the multiple boundary lines of the first character sequence, including: combining the multiple boundary lines of the first character sequence The distance parameter and the angle parameter under the polar coordinate system corresponding to the first feature point are mapped to the Cartesian coordinate system, and a plurality of boundary lines of the first character sequence are obtained under the Cartesian coordinate system corresponding to the parameters of the first feature point; according to The multiple boundary lines of the first character sequence correspond to the parameters of the first feature point in the Cartesian coordinate system, and the prediction parameters of the multiple boundary lines of the first character sequence are determined. In this example, when the number of the first feature points is multiple, the multiple first feature points correspond to different polar coordinate systems, wherein the polar coordinate system corresponding to any first feature point is based on the first feature point. to the extreme. Therefore, for any boundary line of the first character sequence, when regressing the distance parameter and angle parameter of the boundary line in the polar coordinate system corresponding to the plurality of first feature points to obtain the prediction parameter of the boundary line, the prediction parameters of the boundary line can be obtained first. The distance parameters and angle parameters of the boundary line in the polar coordinate system corresponding to the plurality of first feature points are mapped to the same Cartesian coordinate system, and the parameters of the boundary line corresponding to the plurality of feature points in the Cartesian coordinate system are obtained. , and then perform regression according to the parameters of the boundary line corresponding to a plurality of feature points in the Cartesian coordinate system to obtain the prediction parameters of the boundary line. Wherein, by mapping the distance parameters and angle parameters of the multiple boundary lines of the first character sequence in the polar coordinate system corresponding to the first feature point to the Cartesian coordinate system, the multiple boundary lines of the first character sequence are obtained in the Cartesian coordinate system. The parameters corresponding to the first feature point in the Cartesian coordinate system, and according to the parameters corresponding to the first feature point in the Cartesian coordinate system of the multiple boundary lines of the first character sequence, multiple boundaries of the first character sequence are determined The predicted parameters of the boundary line can be obtained by regression based on the parameters in different polar coordinate systems.

、

和

，即預測的第一字元序列的上邊界線的直線方程可以表示為

；第一字元序列的右邊界線的預測參數為

、

和

，即預測的第一字元序列的右邊界線的直線方程可以表示為

；第一字元序列的下邊界線的預測參數為

、

和

，即預測的第一字元序列的下邊界線的直線方程可以表示為

；第一字元序列的左邊界線的預測參數為

、

和

，即，所預測的第一字元序列的左邊界線的直線方程可以表示為

。即可根據公式（4）至（6），得到第一字元序列的邊界框的各個頂點的座標：

公式（4）；

公式（5）；

公式（6）； 其中，

，

，

和

均為整數。例如，

可以表示第一字元序列的邊界框的右上角頂點的座標，

可以表示第一字元序列的邊界框的右下角頂點的座標，

可以表示第一字元序列的邊界框的左下角頂點的座標，

可以表示第一字元序列的邊界框的左上角頂點的座標。 As shown in Fig. 3, the prediction parameter of the upper boundary line of the first character sequence is

,

and

, that is, the straight line equation of the upper boundary line of the predicted first character sequence can be expressed as

; the prediction parameter of the right boundary line of the first character sequence is

,

and

, that is, the straight line equation of the predicted right boundary line of the first character sequence can be expressed as

; the prediction parameter of the lower boundary line of the first character sequence is

,

and

, that is, the straight line equation of the lower boundary line of the predicted first character sequence can be expressed as

; the prediction parameter of the left boundary line of the first character sequence is

,

and

, that is, the straight line equation of the predicted left boundary line of the first character sequence can be expressed as

. Then, according to formulas (4) to (6), the coordinates of each vertex of the bounding box of the first character sequence can be obtained:

formula (4);

formula (5);

Formula (6); where,

,

,

and

All are integers. E.g,

may represent the coordinates of the upper-right vertex of the bounding box of the first sequence of characters,

may represent the coordinates of the lower-right vertex of the bounding box of the first sequence of characters,

may represent the coordinates of the lower-left corner vertex of the bounding box of the first sequence of characters,

The coordinates of the upper-left corner vertex of the bounding box that may represent the first sequence of characters.

In other examples, the parameters of any boundary line of the first character sequence corresponding to the first feature point may include parameters of the boundary line predicted based on the first feature point in a Cartesian coordinate system, which is not limited herein.

In an example, based on the image to be processed, for the first feature point related to the first character sequence, respectively predicting the parameters corresponding to the first feature point of multiple boundary lines of the first character sequence, including: The processing image is input to a pre-trained neural network, and the parameters corresponding to the first feature points of the plurality of boundary lines of the first character sequence are respectively predicted through the neural network for the first feature points related to the first character sequence. Wherein, for the first feature points related to the first character sequence, the pre-trained neural network respectively predicts the parameters of the multiple boundary lines of the first character sequence corresponding to the first feature points, so that the prediction parameters can be improved. speed and accuracy. At the same time, the parameters corresponding to the first feature point of the multiple boundary lines of the first character sequence can also be predicted by using a pre-established model, function, etc., which is not limited herein.

In the embodiment provided by the present invention, the probability that the location of the graphic element in the image to be processed belongs to the character element can also be predicted through the neural network. Among them, the pre-trained neural network is used to predict the probability that the position of the primitive in the image to be processed belongs to the character, so that the speed of predicting the probability that the position of the primitive belongs to the character can be improved, and the predicted probability can be improved. accuracy. Of course, in other examples, a pre-established model, function, etc. can also be used to predict the probability that the location of the graphic element in the image to be processed belongs to the character element, which is not limited here.

In some embodiments of the present invention, before the image to be processed is input into the pre-trained neural network, the training image can also be input into the neural network, and the second character sequence in the training image can be compared via the neural network. Relevant second feature points, respectively predict the predicted values of the parameters of the second feature point corresponding to the multiple boundary lines of the second character sequence; according to the multiple boundary lines of the second character sequence corresponding to the parameters of the second feature point The predicted value of , and the multiple boundary lines of the second sequence of characters correspond to the true values of the parameters of the second feature point to train the neural network.

In the related art, the bounding box of the character is formed by regressing the four vertices of the quadrilateral. Since the vertex is actually formed by the intersection of two adjacent edges, the regression of each vertex will affect the two edges, therefore, each edge will be disturbed by two different vertices, thus affecting the learning efficiency and detection effect of the network. In the embodiment provided by the present invention, the polygon (for example, quadrilateral) bounding box of the character sequence is decomposed into a plurality of (for example, four) independent boundary lines, and each independent boundary line is independently detected, so that no It will bring training disturbance to the neural network due to the regression vertex, thereby improving the learning efficiency and detection effect of the neural network. The neural network trained according to this embodiment can learn the ability to accurately predict the parameters of the boundary line of the character sequence.

As an example of this embodiment, the parameters of the plurality of boundary lines of the second character sequence corresponding to the second feature points include: distances of the plurality of boundary lines of the second character sequence in the polar coordinate system corresponding to the second feature points parameter and angle parameter, wherein, the polar coordinate system corresponding to the second feature point represents the polar coordinate system with the second feature point as the pole; according to the predicted value of the parameter corresponding to the second feature point according to multiple boundary lines of the second character sequence, And the multiple boundary lines of the second character sequence are corresponding to the true value of the parameter of the second feature point, and the training neural network includes: according to the multiple boundary lines of the second character sequence corresponding to the distance parameter of the second feature point The predicted value of , and the multiple boundary lines of the second character sequence correspond to the true value of the distance parameter of the second feature point, training the neural network; and/or, according to the multiple boundary lines of the second character sequence corresponding to The predicted value of the angle parameter of the second feature point, and the multiple boundary lines of the second character sequence corresponding to the true value of the angle parameter of the second feature point, train the neural network. In this example, by mapping the straight line equation in the Cartesian coordinate system to the polar coordinate system, the correlation between the learning parameters and the parameters is reduced, and the parameters are given the actual physical meaning in the image, which is conducive to network learning . In addition, in this example, by training the neural network to learn the distance and angle of each boundary line of the detected character sequence corresponding to the feature point, the detection of the boundary lines can not interfere with each other, so that the learning efficiency of the neural network can be improved. and detection effects.

In one example, according to the plurality of boundary lines of the second character sequence corresponding to the predicted value of the distance parameter of the second feature point, and the plurality of boundary lines of the second character sequence corresponding to the distance parameter of the second feature point The true value, training the neural network, including: for any one of the multiple boundary lines of the second character sequence, according to the boundary line corresponding to the distance parameter of the second feature point, the smaller of the true value and the predicted value The ratio of the value to the larger value to train the neural network.

可以採用公式（7）得到：

公式（7）； 其中，

表示第二特徵點的數量，

表示該邊界線對應於第二特徵點

的距離參數的真值，

表示該邊界線對應於第二特徵點

的距離參數的預測值，

表示該邊界線對應於第二特徵點

的距離參數的真值和預測值中的較小值，

表示該邊界線對應於第二特徵點

的距離參數的真值和預測值中的較大值。例如，若

，則

，

；若

，則

，

。由於

和

對應的極點相同（均為第二特徵點

），即

和

的一端處於相同點，因此，該距離參數對應的損失函數

可以稱為射線重疊度（Intersection Over Union，IOU）損失函數。 For example, for any one of the multiple boundary lines of the second character sequence, the loss function corresponding to the distance parameter

It can be obtained by formula (7):

Formula (7); where,

represents the number of second feature points,

Indicates that the boundary line corresponds to the second feature point

the true value of the distance parameter,

Indicates that the boundary line corresponds to the second feature point

The predicted value of the distance parameter,

Indicates that the boundary line corresponds to the second feature point

The smaller of the true and predicted values of the distance parameter,

Indicates that the boundary line corresponds to the second feature point

The larger of the true and predicted values of the distance parameter. For example, if

,but

,

;like

,but

,

. because

and

The corresponding poles are the same (both are the second feature points

),which is

and

One end of is at the same point, therefore, the loss function corresponding to the distance parameter

It can be called the Intersection Over Union (IOU) loss function.

In this example, for any one of the plurality of boundary lines of the second character sequence, according to the boundary line corresponding to the distance parameter of the second feature point, the smaller value and the larger value among the true value and the predicted value of the distance parameter The ratio of the value to train the neural network, which can normalize the distance parameters of different application scenarios and different sizes, which can help to perform multi-scale character detection, that is, it is helpful to detect characters at different scales. Achieve higher accuracy in character detection.

Of course, in other examples, for any boundary line among the plurality of boundary lines of the second character sequence, the neural network can also be trained according to the search for the true value and the predicted value of the distance parameter corresponding to the boundary line to the second feature point The road is not limited here.

In one example, according to the plurality of boundary lines of the second character sequence corresponding to the predicted value of the angle parameter of the second feature point, and the plurality of boundary lines of the second character sequence corresponding to the angle parameter of the second feature point The true value, training the neural network, comprising: for any one of the multiple boundary lines of the second character sequence, determining the difference between the true value and the predicted value of the angle parameter of the boundary line corresponding to the second feature point; Absolute value; trains the neural network based on the sine of the half angle of the absolute value.

Among them, the half angle of the absolute value is equal to 0.5 times the absolute value. For example, for any one of the multiple boundary lines of the second character sequence, the difference between the predicted value and the true value of the angle parameter of the boundary line corresponding to any second feature point is 90° or −90°, Then, the absolute value of the difference between the true value and the predicted value of the angle parameter of the boundary line corresponding to the second feature point is 90°, and the half angle of the absolute value is 45°.

可以採用公式（8）得到：

公式（8）； 其中，

表示第二特徵點的數量，

表示該邊界線對應於第二特徵點

的角度參數的真值，

表示該邊界線對應於第二特徵點

的角度參數的預測值，

表示該邊界線對應於第二特徵點

的角度參數的真值與預測值的差值的絕對值，

表示該邊界線對應於第二特徵點

的角度參數的真值與預測值的差值的絕對值的半倍角。 For example, for any one of the multiple boundary lines of the second character sequence, the loss function corresponding to the angle parameter

Equation (8) can be used to get:

Formula (8); where,

represents the number of second feature points,

Indicates that the boundary line corresponds to the second feature point

The true value of the angle parameter of ,

Indicates that the boundary line corresponds to the second feature point

The predicted value of the angle parameter of ,

Indicates that the boundary line corresponds to the second feature point

The absolute value of the difference between the true value of the angle parameter and the predicted value,

Indicates that the boundary line corresponds to the second feature point

Half the angle of the absolute value of the difference between the true value of the angle parameter and the predicted value.

，即

，

。然而在極座標系中，0與

重合。通過對於第二字元序列的多條邊界線中的任意一條邊界線，確定邊界線對應於第二特徵點的角度參數的真值與預測值的差值的絕對值，並根據絕對值的半倍角的正弦值，訓練神經網路，由此不會因為0與

混淆而對神經網路的學習帶來干擾，從而提高神經網路的學習效率和檢測效果。 Wherein, for any one of the multiple boundary lines of the second character sequence, the value range of the true value and the predicted value of the angle parameter of the boundary line corresponding to any second feature point may be

,Right now

,

. However, in the polar coordinate system, 0 and

coincide. For any one of the multiple boundary lines of the second character sequence, determine the absolute value of the difference between the true value and the predicted value of the angle parameter of the boundary line corresponding to the second feature point, and determine the absolute value of the difference according to the half of the absolute value. The sine value of the double angle, training the neural network, so that it will not be caused by 0 and

Confusion interferes with the learning of the neural network, thereby improving the learning efficiency and detection effect of the neural network.

Of course, those skilled in the art can also use a cosine loss function or the like after transforming Equation 8, which is not limited here.

As an example of this embodiment, the second feature points include feature points in an effective area corresponding to the second character area. Wherein, the second feature points may only include feature points in the effective area corresponding to the second character area, and do not include feature points outside the effective area corresponding to the second character area. When calculating the loss function of the neural network, by only supervising the feature points in the effective area corresponding to the second character area, and not supervising the feature points outside the effective area corresponding to the second character area, it helps to reduce the network burden . For any feature point in the real bounding box of the second character area or in the edge area close to the real bounding box, the distance between the feature point and the boundary line of the real bounding box is small, which is difficult to detect accurately and easily causes larger error. For example, for a feature point in the valid area, the predicted value of the distance parameter between the feature point and a certain boundary line of the real bounding box is 9, and the real value is 10, then the error is 10%; Feature point, the predicted value of the distance parameter between the feature point and a certain boundary line of the real bounding box is 1, and the real value is 2, then the error is 50%. Therefore, by ignoring feature points outside the effective area, it helps to reduce the network burden. Of course, in other examples, all the feature points in the real bounding box of the second character sequence are not limited here.

In one example, the character detection method provided by the embodiment of the present invention further includes: acquiring position information of the real bounding box of the second character sequence; reducing the real bounding box according to the position information of the real bounding box and a preset ratio, Obtain the valid area corresponding to the second character sequence. In this example, the range of the valid area corresponding to the second character sequence is within the real bounding box of the second character sequence, and the size of the valid area corresponding to the second character sequence is smaller than the real bounding box of the second character sequence size of. FIG. 4 shows a schematic diagram of the real bounding box 31 and the effective area 32 of the second character area. Based on this example, the effective area corresponding to the second character sequence is obtained, and the neural network is trained based on the feature points in the effective area corresponding to the second character sequence, which helps to reduce the network load.

For example, reducing the real bounding box according to the position information of the real bounding box and the preset ratio to obtain the valid area corresponding to the second character sequence, including: determining the anchor point of the real bounding box according to the position information of the real bounding box, wherein , the anchor point of the real bounding box is the intersection of the diagonal lines of the real bounding box; according to the position information of the real bounding box, the position information of the anchor point of the real bounding box, and the preset scale, reduce the real bounding box to get the second word The valid region corresponding to the meta-sequence, where the ratio of the first distance to the second distance is equal to the preset ratio, the first distance represents the distance between the first vertex of the valid region and the anchor point, and the second distance represents the first distance in the real bounding box. The distance between the vertex corresponding to a vertex and the anchor point, where the first vertex represents any vertex of the valid area. For example, the preset ratio may be 0.35, 0.4, 0.3, etc., which is not limited herein. For example, if the first vertex is the upper left vertex of the valid area, the vertex corresponding to the first vertex in the real bounding box is the upper left vertex of the real bounding box, and so on. According to this example, the effective area corresponding to the second character sequence is obtained, and the neural network is trained based on the feature points in the effective area corresponding to the second character sequence, which helps to improve the learning efficiency and prediction accuracy of the neural network. .

，

。其中，第二字元序列的真實邊界框的4個頂點可以按順時針方向排序，

可以表示第二字元序列的真實邊界框的左上角頂點，

可以表示第二字元序列的真實邊界框的右上角頂點，

可以表示第二字元序列的真實邊界框的右下角頂點，

可以表示第二字元序列的真實邊界框的左下角頂點。對於任一第二特徵點

，第二字元序列的真實邊界框的任意一條邊界線對應於該第二特徵點的距離參數的真值

和角度參數的真值

可以採用公式（9）至（16）確定：

公式（9）；

公式（10）；

公式（11）；

公式（12）；

公式（13）；

公式（14）；

公式（15）；

公式（16）； 其中，

表示第二特徵點到該邊界線的垂線向量的真值，

與第二特徵點到該邊界線的垂線平行，且由第二特徵點指向垂點，

表示

在極軸下方。 In one example, the coordinates of the 4 vertices of the true bounding box of the second sequence of characters can be expressed as

,

. Among them, the 4 vertices of the real bounding box of the second character sequence can be sorted clockwise,

can represent the top-left corner vertex of the true bounding box of the second sequence of characters,

can represent the upper-right corner vertex of the true bounding box of the second sequence of characters,

can represent the bottom-right vertex of the true bounding box of the second sequence of characters,

The lower left vertex of the ground-truth bounding box that may represent the second sequence of characters. for any second feature point

, any boundary line of the true bounding box of the second character sequence corresponds to the true value of the distance parameter of the second feature point

and the truth value of the angle parameter

Equations (9) to (16) can be used to determine:

formula (9);

formula (10);

formula (11);

formula (12);

formula (13);

formula (14);

formula (15);

Formula (16); where,

represents the true value of the vertical vector from the second feature point to the boundary line,

is parallel to the vertical line from the second feature point to the boundary line, and points from the second feature point to the vertical point,

express

below the polar axis.

As an example of this embodiment, the method further includes: predicting the probability that the location of the primitive in the training image belongs to the character via the neural network; the probability that the location of the primitive in the training image belongs to the character, and The location of the primitives in the training image belongs to the labeling data of the characters, and the neural network is trained. In this example, the neural network can be a multi-task learning model, learning character segmentation (ie, learning to detect the probability that the position of a primitive in an image belongs to a character) and the parameter prediction of the boundary line. According to this example, the neural network can be made to learn the ability to predict the probability that the location of the primitive belongs to the character.

In one example, according to the probability that the position of the primitive in the training image belongs to the character, and the annotation data that the position of the primitive in the training image belongs to the character, training the neural network includes: according to the second character The probability that the position of the primitive in the effective area corresponding to the sequence belongs to the character, and the position of the primitive in the effective area belongs to the labeling data of the character to train the neural network.

公式（17）； 其中，

表示第二字元序列對應的有效區域，

表示第二字元序列對應的有效區域中的圖元數；

表示第二字元序列對應的有效區域中的圖元

所在位置屬於字元的標注資料，例如，若圖元

所在位置屬於字元，則

，若圖元

所在位置不屬於字元，則

；

表示第二字元序列對應的有效區域中的圖元

所在位置屬於字元的概率，

。 For example, the loss function corresponding to character segmentation can be obtained by using formula (17):

Formula (17); where,

Indicates the valid area corresponding to the second character sequence,

Indicates the number of primitives in the valid area corresponding to the second character sequence;

Indicates the primitives in the valid area corresponding to the second character sequence

Label data where the location belongs to the character, for example, if the element

the position belongs to a character, then

, if the primitive

position does not belong to a character, then

;

Indicates the primitives in the valid area corresponding to the second character sequence

the probability that the location belongs to the character,

.

In this example, the neural network is trained according to the probability that the position of the primitive in the effective area corresponding to the second character sequence belongs to the character, and the position of the primitive in the effective area belongs to the labeling data of the character. This enables the neural network to learn the ability of character segmentation, and can improve the efficiency of the neural network to learn character segmentation.

訓練神經網路：

公式（18）； 其中，

表示字元分割對應的損失函數，

表示距離參數對應的損失函數，

表示角度參數對應的損失函數，

表示

對應的權重，

表示

對應的權重，

表示

對應的權重，

、

和

可以根據經驗或者訓練策略等靈活設置，例如，

，在此不作限定。 In one example, a loss function as shown in Equation (18) can be used

Train the neural network:

Formula (18); where,

represents the loss function corresponding to the character segmentation,

represents the loss function corresponding to the distance parameter,

represents the loss function corresponding to the angle parameter,

express

corresponding weight,

express

corresponding weight,

express

corresponding weight,

,

and

It can be flexibly set according to experience or training strategy, for example,

, which is not limited here.

As an example of this embodiment, the neural network may include at least one channel reduction module, so as to reduce the calculation amount of the neural network and improve the speed of the boundary line detection by the neural network.

As an example of this embodiment, the neural network may include at least one feature aggregation module, so as to fully utilize multi-scale features and improve the accuracy of boundary line detection performed by the neural network.

An application scenario of the embodiment of the present invention is described below. FIG. 5 shows a schematic diagram of an application scenario of an embodiment of the present invention. As shown in Figure 5, the neural network can be an encoder-decoder structure. In Figure 5, 506 denotes a channel reduction module. For example, the channel reduction module 506 can be implemented using 1×1 convolution. Of course, the channel reduction module 506 can also be implemented by using 3×3 convolution, etc., which is not limited here. 507 represents a feature aggregation module. The feature aggregation module 507 may be used to perform at least one operation of multiplying, adding, concat (merging), and the like, on the input feature maps. For example, as shown in FIG. 5 , the feature aggregation module 507 can double the size (width and height) of the input feature map, and then perform concat, 1 based on the expanded feature map and the output of the channel reduction module 506 . ×1 non-linear convolution and 3×3 non-linear convolution. As shown in Figure 5, the neural network can use the skeleton network to extract basic features, and continuously integrate features of different scales through the feature aggregation module, and finally obtain feature maps of 9 channels, one of which is the text confidence level 504 (that is, the input The probability of each primitive in the image inputting the character), the other 8 channels are the distance parameters and angle parameters of the straight line equation of the four boundary lines, that is, the parameters 503 of the four boundary lines. According to the distance parameters and angle parameters of each boundary line of the three character sequences in the input image 501 in the polar coordinate system, the straight line equation of each boundary line of the three character sequences in the Cartesian coordinate system can be obtained. The straight line equation of the 4 boundary lines is visualized in the dashed box 505 on the right side of FIG. 5, wherein the upper boundary, right boundary, lower boundary and left side of the 3 character sequence are shown sequentially from top to bottom boundaries. According to the straight line equation of each boundary line of the 3-character sequence in the input image, the bounding box 502 of the 3-character sequence can be obtained, as shown in the lower left of FIG. 5 .

The character detection method provided by the embodiment of the present invention can be applied to character detection in general natural scenarios, as well as application scenarios such as real-time text translation, document recognition, certificate recognition (such as ID cards, bank cards), and license plate recognition. Not limited. In some natural scenes, characters in the image will appear as irregular quadrilaterals due to camera perspective distortion. By adopting the embodiments of the present invention, the boundary of the character can be accurately detected, thereby further correcting the shape of the character, which is beneficial to the subsequent character recognition. In addition, in addition to characters, some character carriers also exhibit the above phenomenon, such as rigid ID cards, bank cards, and license plates. By using the embodiments of the present invention to detect the boundaries of these quadrilateral carriers containing characters, it is also beneficial to the subsequent character identification process.

It can be understood that the above method embodiments mentioned in the present invention can be combined with each other to form a combined embodiment without violating the principle and logic. Due to space limitations, the present invention will not repeat them. Those skilled in the art can understand that, in the above method of the specific embodiment, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the present invention also provides a character detection device, an electronic device, a storage medium and a program, all of which can be used to implement any character detection method provided by the present invention. For the corresponding technical solutions and technical effects, please refer to the corresponding records in the method section, No longer.

FIG. 6 shows a block diagram of an apparatus for character detection provided by an embodiment of the present invention. As shown in Figure 6, the character detection device 6 includes: The first prediction module 61 is configured to predict the multiple boundary lines of the first character sequence in the image to be processed respectively, and obtain the prediction parameters of the multiple boundary lines of the first character sequence. A boundary line of a character sequence represents the boundary between the area where the first character sequence is located and the area not where the first character sequence is located; a first determination module 62, configured to determine the position information of the vertices of the bounding box of the first character sequence according to the prediction parameters of a plurality of boundary lines of the first character sequence; The second determining module 63 is configured to determine the position information of the bounding box of the first character sequence according to the position information of the vertices of the bounding box of the first character sequence.

In some embodiments of the present invention, the first prediction module 61 is further configured to, based on the to-be-processed image, respectively predict the first character for the first feature point related to the first character sequence A plurality of boundary lines of the sequence correspond to the parameters of the first feature point; The prediction parameters of the plurality of boundary lines of the first character sequence are determined according to the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature points.

In some embodiments of the present invention, the character detection device 6 further includes: The second prediction module is configured to predict the probability that the position of the primitive in the to-be-processed image belongs to the character; The third determination module is configured to determine the first feature point according to the probability that the position of the graphic element in the image to be processed belongs to the character element.

In some embodiments of the present invention, the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature point include: The distance parameters and angle parameters of the plurality of boundary lines of the first character sequence in the polar coordinate system corresponding to the first feature point, wherein the polar coordinate system corresponding to the first feature point is represented by the first feature point. A polar coordinate system with a point as a pole.

In some embodiments of the present invention, the first prediction module 61 is further configured to calculate distance parameters and angles of multiple boundary lines of the first character sequence in the polar coordinate system corresponding to the first feature point The parameter is mapped to the Cartesian coordinate system, and the multiple boundary lines of the first character sequence are obtained corresponding to the parameters of the first feature point under the Cartesian coordinate system; The prediction parameters of the plurality of boundary lines of the first character sequence are determined according to the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature point in the Cartesian coordinate system.

In some embodiments of the present invention, the plurality of boundary lines of the first sequence of characters include an upper boundary, a right boundary, a lower boundary and a left boundary of the first sequence of characters.

In some embodiments of the present invention, the first prediction module 61 is further configured to input the to-be-processed image into a pre-trained neural network, and use the neural network to analyze the predictions related to the first character sequence through the neural network. A first feature point, respectively predicting that a plurality of boundary lines of the first character sequence correspond to parameters of the first feature point.

In some embodiments of the present invention, the character detection device 6 further includes: The third prediction module is configured to predict the probability that the location of the primitive in the image to be processed belongs to the character through the neural network.

In some embodiments of the present invention, the character detection device 6 further includes: The fourth prediction module is configured to input the training image into the neural network, and through the neural network, respectively predict the second feature points related to the second character sequence in the training image. A plurality of boundary lines of the second character sequence corresponds to the predicted value of the parameter of the second feature point; The first training module is configured to correspond to the predicted value of the parameter of the second feature point according to the multiple boundary lines of the second character sequence, and the multiple boundary lines of the second character sequence correspond to The true value of the parameter of the second feature point is used to train the neural network.

In some embodiments of the present invention, the parameters of the plurality of boundary lines of the second character sequence corresponding to the second feature point include: the plurality of boundary lines of the second character sequence are in the second The distance parameter and the angle parameter under the polar coordinate system corresponding to the feature point, wherein, the polar coordinate system corresponding to the second feature point represents a polar coordinate system with the second feature point as a pole;

The first training module is configured to correspond to the predicted value of the distance parameter of the second feature point according to the plurality of boundary lines of the second character sequence, and the plurality of boundary lines of the second character sequence training the neural network corresponding to the true value of the distance parameter of the second feature point; and/or, According to the plurality of boundary lines of the second character sequence corresponding to the predicted value of the angle parameter of the second feature point, and the plurality of boundary lines of the second character sequence corresponding to the second feature point The true value of the angle parameter to train the neural network.

In some embodiments of the present invention, the first training module is configured to, for any one of a plurality of boundary lines of the second character sequence, correspond to the second feature according to the boundary line The ratio of the smaller value to the larger value of the true value and the predicted value of the distance parameter of the point, the neural network is trained.

In some embodiments of the present invention, the first training module is configured to, for any one of a plurality of boundary lines of the second character sequence, determine that the boundary line corresponds to the second feature The absolute value of the difference between the true value of the point's angle parameter and the predicted value;

The neural network is trained according to the sine of the half angle of the absolute value.

In some embodiments of the present invention, the second feature points include feature points in an effective area corresponding to the second character area.

In some embodiments of the present invention, the apparatus further comprises: a fifth prediction module, configured to predict the probability that the position of the primitive in the training image belongs to the character through the neural network; The second training module is configured to train the neural network according to the probability that the position of the graphic element in the training image belongs to the character, and the labeling data that the position of the graphic element in the training image belongs to the character road.

In some embodiments of the present invention, the second training module is further configured to be based on the probability that the position of the graphic element in the effective area corresponding to the second character sequence belongs to the character, and the probability of the graphic element in the effective area The location of the graphic element belongs to the labeling data of the character, and the neural network is trained.

In some embodiments of the present invention, the character detection device 6 further includes: an acquisition module, configured to acquire the position information of the real bounding box of the second character sequence; The shrinking module is configured to shrink the real bounding box according to the position information of the real bounding box and a preset ratio to obtain an effective area corresponding to the second character sequence.

In some embodiments of the present invention, the reduction module is further configured to determine the anchor point of the real bounding box according to the position information of the real bounding box, wherein the anchor point of the real bounding box is the the intersection of the diagonals of the true bounding box; According to the position information of the real bounding box, the position information of the anchor point of the real bounding box, and the preset ratio, the real bounding box is reduced to obtain the effective area corresponding to the second character sequence, wherein the first The ratio of a distance to a second distance is equal to the preset ratio, the first distance represents the distance between the first vertex of the effective area and the anchor point, and the second distance represents the distance in the real bounding box The distance between the vertex corresponding to the first vertex and the anchor point, where the first vertex represents any vertex of the effective area.

In the embodiment of the present invention, the prediction parameters of the plurality of boundary lines of the first character sequence are obtained by respectively predicting the plurality of boundary lines of the first character sequence in the image to be processed. the prediction parameters of the boundary lines, determine the position information of the vertices of the bounding box of the first character sequence, and determine the position information of the bounding box of the first character sequence according to the position information of the vertices of the bounding box of the first character sequence , so that the polygon (such as quadrilateral) bounding box of the character sequence is decomposed into multiple (such as four) independent boundary lines, and each independent boundary line is detected separately, so that the detection of each boundary line will not is interfered by two different vertices, which in turn can improve the accuracy of character detection.

In some embodiments, the functions or modules included in the character detection device 6 provided in the embodiments of the present invention may be configured to execute the methods described in the above method embodiments, and the specific implementation and technical effects thereof may be implemented with reference to the above methods The description of the example will not be repeated here.

Embodiments of the present invention also provide a computer-readable storage medium, which stores computer program instructions, and the computer program instructions implement the above method when executed by a processor. Wherein, the computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.

Embodiments of the present invention further provide a computer program, including computer-readable codes. When the computer-readable codes are executed in an electronic device, a processor in the electronic device executes the character detection method provided by any of the foregoing embodiments.

The embodiments of the present invention further provide another computer program product for storing computer-readable instructions, and when the instructions are executed, the computer executes the operations of the character detection method provided by any of the above-mentioned embodiments.

An embodiment of the present invention also provides an electronic device, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to Execute the character detection method provided by any of the above embodiments.

The electronic device may be provided as a terminal, server or other form of device.

FIG. 7 shows a block diagram of an electronic device 800 provided by an embodiment of the present invention. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

7 , an electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power supply component 806 , a multimedia component 808 , an audio component 810 , and an Input/Output (I/O) interface 812 , sensor component 814 , and communication component 816 .

The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

The memory 804 is configured to store various types of data to support the operation of the electronic device 800 . Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Design Read Only Memory (Electrically Erasable Programmable read only memory, EEPROM), Erasable Programmable Read Only Memory (Electrical Programmable Read Only Memory, EPROM), Programmable Read-Only Memory (Programmable Read-Only Memory, PROM), Read-Only Memory (ROM), Magnetic Memory, Flash Memory, Disk or CD.

Power supply assembly 806 provides power to various components of electronic device 800 . Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (Microphone, MIC) configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a calling mode, a recording mode, and a voice recognition mode. The received audio signal may be further stored in memory 804 or transmitted via communication component 816 . In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules. The peripheral interface modules may be keyboards, click wheels, buttons, and the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for electronic device 800 . For example, the sensor assembly 814 can detect the open/closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, the sensor assembly 814 can also detect the electronic device 800 or Changes in the position of a component of the electronic device 800 , presence or absence of user contact with the electronic device 800 , orientation or acceleration/deceleration of the electronic device 800 and changes in the temperature of the electronic device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. The sensor assembly 814 may also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge-coupled Device (CCD) image sensor, for use in imaging applications used in. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (Wi-Fi), a second-generation mobile communication technology (2-Generation, 2G), a third-generation mobile communication technology (3rd-Generation, 3G) ), the fourth generation mobile communication technology (4-Generation, 4G)/, the long term evolution (Long Term Evolution, LTE) of the universal mobile communication technology, the fifth generation mobile communication technology (5-Generation, 5G) or their combination. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BitTorrent, BT) technology and other technologies to achieve.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more of Application Specific Integrated Circuits (ASICs), Digital Signal Processes (DSPs), Digital Signal Processes (Digital Signal Processes) Device, DSPD), programmable logic device (Programmable Logic Device, PLD), Field Programmable Gate Array (Field Programmable Gate Array, FPGA), controller, microcontroller, microprocessor or other electronic components to achieve, with to execute the above method.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions executable by the processor 820 of the electronic device 800 to accomplish the above method.

FIG. 8 shows a block diagram of an electronic device 1900 provided by an embodiment of the present invention. For example, the electronic device 1900 may be provided as a server. 8, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions executable by the processing component 1922, such as applications. An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input-output interface 1958. The electronic device 1900 can operate an operating system based on the memory 1932, such as Microsoft Server Operating System (Windows ServerTM), a graphical user interface based operating system (Mac OS XTM) introduced by Apple, a multi-user multi-process operating system. Computer Operating System (UnixTM), Free and Open Source Unix-like Operating System (LinuxTM), Open Source Unix-like Operating System (FreeBSDTM) or similar.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions executable by the processing component 1922 of the electronic device 1900 to accomplish the above method.

The present invention may be a system, method and/or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), ROM, EPROM or flash memory, SRAM, Portable Compact Disc Read-Only Memory (CD-ROM), Digital Video Disc (DVD), Memory Sticks, Floppy Disks, Mechanical Encoding Devices, such as A punched card or a raised structure in a groove storing instructions, and any suitable combination of the above. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or Electrical signals carried by wires.

The computer-readable program instructions described herein may be downloaded from computer-readable storage media to various computing/processing devices, or downloaded to external computers or external storage over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network device. Networks may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. A network interface card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for computer-readable storage stored in each computing/processing device in the medium.

The computer program instructions for carrying out the operations of the present invention may be assembly instructions, Industry Standard Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or in one or more programs Source or object code written in any combination of design languages, including object-oriented programming languages - such as Smalltalk, C++, etc., and conventional procedural programming languages - such as "C" or the like programming language. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely remotely. run on a client computer or server. In the case of a remote computer, the remote computer can be connected to the user computer via any kind of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or, alternatively, it can be connected to to an external computer (eg using an Internet service provider to connect via the Internet). In some embodiments, by utilizing state information of computer readable program instructions to personalize custom electronic circuits, such as programmable logic circuits, FPGAs, or Programmable Logic Arrays (PLAs), the electronic circuits may execute Computer readable program instructions to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to the processor of a general purpose computer, special purpose computer or other programmable data processing device to produce a machine for execution of the instructions by the processor of the computer or other programmable data processing device When, means are created that implement the functions/acts specified in one or more of the blocks in the flowchart and/or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium, the instructions causing the computer, programmable data processing device and/or other equipment to operate in a particular manner, so that the computer readable medium storing the instructions Included is an article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded into a computer, other programmable data processing device, or other equipment, so that a series of operational steps are performed on the computer, other programmable data processing device, or other equipment to generate a computer Processes of implementation such that instructions executing on a computer, other programmable data processing apparatus, or other device implement the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions that contains one or more logic for implementing the specified logic Executable instructions for the function. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by dedicated hardware-based systems that perform the specified functions or actions. implementation, or may be implemented in a combination of special purpose hardware and computer instructions.

The computer program product can be implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.

Industrial Applicability The present invention provides a character detection method, an electronic device and a computer-readable storage medium; wherein, a plurality of boundary lines of a first character sequence in a to-be-processed image are predicted respectively, and multiple boundary lines of the first character sequence are obtained. prediction parameters of a boundary line, wherein the boundary line of the first character sequence represents the boundary between the area where the first character sequence is located and the area not where the first character sequence is located; The prediction parameters of a plurality of boundary lines of a character sequence determine the position information of the vertices of the bounding box of the first character sequence; according to the position information of the vertices of the bounding box of the first character sequence, determine the Location information for the bounding box of the first character sequence.

201: Image acquisition terminal 202: Internet 203: Determine the location terminal 31: Ground-truth bounding box 32: Effective area 501: input image 502: Bounding Box 503: Parameters of the four boundary lines 504: Text Confidence 505: Dotted box 506: Channel Cut Module 507: Feature Aggregation Module 6: Character detection device 61: The first prediction module 62: First determine the module 63: The second determination module 800: Electronics 802: Process component 804: memory 806: Power Components 808: Multimedia Components 810: Audio Components 812: Input/Output Interface 814: Sensor Assembly 816: Communication Components 820: Processor 1900: Electronic equipment 1922: Processing components 1926: Power Components 1932: Memory 1950: Web Interface 1958: Input and output interface S11~S13: Steps

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present invention, and together with the description, serve to explain the technical solutions of the present invention. 1 shows a flowchart of a method for detecting a character provided by an embodiment of the present invention; 2 shows a schematic diagram of a system architecture applying the character detection method according to an embodiment of the present invention; Fig. 3 shows the schematic diagram of the distance parameter and the angle parameter of 4 boundary lines of the first character sequence under the polar coordinate system corresponding to a certain first feature point; Fig. 4 shows the schematic diagram of the real bounding box 31 and the effective area 32 of the second character area; 5 shows a schematic diagram of an application scenario of an embodiment of the present invention; 6 shows a block diagram of a character detection device provided by an embodiment of the present invention; FIG. 7 shows a block diagram of an electronic device 800 provided by an embodiment of the present invention; FIG. 8 shows a block diagram of an electronic device 1900 provided by an embodiment of the present invention.

S11~S13: Steps

Claims (19)

A character detection method, comprising: Predicting the multiple boundary lines of the first character sequence in the image to be processed respectively, to obtain the prediction parameters of the multiple boundary lines of the first character sequence, wherein the boundary line of the first character sequence represents the the boundary between the area where the first character sequence is located and the area not where the first character sequence is located; determining the position information of the vertices of the bounding box of the first character sequence according to the prediction parameters of the plurality of boundary lines of the first character sequence; The position information of the bounding box of the first character sequence is determined according to the position information of the vertices of the bounding box of the first character sequence. According to the method described in claim 1, the multiple boundary lines of the first character sequence in the image to be processed are predicted respectively, and the prediction parameters of the multiple boundary lines of the first character sequence are obtained, including: Based on the to-be-processed image, for the first feature point related to the first character sequence, predict the parameters corresponding to the first feature point of a plurality of boundary lines of the first character sequence respectively; The prediction parameters of the plurality of boundary lines of the first character sequence are determined according to the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature points. According to the method of claim 2, the method further includes: Predict the probability that the position of the primitive in the image to be processed belongs to the character; The first feature point is determined according to the probability that the position of the graphic element in the image to be processed belongs to the character element. According to the method described in claim 2 or 3, the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature point include: The distance parameters and angle parameters of the plurality of boundary lines of the first character sequence in the polar coordinate system corresponding to the first feature point, wherein the polar coordinate system corresponding to the first feature point is represented by the first feature point. A polar coordinate system with a point as a pole. According to the method of claim 4, the prediction of the plurality of boundary lines of the first character sequence is determined according to the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature points parameters, including: The distance parameters and angle parameters of the plurality of boundary lines of the first character sequence under the polar coordinate system corresponding to the first feature point are mapped to the Cartesian coordinate system to obtain the plurality of boundaries of the first character sequence The line corresponds to the parameter of the first feature point in the Cartesian coordinate system; The prediction parameters of the plurality of boundary lines of the first character sequence are determined according to the parameters of the plurality of boundary lines of the first character sequence corresponding to the first feature point in the Cartesian coordinate system. According to the method of any one of claims 1 to 3, the plurality of boundary lines of the first character sequence includes an upper boundary line, a right boundary line, a lower boundary line and a left boundary line of the first character sequence. The method according to claim 2, wherein, based on the to-be-processed image, for the first feature points related to the first character sequence, it is respectively predicted that multiple boundary lines of the first character sequence correspond to the Describe the parameters of the first feature point, including: Input the image to be processed into a pre-trained neural network, and through the neural network, for the first feature points related to the first character sequence, respectively predict the correspondence of multiple boundary lines of the first character sequence parameters at the first feature point. According to the method of claim 7, the method further includes: The probability that the location of the primitive in the to-be-processed image belongs to the character is predicted through the neural network. According to the method of claim 7 or 8, before inputting the to-be-processed image into the pre-trained neural network, the method further includes: Input the training image into the neural network, and through the neural network, for the second feature points related to the second character sequence in the training image, respectively predict multiple items of the second character sequence. The boundary line corresponds to the predicted value of the parameter of the second feature point; According to the plurality of boundary lines of the second character sequence corresponding to the predicted value of the parameter of the second feature point, and the plurality of boundary lines of the second character sequence corresponding to the parameter of the second feature point The true value of , to train the neural network. According to the method of claim 9, the parameters of the plurality of boundary lines of the second character sequence corresponding to the second feature points include: the plurality of boundary lines of the second character sequence are in the second The distance parameter and the angle parameter under the polar coordinate system corresponding to the feature point, wherein, the polar coordinate system corresponding to the second feature point represents a polar coordinate system with the second feature point as a pole; The plurality of boundary lines according to the second character sequence corresponds to the predicted value of the parameter of the second feature point, and the plurality of boundary lines of the second character sequence corresponds to the second feature point The ground-truth values of the parameters that train the neural network, including: The plurality of boundary lines of the second character sequence correspond to the predicted value of the distance parameter of the second feature point, and the plurality of boundary lines of the second character sequence correspond to the predicted value of the distance parameter of the second feature point. the true value of the distance parameter to train the neural network; and/or, According to the plurality of boundary lines of the second character sequence corresponding to the predicted value of the angle parameter of the second feature point, and the plurality of boundary lines of the second character sequence corresponding to the second feature point The true value of the angle parameter to train the neural network. The method according to claim 10, wherein the plurality of boundary lines according to the second character sequence corresponds to the predicted value of the distance parameter of the second feature point, and the plurality of the second character sequence The boundary line corresponds to the true value of the distance parameter of the second feature point, and the training of the neural network includes: For any boundary line among the plurality of boundary lines of the second character sequence, according to the boundary line corresponding to the actual value and the predicted value of the distance parameter of the second feature point, the smaller value and the larger value The ratio of values to train the neural network. The method according to claim 10, wherein the plurality of boundary lines according to the second character sequence corresponds to the predicted value of the angle parameter of the second feature point, and the plurality of the second character sequence The boundary line corresponds to the true value of the angle parameter of the second feature point, and the training of the neural network includes: For any one of the multiple boundary lines of the second character sequence, determine the absolute value of the difference between the true value and the predicted value of the boundary line corresponding to the angle parameter of the second feature point; The neural network is trained according to the sine of the half angle of the absolute value. According to the method of claim 9, the second feature points include feature points in an effective area corresponding to the second character area. According to the method of claim 9, the method further includes: Predicting the probability that the position of the primitive in the training image belongs to the character through the neural network; The neural network is trained according to the probability that the position of the graphic element in the training image belongs to the character element, and the labeling data that the position of the graphic element in the training image belongs to the character element. The method according to claim 14, wherein the training image is based on the probability that the position of the graphic element in the training image belongs to the character element, and the labeling data of the position of the graphic element in the training image belonging to the character element. The neural network described above, including: The neural network is trained according to the probability that the position of the graphic element in the effective area corresponding to the second character sequence belongs to the character element, and the labeling data of the position of the graphic element in the effective area belonging to the character element. According to the method of claim 13, the method further comprises: obtaining location information of the real bounding box of the second character sequence; According to the position information of the real bounding box and a preset ratio, the real bounding box is reduced to obtain an effective area corresponding to the second character sequence. According to the method described in claim 16, according to the position information of the real bounding box and a preset ratio, reducing the real bounding box to obtain the effective area corresponding to the second character sequence, including: According to the position information of the real bounding box, the anchor point of the real bounding box is determined, wherein the anchor point of the real bounding box is the intersection of the diagonal lines of the real bounding box; According to the position information of the real bounding box, the position information of the anchor point of the real bounding box, and the preset ratio, the real bounding box is reduced to obtain the effective area corresponding to the second character sequence, wherein the first The ratio of a distance to a second distance is equal to the preset ratio, the first distance represents the distance between the first vertex of the effective area and the anchor point, and the second distance represents the distance in the real bounding box The distance between the vertex corresponding to the first vertex and the anchor point, where the first vertex represents any vertex of the effective area. An electronic device comprising: one or more processors; memory used to store executable instructions; The one or more processors are configured to invoke executable instructions stored in the memory to execute the character detection method described in any one of request items 1 to 17. A computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, implement the character detection method described in any one of request items 1 to 17.

Images