CN111767889A - Formula recognition method, electronic device and computer readable medium - Google Patents

Abstract

The embodiment of the present invention discloses a formula recognition method: preprocessing a picture containing a formula, performing formula symbol detection on the preprocessed picture, and obtaining category information and position information of the formula symbol contained in the above formula; based on the formula symbol The category information and location information of , construct a mixed feature vector; based on the mixed feature vector, identify and convert the above formula symbols, and obtain the character string corresponding to the formula contained in the above picture. Since this scheme constructs the detected category information and position information of the formula symbols as mixed feature vectors, the accuracy rate is high in the process of identifying and converting the formula symbols.

Description

Formula recognition method, electronic device and computer readable medium

technical field

Embodiments of the present invention relate to the technical field of text recognition, and in particular, to a formula recognition method, an electronic device, and a computer-readable medium in a natural scene.

Background technique

Recognition of formulas in natural scenes is a process of obtaining pictures containing formulas through operations such as taking pictures and scanning in natural scenes, and then identifying the formulas in the pictures as latex strings.

At present, although a variety of algorithms and neural network models can be used to identify formulas in pictures, due to the complex structure of formulas and the extremely rich expressions of formulas in natural scenes: for example, formulas can have different sizes, fonts, and colors. , brightness, contrast, etc., and may be bent, rotated, distorted, etc. As a result, the recognition accuracy of the formula in the picture is not high, and the recognition result is not accurate enough.

SUMMARY OF THE INVENTION

The present invention provides a formula recognition scheme to at least partially solve the above problems.

According to a first aspect of the embodiments of the present invention, there is provided a formula recognition method, the method includes: preprocessing a picture containing formulas to obtain a preprocessed picture; Symbol detection, to obtain the category information and position information of the formula symbol contained in the formula; based on the category information and position information of the formula symbol, construct a mixed feature vector; Convert, to obtain the character string corresponding to the formula contained in the picture.

According to a second aspect of the embodiments of the present invention, there is provided an electronic device, the device comprising: one or more processors; a computer-readable medium configured to store one or more programs, when the one or more processors A program is executed by the one or more processors so that the one or more processors implement the formula recognition method as described in the first aspect.

According to a third aspect of the embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, implements the formula identification method according to the first aspect.

According to the solution provided by the embodiment of the present invention: preprocess the picture containing the formula, and perform formula symbol detection on the preprocessed picture to obtain the category information and position information of the formula symbol contained in the above formula; based on the category information of the formula symbol and position information to construct a mixed feature vector; based on the mixed feature vector, identify and convert the above formula symbols, and obtain the character string corresponding to the formula contained in the above picture. Since the hybrid feature vector constructed in this scheme includes the position information and category information of the formula symbol, the category of the formula symbol can be more accurately determined by the category information, and the position of the formula symbol can be clearly indicated by the position information, so that the The information used for the identification and conversion of formula symbols is more comprehensive and complete, the formula symbols can be identified more accurately, and the accuracy and efficiency of the identification and conversion of formula symbols are higher.

Description of drawings

Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG. 1 is a flow chart of the steps of a formula recognition method according to Embodiment 1 of the present invention;

2 is a schematic diagram of a neural network model based on a Yolo structure according to Embodiment 1 of the present invention;

3 is a schematic diagram of an attention-based sequence-to-sequence model according to Embodiment 1 of the present invention;

FIG. 4 is a flowchart of a formula recognition method according to Embodiment 2 of the present invention;

FIG. 5 is a schematic structural diagram of an electronic device according to Embodiment 3 of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only configured to explain the related invention, rather than limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that the embodiments of the present invention and the features of the embodiments may be combined with each other under the condition of no conflict. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

Example 1

Referring to FIG. 1 , a flow chart of steps of a formula recognition method according to Embodiment 1 of the present invention is shown.

The formula recognition method of this embodiment includes the following steps:

Step 101: Preprocess the picture containing the formula to obtain a preprocessed picture.

In order to reduce the influence of pictures containing formulas of different scales on the recognition and positioning of formula symbols, and to reduce the proportion of the formula symbol area in the pictures containing formulas, and the influence of the size of the formula symbols on the identification and positioning of formula symbols, you can Images containing formulas are preprocessed.

In this embodiment, optionally, the image containing the formula may be preprocessed in the following manner:

First, binarize the image containing the formula to obtain a binarized image; then, from the binarized image, determine the image area where the above formula is located; finally, according to the image area cut out from the binarized image, obtain a Processed picture. In the preprocessed picture obtained in this way, on the one hand, the binarized picture removes a lot of unnecessary information, especially non-formula information; on the other hand, the area where the formula is located in the picture can be preserved and the redundant The picture area can improve the detection efficiency in subsequent formula symbol detection.

In this embodiment, binarization processing is performed on a picture containing a formula, wherein a specific manner of binarization processing can be implemented by those skilled in the art in any appropriate manner, which is not limited in this embodiment of the present invention. In this embodiment, in the binarized image after the binarization process, the pixel value corresponding to the formula symbol pixel is 1, and the pixel value of the non-formula symbol pixel is 0.

Then, a plurality of coordinate points with a pixel value of 1 may be obtained from the binarized picture, and the cutting range of the above-mentioned binarized picture may be determined according to the plurality of coordinate points, as the picture area where the formula is located. In this way, the positions of multiple pixel points with a pixel value of 1 can be obtained, and then the approximate area of the formula in the picture can be determined. It can be understood that the more coordinate points obtained, the more accurate the determination of the area of the formula in the picture.

Optionally, the vertex coordinates of the circumscribed quadrilateral of the formula can be obtained according to the maximum abscissa, the minimum abscissa, the maximum ordinate and the minimum ordinate in the multiple coordinate points; then the cutting range is determined according to the vertex coordinates, and the The above binarized image is cut. In this way, a picture of the area where the formula is located can be efficiently obtained.

For example, the minimum abscissa x1, the maximum abscissa x2, the minimum ordinate y1, and the maximum ordinate y2 can be obtained from all the coordinate points with the pixel value of 1 obtained from the binarized image; thus, the pixel value of 1 can be obtained. The upper left corner coordinate point (x1, y2), the lower left corner coordinate point (x1, y1), the upper right corner coordinate point (x2, y2) and the lower right corner coordinate point (x2, y1) of the image area. Finally, the above-mentioned binarized image can be cut by using the quadrilateral formed by the four coordinate points as the cutting range. The binarized picture is cut through the above four coordinate points. Since the minimum abscissa, the maximum abscissa, the minimum ordinate and the maximum ordinate with the pixel value of 1 are included, the picture where the formula is located can be quickly and completely obtained. area.

Optionally, the image area cut out from the binarized image can be scaled according to a preset ratio, and then subjected to edge-filling processing to obtain a preprocessed image. Wherein, the preset ratio can be set by those skilled in the art according to the actual situation, and can meet the requirements of subsequent processing.

In a feasible manner, the image area cut out from the binarized image can be scaled according to the scaling factor (nx/(x2-x1), ny/(y2-y1)), where nx represents n and The product of x, ny represents the product of n and y. Among them, n is the number of components corresponding to the formula in the binarized picture, which is specifically realized as the number of connected domains corresponding to the formula, which can be obtained by using any appropriate connected domain detection method. Wherein, in the formula, a connected domain corresponds to a component of a formula, and one or more components constitute a component of the formula. For example: in the formula 6+3=9, the number of components is 6, "6", "3", "9", "+" correspond to one component respectively, and "=" has two connected domains, corresponding to two components. The components of the formula are formed by various formula symbols. As shown in the above formula, there are 5 components, namely "6", "3", "9", "+" and "=".

The above-mentioned values of x and y can be obtained through big data statistics. For example, a preset number of formulas can be selected, the size of the formula symbols in each formula can be calculated, and then the sizes of all formula symbols can be averaged to obtain. There is no limit to the number of presets. Among them, x can represent the height of the formula symbol, y can represent the width of the formula symbol, so (x, y) can represent the pixel size of the formula symbol.

In this embodiment, according to the zoom factor (nx/(x2-x1), ny/(y2-y1)), the picture area cut out from the binarized picture is scaled, so that the formula symbol contained in the formula in the picture can be scaled Similar in size.

Next, the scaled image can be edge-replenished to obtain a preprocessed image, and the preprocessed image can be a (1024, 256) size image. However, the size is only an exemplary description, and in practical applications, those skilled in the art can process the preprocessed picture into a required size according to actual needs. Performing edge-filling processing on the scaled picture can enable the pixels at the outermost periphery of the preprocessed picture to be completely detected when formula symbols are detected subsequently.

In this embodiment, the preprocessing of pictures containing formulas can reduce the influence of pictures containing formulas of different scales on the identification and positioning of formula symbols, and can also reduce the size of the proportion of the formula symbol area in the pictures containing formulas, And the influence of the size of the formula symbol on the identification and positioning of the formula symbol, so that the accuracy of formula identification and positioning can be improved.

Step 102: Detect formula symbols in the preprocessed picture to obtain category information and location information of formula symbols included in the formula.

In this embodiment, the category information of the formula symbol is used to indicate the category of the formula symbol, including but not limited to number category, letter category, operation symbol category, punctuation category, etc. This embodiment does not limit, and the position information can represent formula The coordinates of the position of the symbol in the picture.

Optionally, the preprocessed picture may be input into the first neural network model for formula symbol detection, to obtain category information and position information of formula symbols contained in the above formula. Formula symbol detection is performed through the first neural network model, and the detection accuracy is high.

Specifically, the preprocessed picture can be input into the first neural network model for formula symbol detection, and the first neural network model can perform multi-scale feature extraction and symbol detection on the preprocessed picture to obtain the formula contained in the formula. Category information and location information of the symbol. The multi-scale feature extraction is performed on the preprocessed image through the first neural network model, which can ensure higher accuracy of formula symbol detection.

In this embodiment, the first neural network model may be a neural network model based on the Yolo structure. Taking Yolo_v3 as an example, the existing Yolo_v3 uses three feature maps of different scales for object detection. In order to avoid the loss of shallow information as the number of network layers deepens, the accuracy of formula recognition and positioning will be affected. The neural network model based on the Yolo structure in this embodiment can perform feature extraction of at least four scales, and obtain corresponding feature maps of at least four scales. Among them, the feature extraction of at least four scales includes low-scale feature extraction. Then, based on the feature maps of at least four scales, symbol frame detection and symbol recognition are performed to obtain the category information and position information of the formula symbols contained in the above formula. Among them, low-scale feature extraction is based on feature extraction between pixels, and the extracted low-scale features are basic features that can be automatically extracted from images without any shape/spatial relationship information. In this embodiment, by adding low-scale feature extraction, symbols with unconventional shapes and proportions, such as small target symbols such as dotted symbols ".", and long horizontal line symbols "------", can be effectively used. Feature extraction.

等符号。其中，预设比例可以由本领域技术人员根据实际需求适当设置，本发明实施例对此不作限制。在基于Yolo结构的神经网络模型中设置上述大小的锚框，可以提高不常见的公式符号识别准确率和定位精度，例如

“Ω”、“─”、“∫”等公式符号识别准确率和定位精度。In the existing Yolo_v3, 9 anchor boxes of different scales are set, namely: (10, 13), (16, 30), (33, 23), (30, 61), (62, 45), (59, 119), (116, 90), (156, 198), (373, 326), these anchor boxes are not accurate enough to recognize some target objects, such as the recognition of some mathematical symbols. In this embodiment, 12 anchor boxes with different scales can be set in the neural network model based on the Yolo structure, and these anchor boxes include anchor boxes for detecting the set symbols; the size of the anchor boxes can be set according to actual needs or adjust. As a set of example data, the anchor box sizes can be: (10, 10), (6, 40), (40, 40), (40, 6), (40, 80), (80, 40), ( 80, 80), (12, 100), (12, 120), (120, 120), (100, 12), (228, 228). The above-mentioned setting symbols include at least one of the following: symbols whose size is within a preset size range, such as "=", "+", "x", and other common symbols; symbols whose aspect ratio is greater than the preset ratio, E.g".",

etc. symbols. The preset ratio may be appropriately set by those skilled in the art according to actual needs, which is not limited in this embodiment of the present invention. Setting the anchor box of the above size in the neural network model based on the Yolo structure can improve the recognition accuracy and positioning accuracy of uncommon formula symbols, such as

"Ω", "─", "∫" and other formula symbols recognition accuracy and positioning accuracy.

As shown in Figure 2, which is a schematic diagram of the above-mentioned neural network model based on the Yolo structure, for an input image, the neural network model based on the Yolo structure maps it to feature maps of multiple scales. Among them, DBL is the basic component of Yolo_v3, which is convolution + BN + Leakyrelu, and BN and Leakyrelu together constitute the smallest component of Yolo_v3. The n in resn in Figure 2 represents a number, including res1, res2,...res8, etc., indicating how many res_units are contained in this res_block, which is a large component of Yolo_v3. Yolo_v3 draws on the residual structure of ResNet and uses this structure. The network structure can be made deeper, and the basic component of res_block is also DBL. The function of concat is tensor splicing, splicing the darknet intermediate layer and the upsampling of a later layer. The operation of splicing is different from that of the residual layer add. The splicing will expand the dimension of the tensor, and add is only Direct addition does not result in a change in tensor dimension.

There will be some symbols with irregular shapes and proportions in the formula symbols, such as the dotted symbol ".", the long horizontal line symbol "------", etc. The conventional Yolo_v3 is not easy to detect. In order to ensure the recognition accuracy of these symbols in the formula , in this embodiment, a low-scale feature extraction channel is set. See Figure 2. Figure 2 shows a schematic diagram of four-scale feature extraction through the neural network model based on the Yolo structure. The dashed box in the figure shows four Feature map, where y4 is the obtained low-scale feature map. For an input preprocessed image, the neural network model based on the Yolo structure can map it to 4-scale feature maps. Based on these feature maps and the aforementioned 12 anchor boxes, symbol recognition and symbol recognition can be performed. The frame detection corresponds to the category information and position information of the formula symbol contained in the formula in the inputted preprocessed picture.

In this embodiment, the multi-scale feature extraction and symbol detection are performed on the preprocessed image through the neural network model based on the Yolo structure, which can avoid the missed detection of unconventional target symbols, thereby avoiding the impact on the translation process in the next stage. , the accuracy of formula recognition is improved, and the robustness and generalization ability of pictures containing formulas in different scenarios are also better.

Step 103: Construct a mixed feature vector based on the category information and location information of the above formula symbols.

Wherein, the position information of the formula symbol obtained in the above steps may include the coordinates of the upper left corner, the lower left corner, the upper right corner and the lower right corner of the symbol frame. A vector can be formed that represents the position information of the formula symbols.

In this embodiment, the category information of the formula symbol can be vectorized to obtain a category vector; now, the category vector and the position information of the formula symbol are spliced to obtain a mixed feature vector. For example, the category information of the formula symbol can be vectorized first to generate a 122-dimensional category vector, and the value of each dimension is between 0-1. Then, the category vector can be spliced with the above position information to generate a 130-dimensional mixed feature vector containing the category information and position information of the above formula symbols. Among them, in the 130-dimensional mixed feature vector, 122 dimensions represent the category information of the formula symbols, and the remaining 8 dimensions represent the position information of the formula symbols. It should be noted that the above-mentioned specific numerical values are only illustrative, and in practical applications, those skilled in the art can set the mixed feature vector as the dimension of other numerical values according to actual needs.

In this embodiment, optionally, the mixed feature vector can also be obtained in other ways, for example, vertically stacking the above two sets of data of the category vector and the position information of the formula symbol, that is, stacking the position information of the formula symbol into the category The next few lines of the vector. The mixed feature vector can also be obtained by overlapping and merging, that is, supplementing one of the two sets of data of the above-mentioned category vector and the position information of the formula symbol into the other set of data.

In this embodiment, since the constructed mixed feature vector includes the position information and category information of the formula symbol, compared to the prior art that only recognizes the formula symbol based on the category information, the positioning of the formula symbol will be faster and more accurate , there will be no situation such as position disorder, and the accuracy rate is higher.

Step 104: Identify and convert formula symbols based on the mixed feature vector to obtain a character string corresponding to the formula contained in the picture.

In this embodiment, the above-mentioned character string may be a latex character string. Specifically, the above-mentioned mixed feature vector may be input into the second neural network model to recognize and convert formula symbols, thereby obtaining the character string corresponding to the formula contained in the picture.

Optionally, the second neural network model may be a sequence-to-sequence model based on an attention mechanism.

As shown in Figure 3, it is a schematic diagram of the sequence-to-sequence model based on the attention mechanism, which is a neural network with an Encoder-Decoder structure added to the attention mechanism. The input sequence of the encoder is the above-mentioned mixed feature vector. If the formula represented by the mixed feature vector is x ² +y ² =125÷5, then the mixed feature vector will pass through the encoder to generate a coding vector, and then enter the Attention module to perform features. Extract, and then output a latex string after passing through the decoder, where the string is a latex string corresponding to the formula x ² +y ² =125÷5.

The traditional method of converting mathematical symbols in formulas into latex strings utilizes a position-based context-free grammar, which is computationally inefficient when many mathematical symbols are recognized. However, the sequence-to-sequence model provided by this embodiment uses the attention mechanism, and the conversion process is regarded as a translation process, and a mixed feature vector based on the category information and position information of the formula symbol is constructed. Since the mixed feature vector contains the formula The category information and position information of symbols are more accurate, clear and complete. Compared with the traditional method, the extracted formula symbol feature information is not lost or biased, so the translated strings have higher accuracy and faster calculation efficiency.

Through this embodiment, the pictures containing the formula are preprocessed, and the formula symbols are detected on the preprocessed pictures to obtain the category information and position information of the formula symbols contained in the above formula; based on the category information and position information of the formula symbols, Construct a mixed feature vector; based on the mixed feature vector, identify and convert the above formula symbols, and obtain the character string corresponding to the formula contained in the above picture. Since the hybrid feature vector constructed in this scheme includes the position information and category information of the formula symbol, the category of the formula symbol can be more accurately determined by the category information, and the position of the formula symbol can be clearly indicated by the position information, so that the The information used for the identification and conversion of formula symbols is more comprehensive and complete, the formula symbols can be identified more accurately, and the accuracy and efficiency of the identification and conversion of formula symbols are higher.

The formula identification method in this embodiment may be executed by any appropriate electronic device with data processing capability, including but not limited to: a server, a mobile terminal (such as a mobile phone, a PAD, etc.), a PC, and the like.

Embodiment 2

As shown in FIG. 4 , a flowchart of a formula recognition method according to Embodiment 2 of the present invention is shown.

Exemplarily, if the formula in the preprocessed picture is x ² +y ² =125÷5, the preprocessed picture is used as the input of the neural network model based on the Yolo structure, and the preprocessed image is processed by the neural network model. The latter image is subjected to multi-scale feature extraction and symbol detection to obtain corresponding feature maps of multiple scales. Then, based on these feature maps, symbol frame detection and symbol recognition are performed to obtain the category information and position information of the formula symbols contained in the formula x ² +y ² =125÷5. As shown in the dotted box in Figure 4, each formula symbol in the preprocessed picture is finally obtained: "x", "2", "+", "y", "2", "=", "1", "2", "5", "÷", "5", and the position of each formula symbol in the preprocessed picture. For example, the coordinates of the four corners of the symbol box in Figure 4. Then, the category information and position information of the formula symbols output by the neural network model based on the Yolo structure can be constructed as a mixed feature vector, and the mixed feature vector can be used as seq2seq, that is, the input of the sequence-to-sequence model based on the attention mechanism, through seq2seq The model recognizes and converts the mixed feature vector, and finally obtains the string corresponding to the formula contained in the picture: "x", "^", "2", "+", "y", "^", "2" , "=", "1", "2", "5", "\div", "5".

Wherein, the process of the preprocessing and the specific processing process of each model can be referred to the description of the relevant part in the foregoing Embodiment 1, which will not be repeated here.

Through this embodiment, when formula recognition is performed, especially when formula recognition includes unconventional symbols such as small target symbols or symbols whose aspect ratio is greater than a preset ratio, the pictures containing formulas can be pre-processed to reduce the difference The influence of scaled pictures containing formulas on the recognition and positioning of formula symbols, so that the accuracy of formula identification and positioning can be improved; then, the preprocessed pictures are detected by the neural network model based on the Yolo structure, and the formula symbols contained in the above formulas are obtained. The category information and position information of the formula symbols can avoid the missed detection of the target symbols, thereby avoiding the impact on the next stage, and improving the accuracy of the identification of formulas containing small target symbols; finally, based on the category information and position information of the above formula symbols information, construct a mixed feature vector; then based on the mixed feature vector, identify and convert the above formula symbols to obtain the character string corresponding to the formula contained in the above picture, because the category information and position information of the detected formula symbols are constructed as mixed The feature vector makes the recognition and conversion of formula symbols more accurate.

The formula identification method in this embodiment may be executed by any appropriate electronic device with data processing capability, including but not limited to: a server, a mobile terminal (such as a mobile phone, a PAD, etc.), a PC, and the like.

It should be noted that the formula recognition scheme of the embodiment of the present invention can be widely used in a variety of scenarios, including but not limited to: performing formula recognition on pictures including pure printed font formulas, and performing formula recognition on pictures including pure handwritten font formulas , formula recognition for images that include both printed font formulas and handwritten font formulas, etc. Therefore, the formula recognition scheme of the embodiment of the present invention can be widely applied to scenarios of various formula pictures, and the compatibility is better.

Embodiment 3

FIG. 5 is a hardware structure of an electronic device in Embodiment 3 of the present invention; as shown in FIG. 5 , the electronic device may include: a processor (processor) 301, a communications interface (Communications Interface) 302, a memory (memory) 303, and a communication bus 304 .

in:

The processor 301 , the communication interface 302 , and the memory 303 communicate with each other through the communication bus 304 .

The communication interface 302 is used to communicate with other electronic devices or servers.

The processor 301 is configured to execute the program 305, and specifically may execute the relevant steps in the above-mentioned embodiments of the formula identification method.

Specifically, the program 305 may include program code including computer operation instructions.

The processor 301 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. One or more processors included in the smart device may be the same type of processors, such as one or more CPUs; or may be different types of processors, such as one or more CPUs and one or more ASICs.

The memory 303 is used to store the program 305 . The memory 303 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

The program 305 can specifically be used to make the processor 301 perform the following operations: preprocess the pictures containing the formula to obtain the preprocessed pictures; perform formula symbol detection on the preprocessed pictures to obtain the categories of the formula symbols contained in the formulas. information and location information; construct a mixed feature vector based on the category information and location information of the formula symbol; identify and convert the formula symbol based on the mixed feature vector to obtain the string corresponding to the formula contained in the picture.

In an optional implementation manner, the program 305 is further configured to cause the processor 301 to: vectorize the category information of the formula symbol to obtain a category vector when constructing the mixed feature vector based on the category information and position information of the formula symbol; The category vector is spliced with the position information of the formula symbol to obtain the mixed feature vector.

In an optional implementation manner, the program 305 is further configured to cause the processor 301 to perform a binarization process on the image containing the formula when obtaining the preprocessed image by preprocessing the image containing the formula to obtain two Value the image; from the binarized image, determine the image area where the formula is located; obtain the preprocessed image according to the image area cut out from the binarized image.

In an optional implementation manner, the program 305 is further configured to cause the processor 301 to obtain, from the binarized picture, a plurality of coordinates with a pixel value of 1 when determining the picture area where the formula is located from the binarized picture. Point; determine the cutting range of the binarized image according to multiple coordinate points, as the image area where the formula is located.

In an optional implementation manner, the program 305 is further configured to cause the processor 301 to obtain a preprocessed picture according to the picture region cut out from the binarized picture: The image area is scaled according to a preset ratio, and then edge-filling processing is performed to obtain a pre-processed image.

In an optional implementation manner, the program 305 is further configured to cause the processor 301 to determine the cutting range of the picture according to multiple coordinate points as the picture area where the formula is located: according to the maximum horizontal width among the multiple coordinate points Coordinates, the minimum abscissa, the maximum ordinate, and the minimum ordinate, the vertex coordinates of the circumscribed quadrilateral of the formula are obtained; the cutting range is determined according to the vertex coordinates, and the binarized image is cut.

In an optional implementation manner, the program 305 is further configured to cause the processor 301 to perform formula symbol detection in the preprocessed picture to obtain the category information and position information of the formula symbol contained in the formula: The picture of is input into the first neural network model for formula symbol detection, and the category information and position information of the formula symbols contained in the formula are obtained.

In an optional implementation manner, the program 305 is further configured to enable the processor 301 to input the preprocessed picture into the first neural network model for formula symbol detection to obtain category information of formula symbols included in the formula and For location information: input the preprocessed image into the first neural network model for formula symbol detection, and perform multi-scale feature extraction and symbol detection on the preprocessed image through the first neural network model to obtain the formula contained in the formula Category information and location information of the symbol.

In an optional implementation manner, the first neural network model is a neural network model based on the Yolo structure; the program 305 is further configured to enable the processor 301 to perform multi-scale features on the preprocessed picture through the first neural network model Extraction and symbol detection to obtain the category information and position information of the formula symbols contained in the formula: perform feature extraction of at least four scales through the neural network model based on the Yolo structure, and obtain the corresponding feature maps of at least four scales, wherein, The feature extraction of at least four scales includes low-scale feature extraction; based on the feature maps of at least four scales, symbol recognition and symbol frame detection are performed, and the category information and position information of the formula symbols contained in the formula are correspondingly obtained.

In an optional implementation manner, 12 anchor boxes with different scales are set in the neural network model, and the anchor boxes include anchor boxes for detecting set symbols.

In an optional implementation manner, the set symbol includes at least one of the following: a symbol whose size is within a preset size range, and a symbol whose aspect ratio is greater than a preset ratio.

In an optional implementation manner, the program 305 is further configured to cause the processor 301 to recognize and convert formula symbols based on the mixed feature vector to obtain the character string corresponding to the formula contained in the picture: input the mixed feature vector into The second neural network model recognizes and converts formula symbols, and obtains a string corresponding to the formula contained in the picture, wherein the second neural network model is a sequence-to-sequence model based on an attention mechanism.

For the specific implementation of each step in the program 305, reference may be made to the corresponding description in the corresponding step in the above-mentioned embodiment of the formula identification method, which is not repeated here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the above-described devices and modules, reference may be made to the corresponding process descriptions in the foregoing method embodiments, which will not be repeated here.

By using the electronic device of this embodiment, the picture containing the formula is preprocessed, and the formula symbol is detected on the preprocessed picture, so as to obtain the category information and position information of the formula symbol contained in the above formula; the category information based on the formula symbol and The position information is used to construct a mixed feature vector; based on the mixed feature vector, the identification and conversion of the above formula symbols are performed, and the character string corresponding to the formula contained in the above picture is obtained. Since the hybrid feature vector constructed in this scheme includes the position information and category information of the formula symbol, the category of the formula symbol can be more accurately determined by the category information, and the position of the formula symbol can be clearly indicated by the position information, so that the The information used for the identification and conversion of formula symbols is more comprehensive and complete, the formula symbols can be identified more accurately, and the accuracy and efficiency of the identification and conversion of formula symbols are higher.

In particular, the processes described above with reference to the flowcharts may be implemented as computer software programs according to embodiments of the present invention. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code configured to perform the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion, and/or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-described functions defined in the methods in the embodiments of the present invention are performed. It should be noted that the computer-readable medium described in this embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the foregoing two. The computer readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access storage media (RAM), read only storage media (ROM), erasable storage media programmable read-only storage media (EPROM or flash memory), optical fiber, portable compact disk read-only storage media (CD-ROM), optical storage media devices, magnetic storage media devices, or any suitable combination of the foregoing. In the embodiments of the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device. In the embodiments of the present invention, however, the computer-readable signal medium may include a data signal in a baseband or propagated as part of a carrier wave, carrying computer-readable program codes therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program configured for use by or in connection with an instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code configured to perform operations of embodiments of the present invention may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, and This includes conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network: including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, through the Internet using an Internet service provider) connect).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions configured to implement the specified functions executable instructions. There are specific sequence relationships in the above specific embodiments, but these sequence relationships are only exemplary, and during specific implementation, these steps may be fewer, more, or the execution order may be adjusted. That is, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The modules involved in the embodiments of the present invention may be implemented in a software manner, and may also be implemented in a hardware manner. The described modules can also be provided in the processor, for example, it can be described as: a processor includes an access module and a transmission module. Among them, the names of these modules do not constitute a limitation on the module itself under certain circumstances.

As another aspect, an embodiment of the present invention further provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, implements the formula identification method described in the foregoing embodiments.

As another aspect, an embodiment of the present invention also provides a computer-readable medium. The computer-readable medium may be included in the apparatus described in the above embodiments; or may exist alone without being assembled into the apparatus. middle. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the device, the device can: preprocess the picture containing the formula to obtain the preprocessed picture; The formula symbol is detected in the picture of the formula symbol, and the category information and position information of the formula symbol contained in the formula are obtained; based on the category information and position information of the formula symbol, a mixed feature vector is constructed; based on the mixed feature vector, the formula symbol is identified and converted, and obtained The string corresponding to the formula contained in the picture.

The expressions "first," "second," "the first," or "the second" as used in various embodiments of the present invention may modify various elements regardless of order and/or importance , but these expressions do not limit the corresponding parts. The above expressions are only configured for the purpose of distinguishing an element from other elements.

The above description is only a preferred embodiment of the present invention and an illustration of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, and should also cover the above-mentioned technical solutions without departing from the above-mentioned inventive concept. Other technical solutions formed by any combination of features or their equivalent features. For example, a technical solution is formed by replacing the above features with the technical features disclosed in the embodiments of the present invention (but not limited to) with similar functions.

Claims (14)

1. a formula identification method, is characterized in that, described method comprises: Preprocess the image containing the formula to obtain the preprocessed image; Performing formula symbol detection on the preprocessed picture to obtain category information and position information of the formula symbol contained in the formula; Construct a mixed feature vector based on the category information and position information of the formula symbol; Based on the mixed feature vector, the identification and conversion of the formula symbol is performed, and the character string corresponding to the formula contained in the picture is obtained. 2. The method according to claim 1, wherein the constructing a mixed feature vector based on the category information and the position information of the formula symbol comprises: Vectorizing the category information of the formula symbol to obtain a category vector; The category vector is spliced with the position information of the formula symbol to obtain a mixed feature vector. 3. The method according to claim 1, wherein the preprocessing is performed on the picture containing the formula to obtain the preprocessed picture, comprising: Binarize the image containing the formula to obtain a binarized image; From the binarized picture, determine the picture area where the formula is located; The preprocessed picture is obtained according to the picture region cut out from the binarized picture. 4. The method according to claim 3, wherein, determining the picture area where the formula is located from the binarized picture, comprising: Obtain a plurality of coordinate points with a pixel value of 1 from the binarized image; The cutting range of the binarized picture is determined according to the plurality of coordinate points as the picture area where the formula is located. 5. The method according to claim 3, wherein the obtaining the preprocessed picture according to the picture region cut out from the binarized picture, comprising: The image area cut out from the binarized image is scaled according to a preset ratio, and then edge-filling processing is performed to obtain a preprocessed image. 6. The method according to claim 4, wherein the determining the cutting range of the picture according to the plurality of coordinate points, as the picture area where the formula is located, comprises: According to the maximum abscissa, the minimum abscissa, the maximum ordinate, and the minimum ordinate in the plurality of coordinate points, the vertex coordinates of the circumscribed quadrilateral of the formula are obtained; The cutting range is determined according to the vertex coordinates, and the binarized image is cut. 7. The method according to claim 1, characterized in that, performing formula symbol detection in the preprocessed picture to obtain category information and position information of the formula symbol contained in the formula, including: Inputting the preprocessed picture into a first neural network model for formula symbol detection, to obtain category information and position information of formula symbols contained in the formula. 8 . The method according to claim 7 , wherein the preprocessed picture is input into a first neural network model for formula symbol detection to obtain category information of formula symbols included in the formula. 9 . and location information, including: Inputting the preprocessed picture into a first neural network model for formula symbol detection, and performing multi-scale feature extraction and symbol detection on the preprocessed picture through the first neural network model to obtain the The category information and position information of the formula symbol contained in the formula. 9. method according to claim 8, is characterized in that, described first neural network model is the neural network model based on Yolo structure; The first neural network model is used to perform multi-scale feature extraction and symbol detection on the preprocessed picture, so as to obtain the category information and position information of the formula symbol contained in the formula, including: Perform feature extraction of at least four scales by using the neural network model based on the Yolo structure, and obtain corresponding feature maps of at least four scales, wherein the feature extraction of at least four scales includes low-scale feature extraction; Based on the feature maps of the at least four scales, symbol recognition and symbol frame detection are performed, and category information and position information of the formula symbols contained in the formula are correspondingly obtained. 10 . The method according to claim 9 , wherein 12 anchor boxes with different scales are set in the neural network model, and the anchor boxes include anchor boxes for detecting set symbols. 11 . 11 . The method according to claim 10 , wherein the set symbol comprises at least one of the following: a symbol whose size is within a preset size range, and a symbol whose aspect ratio is greater than a preset ratio. 12 . 12 . The method according to claim 1 , wherein the identification and conversion of the formula symbol is performed based on the mixed feature vector, and the character string corresponding to the formula contained in the picture is obtained, comprising: 12 . : Input the mixed feature vector into a second neural network model to identify and convert the formula symbols, and obtain the character string corresponding to the formula contained in the picture, wherein the second neural network model is based on attention A sequence-to-sequence model of the mechanism. 13. An electronic device, characterized in that the device comprises: one or more processors; A computer-readable medium configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the formula recognition method according to any one of claims 1-12. 14. A computer-readable medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the formula recognition method according to any one of claims 1-12 is implemented.

Images