CN116229490A - Layout analysis method, device, equipment and medium of graphic neural network - Google Patents

Abstract

The invention discloses a layout analysis method, device, equipment and medium of a graph neural network. A model training method comprising: inputting the text image sample to a text detection module to obtain a text detection sample box; creating a first graph to be analyzed according to the text detection sample box, and determining node characteristics and edge characteristics of the first graph to be analyzed; inputting the node characteristics and the edge characteristics into an original graph neural network model to perform classification training of node types and edge types, so as to obtain a target graph neural network model; the target graph neural network model is used for carrying out layout analysis on texts in the text images. The technical scheme of the embodiment of the invention can improve the layout analysis performance and reduce the application limitation.

Description

Layout analysis method, device, equipment and medium of a graph neural network

technical field

The invention relates to the technical field of layout analysis, in particular to a layout analysis method, device, equipment and medium of a graph neural network.

Background technique

Layout analysis tasks are widely used in real life. For example, when an image format document is converted to a text format, accurate layout analysis results are required, and then follow-up tasks such as document comparison tasks and information extraction tasks are performed based on the layout analysis results.

At present, one of the commonly used layout analysis schemes is the target detection method, and the other is the method of using the TransformerEncoder series. When performing layout analysis based on the target detection method, the obtained detection frame is not very accurate, and the overlapping problem of the detection frame is also difficult to solve, and post-processing rules are needed to solve the matching relationship with the text, resulting in poor layout analysis effect. However, the Transformer Encoder series has a maximum length limit on the number of characters. In reality, it is common for the number of Tokens on a page to exceed 512 or even 1024. However, the Transformer Encoder series has a large model and high computing resources, and needs to be adapted to a specific tokenizer. The language is highly correlated, and different languages cannot be transferred directly. Moreover, most of the public datasets in this field are in English. For example, DocBank and PubLayNet are datasets with hundreds of thousands of data, and the number of Chinese or other languages is very small. , even if there is, there are cases where the labeling quality is not good.

Contents of the invention

The invention provides a graph neural network layout analysis method, device, equipment and medium to solve the problems of poor analysis effect and large application limitation of the existing layout analysis method.

According to an aspect of the present invention, a kind of model training method is provided, comprising:

The text image sample is input to the text detection module to obtain a text detection sample box;

Create the first graph to be analyzed according to the text detection sample frame, and determine the node features and edge features of the first graph to be analyzed; wherein, the nodes in the first graph to be analyzed correspond to the text detection sample frame;

Input the node features and edge features into the original graph neural network model for classification training of node types and edge types to obtain the target graph neural network model;

Object graph neural network model for layout analysis of text in text images.

According to another aspect of the present invention, a layout analysis method is provided, including: acquiring a text image to be analyzed, inputting the text image to be analyzed into a text detection module, and obtaining a text detection frame;

Create the second graph to be analyzed according to the text detection frame, and determine the target node feature and the target edge feature of the second graph to be analyzed; wherein, the target node in the second graph to be analyzed corresponds to the text detection frame;

Input the target node features and target edge features into the target graph neural network model to obtain the target node type and target edge type;

Wherein, the target graph neural network model is a model trained by the model training method in any embodiment.

According to another aspect of the present invention, a model training device is provided, comprising:

The text detection sample frame acquisition module is used to input the text image sample to the text detection module to obtain the text detection sample frame;

The first feature determination module is used to create the first graph to be analyzed according to the text detection sample frame, and determine the node features and edge features of the first graph to be analyzed; wherein, the nodes in the first graph to be analyzed correspond to the text detection sample frame;

The target graph neural network model determination module is used to input node features and edge features into the original graph neural network model to perform classification training of node types and edge types to obtain the target graph neural network model;

Object graph neural network model for layout analysis of text in text images.

According to another aspect of the present invention, a layout analysis device is provided, comprising:

The text detection frame acquisition module is used to obtain the text image to be analyzed, and input the text image to be analyzed to the text detection module to obtain the text detection frame;

The second feature determination module is used to create a second graph to be analyzed according to the text detection frame, and determine the target node features and target edge features of the second graph to be analyzed; wherein, the target node in the second graph to be analyzed corresponds to the text detection frame ;

The classification module is used for inputting target node features and target edge features into the target graph neural network model to obtain target node types and target edge types;

Wherein, the target graph neural network model is a model trained by the model training method in any embodiment.

According to another aspect of the present invention, an electronic device is provided, and the electronic device includes:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor, so that the at least one processor can execute the method described in any embodiment of the present invention. Model training methods, or layout analysis methods.

According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement any of the embodiments of the present invention when executed. model training methods, or layout analysis methods.

In the technical solution of the embodiment of the present invention, the text detection sample frame is obtained by inputting the text image sample into the text detection module, thereby creating the first graph to be analyzed according to the text detection sample frame, and determining the node characteristics and edges of the first graph to be analyzed features, and then input the node features and edge features into the original graph neural network model for classification training of node types and edge types, and obtain the target graph neural network model, and realize the layout analysis of the text in the text image by the target graph neural network model . Since the node type and edge type can reflect the layout of the text detection sample frame in the text image sample, the classification training of the node type and edge type in the original graph neural network model through the extracted node features and edge features can make the final obtained The target graph neural network model has a better layout analysis effect, which solves the problems of poor analysis effect and large application limitations of existing layout analysis methods, can improve layout analysis performance, and reduce application limitations.

It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will be easily understood from the following description.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

FIG. 1 is a flowchart of a model training method provided by Embodiment 1 of the present invention;

FIG. 2 is a flow chart of a model training method provided in Embodiment 2 of the present invention;

Fig. 3 is a flowchart of a layout analysis method provided by Embodiment 3 of the present invention;

Fig. 4 is an algorithmic logic diagram of a layout analysis method provided by Embodiment 3 of the present invention;

FIG. 5 is a schematic structural diagram of a model training device provided in Embodiment 4 of the present invention;

FIG. 6 is a schematic structural diagram of a layout analysis device provided in Embodiment 5 of the present invention;

FIG. 7 shows a schematic structural diagram of an electronic device that can be used to implement an embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

Embodiment one

Fig. 1 is a flow chart of a model training method provided by Embodiment 1 of the present invention. This embodiment is applicable to the case of performing efficient and accurate layout analysis on text images. This method can be executed by a model training device. The model training The device can be implemented in the form of hardware and/or software, and the model training device can be configured in electronic equipment. As shown in Figure 1, the method includes:

S110. Input the text image sample into the text detection module to obtain a text detection sample box.

Wherein, the text image sample may be an image sample used for layout analysis training. Exemplarily, the text image samples may include but not limited to bills, newspapers, and magazines. The text detection module can be any model with text detection function. The text detection module can be used to detect text regions in images. The text detection sample frame may be a frame of a text area in a text image sample recognized by the text detection module.

In the embodiment of the present invention, the text image sample can be obtained according to the layout analysis requirements, and then the text image sample can be input into the text detection module, so that the text region in the text image sample can be identified according to the text detection module, and the text detection sample box can be obtained.

S120. Create a first graph to be analyzed according to the text detection sample frame, and determine node features and edge features of the first graph to be analyzed.

Wherein, the first graph to be analyzed may be a graph drawn according to the correlation between each text detection sample frame. The nodes in the first graph to be analyzed correspond to the text detection sample frame, and the edges connecting the nodes in the first graph to be analyzed correspond to the positional relationship and contextual relationship of the connected nodes. The node features may be the features of the nodes constituting the first graph to be analyzed. The edge features may be the features of the edges that constitute the nodes in the first graph to be analyzed.

In the embodiment of the present invention, the correlation between the text detection sample frames can be determined according to the position of the text detection sample frame in the text image sample and the adjacent relationship between the text detection sample frames, so that the correlation between the text detection sample frames Draw the first graph to be analyzed systematically, and then determine the node features of the first graph to be analyzed and the edge features of the edges between nodes in the first graph to be analyzed, and further obtain the label data corresponding to the node features of the first graph to be analyzed and the Label data corresponding to edge features of the first graph to be analyzed.

S130. Input node features and edge features into the original graph neural network model to perform classification training of node types and edge types to obtain a target graph neural network model.

Wherein, the original graph neural network model may be any known graph neural network model. An object graph neural network model that can be used for layout analysis of text in text images. The node type can be used to represent the category to which the text detection sample box corresponding to the node belongs. The category to which the text detection sample box belongs can be understood as the element type that constitutes the layout, such as title, paragraph, header or footer, etc. The edge type can be used to indicate the belonging relationship of the edge-connected nodes in the first graph to be analyzed, and can assist in determining whether the text detection sample frames corresponding to the nodes belong to the same paragraph.

In the embodiment of the present invention, node features, edge features, label data corresponding to node features, and label data corresponding to edge features can be input into the original graph neural network model, so that based on node features and node features corresponding Label data, classify and train the node types of the original graph neural network model, obtain updated node features, and then use the updated node features, input edge features, and label data corresponding to edge features to train the original graph neural network model The edge type classification training is carried out to obtain the target graph neural network model, so that the layout analysis of the text in the text image can be performed through the target graph neural network model.

In the technical solution of the embodiment of the present invention, the text detection sample frame is obtained by inputting the text image sample into the text detection module, thereby creating the first graph to be analyzed according to the text detection sample frame, and determining the node characteristics and edges of the first graph to be analyzed features, and then input the node features and edge features into the original graph neural network model for classification training of node types and edge types, and obtain the target graph neural network model, and realize the layout analysis of the text in the text image by the target graph neural network model . Since the node type and edge type can reflect the layout of the text detection sample frame in the text image sample, the classification training of the node type and edge type in the original graph neural network model through the extracted node features and edge features can make the final obtained The target graph neural network model has a better layout analysis effect, which solves the problems of poor analysis effect and large application limitations of existing layout analysis methods, can improve layout analysis performance, and reduce application limitations.

Embodiment two

Fig. 2 is a flow chart of a model training method provided by Embodiment 2 of the present invention. This embodiment is embodied on the basis of the above-mentioned embodiment, and it provides that node features and edge features are input into the original graph neural network model. A specific optional implementation manner for performing classification training of node types and edge types. As shown in Figure 2, the method includes:

S210. Input the text image sample into the text detection module to obtain a text detection sample frame.

S220. Create a first graph to be analyzed according to the text detection sample frame, and determine node features and edge features of the first graph to be analyzed.

In an optional embodiment of the present invention, creating the first graph to be analyzed according to the text detection sample frame may include: obtaining a preset composition rule; creating the first graph to be analyzed according to the preset composition rule and the text detection sample frame; The preset composition rule may include a full connection rule, a first adjacent text frame border building rule, or a second adjacent text frame border building rule.

Wherein, the preset composition rule may be a pre-established composition rule of the connected graph. The full join rule can be a rule that creates a connection edge between any two nodes in the connected graph. The first adjacent text box edge building rule may be a rule for creating connecting edges for adjacent nodes using Beta-skeleton. The second adjacent text box edge building rule may be a rule for creating connection edges between K nodes with the closest distance using KNN (K-Nearest Neighbor, nearest neighbor node algorithm).

In the embodiment of the present invention, according to the mapping needs of the first graph to be analyzed, the rules of building edges from existing connected graph nodes (such as full connection rules, first adjacent text box edge building rules or second adjacent Text frame border building rules, etc.) to determine the preset composition rules, and then set the nodes corresponding to the text detection sample frame, and connect the nodes corresponding to the text detection sample frame through the preset composition rules to obtain the first graph to be analyzed.

S230. Input node features into the original graph neural network model, determine node update features of each node in the first graph to be analyzed, and perform node type training on the original graph neural network model according to the node update features of each node.

Wherein, the node update feature may be a new feature of the node determined by the neural network model of the original graph according to the intersection nodes in the first graph to be analyzed. Exemplarily, assuming that there is an edge between node 1 and node 2, and an edge between node 1 and node 3, then node 1 is an intersection node of node 2 and node 3.

In the embodiment of the present invention, after the node features are input into the original graph neural network model, the original graph neural network model can be used to update the node features of the first graph to be analyzed according to the node features of the nodes associated with each intersection node , to obtain the node update features of each node in the first graph to be analyzed, so as to perform node type identification training on the original graph neural network model according to the node update features of each node and the label data corresponding to the node features.

In an optional embodiment of the present invention, determining the node features and edge features of the first graph to be analyzed may include: taking the text frame coordinates and text frame image features of each node corresponding to the text detection sample frame as the first graph to be analyzed The node feature of graph; the node feature of each edge connection node in the first graph to be analyzed, the relative distance between the text box coordinates of each edge match, the text box aspect ratio of each edge match and the prediction type of edge corresponding node, As the edge feature of the first graph to be analyzed.

Wherein, the coordinates of the text frame may be the coordinates of the text detection sample frame in the text image sample. The coordinates of the text box may include but not limited to the coordinates of the upper left corner and the lower right corner of the text detection sample box. The text frame image feature may be a partial image of the text detection sample frame corresponding to the text image sample. The aspect ratio of the text box may be a side length ratio of the long side and the wide side of the text detection sample box.

In the embodiment of the present invention, according to the position of the text detection sample frame corresponding to each node in the first graph to be analyzed in the text image sample, the text frame coordinates and text frame image features of the text detection sample frame corresponding to each node can be obtained, and then the Each node corresponds to the text box coordinates and text box image features of the text detection sample box, as the node features of the first graph to be analyzed, and then determine the node features of the connected nodes of each edge in the first graph to be analyzed, and the text frame coordinates matched by each edge The relative distance between (that is, the relative distance between the text box coordinates of the text detection sample box corresponding to the connection nodes of each side) and the aspect ratio of the text box matched by each side (that is, the long side of the text detection sample box corresponding to the connection node of each side ratio of the side length to the wide side), and then predict the node types of the nodes connected by each side respectively, and obtain the predicted type of the nodes connected by each side, so that the node characteristics of the nodes connected by each side in the first graph to be analyzed, and each side The relative distance between the coordinates of the matched text boxes, the aspect ratio of the matched text boxes of each edge, and the predicted type of the corresponding node of each edge are used as edge features of the first graph to be analyzed.

S240. Perform edge type classification training on the original graph neural network model according to the node update features and edge features of each node to obtain a target graph neural network model.

In the embodiment of the present invention, according to the node update feature of each node, the edge feature and the label data corresponding to the edge feature, the original graph neural network model can be trained for the classification training of the two classifications of edge types to obtain the target graph neural network model. To perform layout analysis of text in text images by target graph neural network model.

In a specific example, the document image (text image sample) can be input to the text detection module to obtain the text detection sample box, so that the text detection sample box is used as a node, and according to the preset composition rules (such as the full connection rule, the first Adjacent text frame border building rules or any one of the second adjacent text frame border building rules) to construct the first graph to be analyzed.

After obtaining the first graph to be analyzed, the node features and edge features of the first graph to be analyzed can be constructed. Specifically, the text box coordinates of the text detection sample boxes corresponding to each node in the first graph to be analyzed can be determined. If the text detection sample If the frame is not a rectangle, the number of coordinate points of the text frame coordinates can be extended. After the document image is input to the convolutional neural network, the image features of the document image are obtained, and then according to the image features of the document image and the coordinates of the text box, the local image features of the text detection sample box corresponding to the node are obtained. Since each edge connects two nodes, it indicates whether two text detection sample boxes are related, that is, whether they belong to the same instance, so edge features can be helpful for layout analysis. Further, the node features and edge features are input into the original graph neural network model for node type and edge type classification training to obtain the target graph neural network model.

Among them, the original graph neural network model may include, but not limited to, GCN, GAT, GAT-v2, DGCNN, and GravNet. The original graph neural network model can be stacked in multiple layers and has two classification heads. One classification head of the original graph neural network model is used for classification training of node types, and the other classification head can be used for classification training of edge types. Construct the final original graph neural network model based on the previous parts, and adjust the selection of each part on different data sets to achieve the best configuration.

In the technical solution of the embodiment of the present invention, the text detection sample frame is obtained by inputting the text image sample into the text detection module, thereby creating the first graph to be analyzed according to the text detection sample frame, and determining the node characteristics and edges of the first graph to be analyzed feature, and input the node features into the original graph neural network model, determine the node update features of each node in the first graph to be analyzed, and perform node type training on the original graph neural network model according to the node update features of each node, and then according to The node update characteristics and edge types of each node, the edge type classification training is performed on the original graph neural network model, and the target graph neural network model is obtained, and the target graph neural network model is implemented to perform layout analysis on the text in the text image. Since the node type and edge type can reflect the layout of the text detection sample frame in the text image sample, the classification training of the node type and edge type in the original graph neural network model through the extracted node features and edge features can make the final obtained The target graph neural network model has a better layout analysis effect, which solves the problems of poor analysis effect and large application limitations of existing layout analysis methods, can improve layout analysis performance, and reduce application limitations.

This scheme uses the original graph neural network model as the core of the algorithm, and the amount of calculation is small. Reasonably setting the node features and edge features in the first graph to be analyzed can achieve a good algorithm effect, and this scheme does not use text information, which has nothing to do with language , you can directly use a large number of public English data sets without being limited to a specific language, there is no limit on the number of texts on a single page, and there is no need to match text boxes and perform complex post-processing on overlapping target detection boxes. Compared with the existing solution, this solution has great advantages in terms of overall performance (parameters, video memory usage, inference time, and accuracy), and is suitable for industrialization.

Embodiment three

Fig. 3 is a flow chart of a layout analysis method provided by Embodiment 3 of the present invention, the method can be executed by a layout analysis device, the layout analysis device can be implemented in the form of hardware and/or software, and the layout analysis device can be configured in electronic equipment. As shown in Figure 3, the method includes:

S310. Acquire a text image to be analyzed, and input the text image to be analyzed into a text detection module to obtain a text detection frame.

Wherein, the text image to be analyzed may be an image required for layout analysis. The text detection frame may be a frame of a text area identified by the text detection module in the text image to be analyzed.

In the embodiment of the present invention, the acquired text image to be analyzed can be input to the text detection module, and then the text region in the text image to be analyzed can be identified according to the text detection module to obtain the text detection frame.

S320. Create a second graph to be analyzed according to the text detection frame, and determine target node features and target edge features of the second graph to be analyzed.

Wherein, the second graph to be analyzed may be a graph drawn according to the correlation between text detection frames. The feature of the target node may be the feature of the nodes constituting the second graph to be analyzed. The target edge feature may be a feature of an edge between nodes constituting the second graph to be analyzed. The target node in the second graph to be analyzed corresponds to the text detection frame, and the target edge connecting the target node in the second graph to be analyzed corresponds to the positional relationship and context relationship of the connected target node. The target node may be a node corresponding to the text detection frame in the second graph to be analyzed. The target edge may be an edge connecting the target nodes in the second graph to be analyzed.

In the embodiment of the present invention, the correlation between the text detection frames can be determined according to the position of the text detection frames in the text image to be analyzed and the adjacent relationship of the text detection frames, so that the second can be drawn according to the correlation between the text detection frames The second graph to be analyzed, and then determine the target node features and target edge features of the second graph to be analyzed.

S330. Input the target node features and target edge features into the target graph neural network model to obtain target node types and target edge types.

Wherein, the target graph neural network model may be a model trained by the model training method in any embodiment of the present invention. The target node type can be used to represent the category to which the text detection box belongs. The target node type is the same kind as the category to which the text detection box belongs. The target edge type can be used to indicate whether the edge-connected nodes in the second graph to be analyzed belong to the same instance, and an instance can be understood as a paragraph.

In the embodiment of the present invention, the target node features and target edge features can be input to the target graph neural network model to obtain the target node type and target edge type, so as to determine the text image to be analyzed according to the target node type and target edge type. The layout position of each text detection frame pair realizes the layout analysis of the text image to be analyzed.

In an optional embodiment of the present invention, after obtaining the target node type and the target edge type, it may also include: determining the layout element type matching each target node according to the target node type of each target node; According to the target edge type of each target edge, determine the instance affiliation relationship matching each target edge; according to the layout element type matching each target node and the instance affiliation relationship matching each target edge, determine the layout analysis result of the text image to be analyzed.

Wherein, the layout element type may be the type of elements constituting the layout. The instance belonging relationship can be used to describe whether the text detection boxes belong to the same instance, and an instance can be understood as a paragraph. The layout analysis result may be an analysis result of the text layout corresponding to the text image to be analyzed.

In the embodiment of the present invention, the target node type of each target node can be used as the layout element type matching each target node, and according to the target edge type of each target edge, determine the belonging relationship of the instance matching each target edge, and then According to the type of layout elements matched by each target node and the belonging relationship of instances matched with each target edge, the layout of the text in the text image to be analyzed is determined, and the layout analysis result of the text image to be analyzed is obtained, which can further be based on the layout analysis of the text image to be analyzed As a result, the text in the text image to be analyzed is laid out.

FIG. 4 is an algorithm logic diagram of a layout analysis method provided by Embodiment 3 of the present invention. As shown in Figure 4, the text image to be analyzed is first obtained, and then the text image to be analyzed is input to the text detection module, and the text detection frame is obtained based on the text detection module, and the connected graph is constructed according to the text detection frame, that is, the second For the graph to be analyzed, the target node features and target edge features of the second graph to be analyzed are further constructed, and the target node features are updated based on the target graph neural network model to obtain the updated target node features, so that based on the updated target node features Classify the node type of the target node, and then classify the target edge features based on the updated target node features to obtain the target edge type. Finally, the layout analysis result can be determined according to the node type with the target node and the target edge type.

When comparing documents, it is necessary to analyze the layout of the documents first, and then compare them one by one. The present invention uses the text detection frame obtained after the text image to be analyzed and OCR (Optical Character Recognition, Optical Character Recognition), predicts the category of each text detection frame (ie title, paragraph, header or footer, etc.), and predicts the adjacent Whether the text detection boxes belong to the same instance (that is, both text box A and text box B belong to the same paragraph). If the target detection algorithm is used, since the input is only an image, and there is no prior knowledge of the text detection frame, if the detection frame is slightly wider or slightly narrower than the correct frame, it may cause a paragraph to have one more line or one line less , which is fatal for comparison tasks. In terms of algorithm design, the algorithm using the target graph neural network model greatly reduces the probability of error. Even if there is an error sample, it can be quickly fitted by simply adding it to the training data set, while the target detection algorithm is difficult to force Its prediction box is 100% accurate.

When parsing documents, for double-column or even multi-column documents, it is necessary to analyze the layout of the document first to obtain the correct reading order. The present invention judges the relationship between adjacent text detection frames, and can quickly distinguish the gap between double columns (or multiple columns). If you use the Transformer Encoder series of algorithms, in addition to the maximum limit on the number of words on a single page (such as common newspapers, it is normal for the number of tokens to exceed 512), when judging the relationship between tokens, the general practice is to fully join and calculate two pairs The score, which causes unnecessary resource consumption, also increases the difficulty of algorithm training.

In the technical solution of the embodiment of the present invention, by acquiring the text image to be analyzed, inputting the text image to be analyzed into the text detection module to obtain the text detection frame, thereby creating the second image to be analyzed according to the text detection frame, and determining the second image to be analyzed The target node features and target edge features, and then input the target node features and target edge features into the target graph neural network model to obtain the target node type and target edge type. Since the target node type and target edge type can reflect the layout of the text detection frame in the text image to be analyzed, determining the target node features and target edge features through the target graph neural network model can improve the layout analysis effect, and there is no limit to the number of words detected Constraints solve the problems of poor analysis results and large application limitations of existing layout analysis methods, which can improve layout analysis performance and reduce application limitations.

Embodiment four

FIG. 5 is a schematic structural diagram of a model training device provided in Embodiment 4 of the present invention. As shown in Figure 5, the device includes: a text detection sample frame acquisition module 410, a first feature determination module 420 and a target graph neural network model determination module 430;

The text detection sample frame acquisition module 410 is used to input the text image sample to the text detection module to obtain the text detection sample frame;

The first feature determination module 420 is used to create the first graph to be analyzed according to the text detection sample frame, and determine the node features and edge features of the first graph to be analyzed; wherein, the nodes in the first graph to be analyzed correspond to the text detection sample frame ;

The target graph neural network model determination module 430 is used to input node features and edge features into the original graph neural network model to perform classification training of node types and edge types to obtain the target graph neural network model;

Object graph neural network model for layout analysis of text in text images.

In the technical solution of the embodiment of the present invention, the text detection sample frame is obtained by inputting the text image sample into the text detection module, thereby creating the first graph to be analyzed according to the text detection sample frame, and determining the node characteristics and edges of the first graph to be analyzed features, and then input the node features and edge features into the original graph neural network model for classification training of node types and edge types, and obtain the target graph neural network model, and realize the layout analysis of the text in the text image by the target graph neural network model . Since the node type and edge type can reflect the layout of the text detection sample frame in the text image sample, the classification training of the node type and edge type in the original graph neural network model through the extracted node features and edge features can make the final obtained The target graph neural network model has a better layout analysis effect, which solves the problems of poor analysis effect and large application limitations of existing layout analysis methods, can improve layout analysis performance, and reduce application limitations.

Optionally, the first feature determination module 420 includes a first image-to-be-analyzed creating unit, configured to acquire preset composition rules; and create the first image-to-be-analyzed according to the preset composition rules and the text detection sample frame ; The preset composition rules include the full connection rule, the first adjacent text box border building rule or the second adjacent text box border building rule.

Optionally, the first feature determination module 420 includes a first feature determination unit, configured to use the text frame coordinates and text frame image features of each node corresponding to the text detection sample frame as the node features of the first graph to be analyzed; The node characteristics of the connected nodes of each edge in the first graph to be analyzed, the relative distance between the coordinates of the text boxes matched by each edge, the aspect ratio of the text box matched by each edge, and the prediction type of the corresponding node of each edge, as the Edge features of the first graph to be analyzed.

Optionally, the target graph neural network model determination module 430 is specifically configured to input the node features into the original graph neural network model, determine the node update features of each node in the first graph to be analyzed, and according to The node update feature of each node performs node type training on the original graph neural network model; according to the node update feature of each node and the edge feature, performs edge type classification training on the original graph neural network model .

The model training device provided in the embodiment of the present invention can execute the model training method provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

Embodiment five

FIG. 6 is a schematic structural diagram of a layout analysis device provided in Embodiment 5 of the present invention. As shown in Figure 6, the device includes: a text detection frame acquisition module 510, a second feature determination module 520 and a classification module 530,

The text detection frame acquisition module 510 is used to obtain the text image to be analyzed, and input the text image to be analyzed into the text detection module to obtain the text detection frame;

The second feature determination module 520 is configured to create a second graph to be analyzed according to the text detection frame, and determine the target node features and target edge features of the second graph to be analyzed; wherein, the second graph to be analyzed The target node of corresponds to the text detection frame;

The classification module 530 is configured to input the target node features and the target edge features into the target graph neural network model to obtain target node types and target edge types.

In the technical solution of the embodiment of the present invention, by acquiring the text image to be analyzed, inputting the text image to be analyzed into the text detection module to obtain the text detection frame, thereby creating the second image to be analyzed according to the text detection frame, and determining the second image to be analyzed The target node features and target edge features, and then input the target node features and target edge features into the target graph neural network model to obtain the target node type and target edge type. Since the target node type and target edge type can reflect the layout of the text detection frame in the text image to be analyzed, determining the target node features and target edge features through the target graph neural network model can improve the layout analysis effect, and there is no limit to the number of words detected Constraints solve the problems of poor analysis results and large application limitations of existing layout analysis methods, which can improve layout analysis performance and reduce application limitations.

Optionally, the layout analysis device further includes a layout analysis result determination module, configured to use the target node type of each target node as a layout element type matching each target node; according to the target edge type of each target edge, Determine the instance affiliation that matches each target edge; determine the layout analysis result of the text image to be analyzed according to the layout element type that matches each target node and the instance affiliation that matches each target edge.

The layout analysis device provided by the embodiment of the present invention can execute the layout analysis method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

Embodiment six

FIG. 7 shows a schematic structural diagram of an electronic device that can be used to implement an embodiment of the present invention. Electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (eg, helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the inventions described and/or claimed herein.

As shown in FIG. 7 , the electronic device 10 includes at least one processor 11, and a memory connected in communication with the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores There is a computer program executable by at least one processor, and the processor 11 can operate according to a computer program stored in a read-only memory (ROM) 12 or loaded from a storage unit 18 into a random access memory (RAM) 13. Various appropriate actions and processes are performed. In the RAM 13, various programs and data necessary for the operation of the electronic device 10 are also stored. The processor 11 , ROM 12 , and RAM 13 are connected to each other through a bus 14 . An input/output (I/O) interface 15 is also connected to the bus 14 .

Multiple components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

Processor 11 may be various general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes various methods and processes described above, such as a model training method, or a layout analysis method.

In some embodiments, the model training method, or the layout analysis method can be implemented as a computer program, which is tangibly embodied in a computer-readable storage medium, such as the storage unit 18 . In some embodiments, part or all of the computer program may be loaded and/or installed on the electronic device 10 via the ROM 12 and/or the communication unit 19 . When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the model training method or the layout analysis method described above can be executed. Alternatively, in other embodiments, the processor 11 may be configured in any other appropriate way (for example, by means of firmware) to execute a model training method or a layout analysis method.

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips Implemented in a system of systems (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor Can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.

Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, so that the computer program causes the functions/operations specified in the flowcharts and/or block diagrams to be implemented when executed by the processor. A computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present invention, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus or device. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

In order to provide interaction with the user, the systems and techniques described herein can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display)) for displaying information to the user. monitor); and a keyboard and pointing device (eg, a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.

The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or a cloud host. It is a host product in the cloud computing service system to solve the problems of difficult management and weak business expansion in traditional physical hosts and VPS services. defect.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, each step described in the present invention may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution of the present invention can be achieved, there is no limitation herein.

The above specific implementation methods do not constitute a limitation to the protection scope of the present invention. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A method of model training, comprising:

inputting the text image sample to a text detection module to obtain a text detection sample box;

creating a first graph to be analyzed according to the text detection sample box, and determining node characteristics and edge characteristics of the first graph to be analyzed; the nodes in the first graph to be analyzed correspond to the text detection sample box;

inputting the node characteristics and the edge characteristics into an original graph neural network model to perform classification training of node types and edge types, so as to obtain a target graph neural network model;

The target graph neural network model is used for carrying out layout analysis on texts in the text images.

2. The method of claim 1, wherein creating a first graph to be analyzed from the text detection sample box comprises:

acquiring a preset composition rule;

creating the first diagram to be analyzed according to the preset composition rule and the text detection sample box;

the preset composition rule comprises a full-connection rule, a first adjacent text box edge building rule or a second adjacent text box edge building rule.

3. The method of claim 1, wherein the determining node features and edge features of the first graph to be analyzed comprises:

using text box coordinates and text box image characteristics of the text detection sample boxes corresponding to the nodes as node characteristics of the first to-be-analyzed graph;

and taking the node characteristics of the connecting nodes of the edges, the relative distance between the coordinates of the text boxes matched with the edges, the length-width ratio of the text boxes matched with the edges and the prediction type of the corresponding nodes of the edges in the first graph to be analyzed as the edge characteristics of the first graph to be analyzed.

4. The method according to claim 1, wherein inputting the node features and the edge features into an original graph neural network model for classification training of node types and edge types comprises:

Inputting the node characteristics into the original graph neural network model, determining node updating characteristics of each node in the first graph to be analyzed, and performing node type training on the original graph neural network model according to the node updating characteristics of each node;

and according to the node updating characteristics of each node and the edge characteristics, carrying out edge type classification training on the original graph neural network model.

5. A layout analysis method, comprising:

acquiring a text image to be analyzed, and inputting the text image to be analyzed into a text detection module to obtain a text detection box;

creating a second graph to be analyzed according to the text detection box, and determining target node characteristics and target edge characteristics of the second graph to be analyzed; the target node in the second graph to be analyzed corresponds to the text detection box;

inputting the target node characteristics and the target edge characteristics into a target graph neural network model to obtain a target node type and a target edge type;

wherein the target graph neural network model is a model obtained by training the model training method according to any one of claims 1 to 4.

6. The method of claim 5, further comprising, after the obtaining the target node type and the target edge type:

taking the target node type of each target node as the layout element type matched with each target node;

determining an instance affiliation matched with each target edge according to the target edge type of each target edge;

and determining the layout analysis result of the text image to be analyzed according to the layout element type matched with each target node and the generic relationship of the instance matched with each target edge.

7. A model training device, comprising:

the text detection sample box acquisition module is used for inputting the text image sample into the text detection module to obtain a text detection sample box;

the first feature determining module is used for creating a first graph to be analyzed according to the text detection sample box and determining node features and edge features of the first graph to be analyzed; the nodes in the first graph to be analyzed correspond to the text detection sample box;

the target graph neural network model determining module is used for inputting the node characteristics and the edge characteristics into an original graph neural network model to perform classification training of node types and edge types, so as to obtain a target graph neural network model;

The target graph neural network model is used for carrying out layout analysis on texts in the text images.

8. A layout analysis apparatus, comprising:

the text detection box acquisition module is used for acquiring a text image to be analyzed, and inputting the text image to be analyzed into the text detection module to obtain a text detection box;

the second feature determining module is used for creating a second graph to be analyzed according to the text detection box and determining target node features and target edge features of the second graph to be analyzed; the target node in the second graph to be analyzed corresponds to the text detection box;

and the classification module is used for inputting the target node characteristics and the target edge characteristics into a target graph neural network model to obtain a target node type and a target edge type.

9. An electronic device, the electronic device comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the model training method of any one of claims 1-4 or to perform the layout analysis method of any one of claims 5-6.

10. A computer readable storage medium storing computer instructions for causing a processor to implement the model training method of any one of claims 1-4 or the layout analysis method of any one of claims 5-6 when executed.

Images