WO2023077995A1

WO2023077995A1 - Information extraction method and apparatus, and device, medium and product

Info

Publication number: WO2023077995A1
Application number: PCT/CN2022/121551
Authority: WO
Inventors: 范湉湉; 黄灿; 王长虎
Original assignee: 北京有竹居网络技术有限公司
Priority date: 2021-11-04
Filing date: 2022-09-27
Publication date: 2023-05-11
Also published as: CN114037985A

Abstract

Provided in the present application are an information extraction method and apparatus, and a device, a medium and a product. The method comprises: an electronic device performing text detection on an image to obtain a text area that comprises a plurality of text lines; then constructing a graph network model by taking each text line in the text area as a node; classifying the nodes in the graph network model by means of a node classification model; classifying edges in the graph network model by means of an edge classification model; and then, obtaining a key value pair in the image according to a node classification result and an edge classification result. Therefore, a feature of a text line itself and a feature of an associated text line can be comprehensively taken into consideration, thereby realizing the accurate extraction of information.

Description

Information extraction method, device, equipment, medium and product

This application claims the priority of the Chinese patent application with the application number 202111300845.9 and the application title "Information extraction method, device, equipment, medium and product" filed with the China Patent Office on November 04, 2021, the entire contents of which are incorporated by reference in this application.

technical field

The present application relates to the field of computer technology, and in particular to an information extraction method, device, equipment, computer-readable storage medium and computer program product.

Background technique

With the advent of the information age, a large amount of data, especially in the form of images, has been generated on the Internet. For example, a large number of product images are generated in an e-commerce application; another example is a large number of ticket images are generated in a mobile banking application. These images usually contain rich information, for example, commodity images may include commodity parameter information, and ticket images may include user identity information.

The above information is crucial for product recommendation or identity verification. However, the typesetting of information in these images is often complex and has no fixed format. If you rely on manual entry, it will take a lot of time and consume a lot of labor costs.

How to extract information with complex typesetting and no fixed format from images has become a major concern in the industry.

Contents of the invention

The purpose of the present disclosure is to provide an information extraction method, device, device, computer-readable storage medium and computer program product, capable of accurately extracting information from images with complex typesetting and no fixed format.

In a first aspect, the present disclosure provides an information extraction method, the method comprising:

Perform text detection on the image to obtain a text area in the image, where the text area includes a plurality of text lines;

Constructing a graph network model according to the text area, each text behavior in the text area is a node of the graph network model;

classifying nodes in the graph network model by a node classification model, and classifying edges between nodes in the graph network model by using an edge classification model;

Obtain at least one key-value pair in the image according to the classification results of the nodes and the classification results of the edges.

In a second aspect, the present disclosure provides an information extraction device, the device comprising:

A detection module, configured to perform text detection on the image to obtain a text area in the image, the text area including a plurality of text lines;

A building module, configured to construct a graph network model according to the text region, where each text in the text region acts as a node of the graph network model;

A classification module, configured to classify nodes in the graph network model through a node classification model, and classify edges between nodes in the graph network model through an edge classification model;

An obtaining module, configured to obtain at least one key-value pair in the image according to the classification results of the nodes and the classification results of the edges.

In a third aspect, the present disclosure provides an electronic device, including: a storage device, on which a computer program is stored; a processing device, configured to execute the computer program in the storage device, so as to implement the first aspect of the present disclosure. method steps.

In a fourth aspect, the present disclosure provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the steps of the method described in the first aspect of the present disclosure are implemented.

In a fifth aspect, the present disclosure provides a computer program product containing instructions, which, when run on a device, causes the device to execute the steps of the method described in the first aspect above.

As can be seen from the above technical solutions, the present disclosure has at least the following advantages:

In the above technical solution, the electronic device performs text detection on the image to obtain a text area including multiple text lines, and then uses each text line in the text area as a node to construct a graph network model, and uses the node classification model to classify the text in the graph network model. The nodes are classified, and the edges in the graph network model are classified by the edge classification model, and then the key-value pairs in the image are obtained according to the node classification results and the edge classification results. Among them, the electronic device not only classifies the nodes in the graph network model, but also classifies the edges in the graph network model, so that the characteristics of the text lines in the image and the characteristics of the associated text lines can be considered comprehensively, so that the image Accurately extract information with complex typesetting and no fixed format.

Other features and advantages of the present disclosure will be described in detail in the detailed description that follows.

Description of drawings

In order to more clearly illustrate the technical methods of the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments.

FIG. 1 is a schematic flow diagram of an information extraction method provided in an embodiment of the present application;

FIG. 2 is a schematic diagram of a text bounding box of an image provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of a graph neural network model provided in an embodiment of the present application;

FIG. 4 is a schematic diagram of node embedding of a graph neural network model provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of an information extraction device provided by an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

The terms "first" and "second" in the embodiments of the present application are used for description purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Thus, a feature defined as "first" and "second" may explicitly or implicitly include one or more of these features.

First, some technical terms involved in the embodiments of the present application are introduced.

For the extraction of text information in images, optical character recognition (OCR) technology is usually used. OCR refers to the process in which electronic devices determine the shape of printed characters by detecting dark and bright patterns, and then use character recognition methods to translate the shape into computer text. For printed characters, OCR can optically convert the text of the printed font into a black and white dot matrix image file, and then convert the text in the image into a text format through recognition software. However, in text recognition, especially for images with complex typesetting and no fixed format, there may be a variety of information in different typesetting intervals in the recognized text, making it difficult to accurately classify irrelevant text. Moreover, for an image with a certain part of the text that is long, it may not be possible to combine multiple lines of related text, and it is difficult to accurately combine the related text.

In view of this, the present application provides an accurate information extraction method, which is applied to electronic equipment. An electronic device refers to a device capable of data processing, such as a server or a terminal. Wherein, the terminal includes, but is not limited to, a smart phone, a tablet computer, a notebook computer, a personal digital assistant (personal digital assistant, PDA) or a smart wearable device. The server may be a cloud server, for example, a central server in a central cloud computing cluster, or an edge server in an edge cloud computing cluster. Certainly, the server may also be a server in a local data center. An on-premises data center refers to a data center directly controlled by the user.

Specifically, the electronic device performs text detection on the image, obtains a text area including multiple text lines in the image, and establishes a graph network model with a node of each text behavior graph network model, and then classifies the nodes in the graph network model through the node classification model. Classify the nodes, classify the edges between the nodes through the edge classification model, and then obtain at least one key-value pair in the image according to the classification results of the nodes and the classification results of the edges, so that the image can be accurately extracted. There is no fixed format for the information.

On the one hand, the result of edge classification can provide a reference for the result of node classification, so that the node classification model can obtain more accurate node classification results. On the other hand, the result of node classification can provide a reference for the result of edge classification, so that the edge classification model can obtain more accurate edge classification results. In this way, the electronic device integrates node classification and edge classification, comprehensively considers the characteristics of the text line itself and the characteristics between the associated text lines, and realizes the accurate extraction of information with complex typesetting and no fixed format in the image.

In order to make the technical solution of the present disclosure clearer and easier to understand, the following uses an electronic device as a terminal as an example, as shown in FIG. 1 , to introduce the information extraction method provided by the embodiment of the present disclosure.

Step 102: the terminal performs text detection on the image to obtain a text area in the image.

In this embodiment, an image refers to an image including a text area, and the text area in the image includes multiple text lines. The terminal can perform text detection on images in various ways, for example, the terminal can perform text detection through OCR technology. Typically, OCR includes text detection and text recognition. Text detection is used to find and segment text areas in pictures, and text recognition is used to convert text characters into computer text.

Specifically, the terminal can discover the text area in the picture through the OCR technology, and segment the text area in the form of a text bounding box (bounding box, bbox), as shown in Figure 2, wherein the text area in the image is shown in Figure 202 As shown, the text area 202 includes multiple text bounding boxes 204-1, 204-2, etc., and each text bounding box corresponds to a text line.

The terminal recognizes the text in the text bounding box in the text area, and obtains the text information of the text line corresponding to each text bounding box. In some possible implementations, the text information recognized and acquired by the terminal can be displayed in the image text bounding box.

Step 104: The terminal builds a graph network model according to the text area.

The graph network (GN) model refers to the model established according to the graph structure. A graph (graph) can be used to represent various types of structures or systems, and a graph can be described by nodes (node, N) and edges (edge, E), for example, G=(N, E). The graph includes two basic features of nodes and edges, where each node has the characteristic information of the node, and each node in the graph has the structural information of the node, that is, the edge information.

In this embodiment, the terminal may construct a graph neural network (graph neural network, GNN) model according to the text region. The graph neural network model is a neural network model established based on the corresponding relationship between nodes and edges in the graph. It comprehensively considers the characteristic information of each node in the graph and the structural information (edge information) of the node to realize the Accurate extraction of information.

The terminal can use each text line in the text area as a node to build a graph network model. In some possible implementation manners, the terminal may determine the edges in the graph network model according to the positional relationship in the text lines, as shown in FIG. 3 . Specifically, the terminal can determine the edges in the node according to the viewing circle visibility. Visibility of the viewing circle refers to establishing the edges in the graph network model according to the diameter of the visible circle. The edges determined based on the visibility of the viewing circle can satisfy: the circles generated by using all the edges in the graph as diameters do not intersect. The edge established through the visibility of the viewing circle can avoid the edge connection between non-adjacent text lines, reduce the impact on subsequent model recognition, reduce the difficulty of model learning, and improve the accuracy of the model.

Step 106: The terminal extracts the features of the nodes.

The terminal can extract the features of each node in the image in multiple ways, for example, it can use up-sampling or down-sampling, or a combination of up-sampling and down-sampling. In deep learning, upsampling refers to a technology that can make an image into a higher resolution. The upsampling method can specifically include interpolation, deconvolution, and anti-pooling methods. Interpolation is the use of mathematical formulas to calculate missing pixels from surrounding pixels without generating pixels. Deconvolution is the inverse process of convolution, which can be understood as a special forward convolution. First, the size of the input image is enlarged by filling 0 according to a certain ratio, and then the convolution kernel is rotated to perform forward convolution. Anti-pooling is the inverse operation of pooling, which can specifically include anti-maximum pooling and anti-average pooling. Among them, anti-maximum pooling needs to record the position of the maximum value during pooling. Downsampling refers to a new sequence obtained by sampling once every few samples of the original sequence.

In this embodiment, in order to avoid obtaining too many or too few features in the image, the terminal can first down-sample and then up-sample the features in the image, so that image feature samples with a relatively uniform number of samples can be obtained, for example The terminal can use UNet to extract features from the entire image.

On the basis of acquiring the features of the entire image, the terminal can further determine the features of the corresponding nodes in the image. Among them, the features of the node can include features containing various information of the node, such as image features containing the color, font and font size information of the text in the node, text features containing the text content in the node, and position features containing the coordinates of the node in the image .

In some possible implementations, the terminal can use Region of interest pooling (Region of interest pooling, ROI pooling for short) or ROI align to process the image features in the entire image to obtain the image features corresponding to each node. Wherein, the image feature of the node may be any one or more of the color, font and font size of the text in the text area corresponding to the node.

For the text feature of the node, the terminal may use the language model to extract the text feature of the text line corresponding to the node. A language model (language model, LM) refers to a probability model established for a certain language, which can establish a probability distribution describing the occurrence of a given word sequence in a language. In this embodiment, the terminal can use bidirectional long short term memory (Bi-directional long short term memory, Bi-LSTM) or bidirectional encoding representation based on converters (bidirectional encoder representation from transformers, BERT) and other language models to extract the information in the image. The text features corresponding to each node, so as to obtain the text features corresponding to each node. Among them, Bi-LSTM is generated by combining the forward long short term memory (LSTM) with the backward LSTM. LSTM can learn long-term dependent information, so it can have high recognition accuracy. In the process of recognizing text information, it is usually possible to simply combine the recognized characters to form a corresponding sentence, but this recognition accuracy does not consider the sequence of words in the sentence, and the recognition accuracy is low, while using LSTM can Capturing long-distance dependencies, taking into account the order of words in sentences with long distances. Further, Bi-LSTM can not only learn the information from front to back, but also from back to front, so it can better capture the bidirectional semantic dependence. BERT is a pre-trained language representation model. The model no longer uses the traditional one-way language model or two one-way language models for pre-training, but uses a new masked language model (masked language model, MLM), so as to be able to generate deep bidirectional language representation. Through the above method, the terminal can obtain the text content surrounded by the text bounding box corresponding to each node, so as to obtain the text features of the node.

The terminal can determine the location characteristics of the node according to the location of the node. Specifically, the terminal may determine the location feature of the node according to the location information (for example, coordinate information) of the text bounding box. The terminal may also perform embedding processing on the location information of the text bounding box to obtain the location feature of the node. Among them, the embedding process refers to converting the position feature of the node into a low-dimensional real-valued vector through calculation, and can fuse multiple features into a continuous and computable vector. In this embodiment, the terminal may express the obtained location information of the text bounding box of the node as a low-dimensional computable real-valued vector through embedding processing, so as to obtain the location feature of the node.

In this way, the terminal can extract node features such as image features, text features, and location features of each node in the image. The node features include various information about the node, and the multiple node features corresponding to each node constitute the input graph neural network of the node. node characteristics.

Step 108: The terminal extracts features of edges between nodes.

For the edge features, the terminal can obtain the edge features in the graph network model according to the relative position and relative width and height between the text lines with the edge connection relationship. Wherein, the edge connection relationship is the edge determined in step 104 . Specifically, the relative position between the text lines may be the relative position of the text bounding boxes corresponding to the text lines, and the relative width and height between the text lines may be the relative width and height of the text bounding boxes corresponding to the text lines. For example, the coordinates of the center of the text bounding box A corresponding to the text line A are (xA, yA), the width is wA, and the height is hA, and the coordinates of the center of the text bounding box B corresponding to the text line B are (xB, yB), The width is wB, and the height is hB. Then, the relative position between the text line A and the text line B can be (xB-xA, yB-yA), the relative width is wA/wB, and the relative height is hA/hB.

In this way, the terminal can obtain node features in the graph neural network including image features, text features, and position features, and edge features in the graph neural network including relative positions between text lines and relative width and height.

Step 110: The terminal aggregates the characteristics of the neighbor nodes of the node according to the characteristics of the edges, and obtains the embedding representation of the node.

Among them, the neighbor nodes of a node refer to nodes that have an edge association relationship with the node, and two nodes on the same edge are neighbor nodes. For any node in the image, the terminal can obtain the node features of the node and the edge features of the edges including the node, and then obtain the node features of another node of the edge, that is, the node’s neighbor nodes, so as to obtain the relevant information.

Specifically, the terminal obtains the characteristics of the neighbor nodes of the node according to the node characteristics of the node and the edge characteristics of the corresponding edge of the node through the graph neural network, and jointly obtains the characteristics of the neighbor nodes including the node and The embedded representation of this node for edge features is shown in Figure 4.

In some possible implementations, the terminal can use spatial graph convolutional networks (graph convolutional networks, GCN) to aggregate the characteristics of the neighbor nodes of the node according to the characteristics of the edge, and obtain the embedded representation of the node. Graph convolution network, graph recurrent network (graph recurrent network, GRN), graph attention network (graph attention network, GAT), graph autoencoder (graph autoencoders, GAE) all belong to the graph neural network. In this embodiment, graph Convolutional network is used as an example to introduce. Compared with the traditional CNN, which is mainly used in the Euclidean space where the neighbor nodes are fixed, GCN can be applied in the non-Euclidean space where the neighbor nodes are not fixed. The convolution of a node by the graph convolutional network is actually a weighted summation of the node and the neighbor nodes with edge associations, so that the node's own characteristics, edge characteristics and neighbor node characteristics can be aggregated to obtain the embedding of the node express. Graph convolutional networks are mainly divided into spatial-domain-based graph convolutional networks and frequency-domain-based graph convolutional networks. The graph convolutional network based on the spatial domain can directly convolve the nodes in the image, while the graph convolutional network based on the frequency domain needs to perform Fourier transform first, and then perform convolution.

Step 112: The terminal classifies the nodes in the graph network model through the node classification model.

The node classification model refers to a model that can classify nodes. The input of the node classification model can be the node embedding in the graph neural network, and the output is the type of the node, such as key (key), value (value) and other . The type corresponding to the node can be set by the user. As shown in Figure 2, it is a product label. The product attribute can be set as the key, the product feature corresponding to the product attribute can be set as the value, and the rest of the content can be set as others.

In some possible implementation manners, the end-to-end model of the node classification model may be, for example, a multilayer perceptron (multilayer perceptron, MLP) model. MLP is also called artificial neural network (ANN), which includes an input layer, an output layer, and at least one hidden layer. MLP can often be used to deal with classification problems.

The node classification model can be other trained multi-classification models, such as k-nearest neighbors, decision trees, naive bayes, random forest, and gradient Models such as gradient boosting. In some possible implementation manners, the types of nodes can also be two types, such as key and value, so a binary classification model can also be used to classify the nodes in the graph network model.

Step 114: The terminal classifies the edges between the nodes in the graph network model through the edge classification model.

The edge classification model refers to a model that can classify edges, where the input of the edge classification model can be the splicing of two node embeddings in the graph neural network, and the output is the type of edge, such as the corresponding key-value edge in step 112 , key-key edges, value-value edges, and others. The edge between a commodity attribute and the commodity feature corresponding to the attribute is a key-value edge, the edge between two commodity attributes is a key-key edge, and the edge between two commodity features is a value-value edge, etc.

Similarly, the edge classification model may also be an end-to-end model, and the terminal obtains an edge classification model capable of classifying edges between nodes by training the MLP model.

In some possible implementations, the node classification model and the edge classification model can be jointly trained as input and output of each other. Among them, the node classification model can be verified through the edge classification model. For example, for edge A, the node classification model judges that the two nodes of edge A are key and value respectively, and whether edge A is a key-value edge can be verified through the edge classification model. For edge B , the node classification model judges that both nodes of edge B are keys, and it can be verified whether edge B is a key edge through the edge classification model. For edge C, the node classification model judges that both nodes of edge C are values, and can be verified by edge The classification model verifies whether edge C is a value-value edge, etc.

Similarly, the edge classification model can also be verified through the node classification model. For example, the edge classification model judges that edge D is a key-value edge. E is a key-key edge. You can use the node classification model to judge whether the two nodes of edge E are both keys. The edge classification model judges that edge F is a value-value edge. You can use the node classification model to judge whether the two nodes of edge F are both values. .

Step 116: The terminal obtains at least one key-value pair in the image according to the classification results of the nodes and the classification results of the edges.

In some possible implementations, the terminal can determine two adjacent nodes that are key and value respectively as key-value edges according to the classification results of the nodes, and then verify whether the edge formed by the two nodes is key-value pairs. The terminal can also determine the key-value edge according to the edge classification result, and then judge whether the two nodes of the edge are key and value according to the node classification result. When the edge classification model judges that the edge is a key-value edge and the node classification model judges that the two nodes of the edge are key and value, the terminal determines to obtain a key-value pair in the image, so that at least one key-value pair in the image can be obtained right.

Based on the above description, the present disclosure provides an information extraction method. The terminal performs text detection on the image to obtain a text area including multiple text lines, and then uses each text line in the text area as a node to construct a graph network model, classifies the nodes in the graph network model through the node classification model, and uses the edge The classification model classifies the edges in the graph network model, and then obtains the key-value pairs in the image according to the node classification results and the edge classification results. In this way, the terminal not only classifies the nodes in the graph network model, but also classifies the edges in the graph network model, and can comprehensively consider the characteristics of the text lines in the image itself and the characteristics of the associated text lines, so as to be able to typesetting in the image Accurate extraction of complex and unformatted information.

Fig. 5 is a schematic diagram of an information extraction device according to an exemplary disclosed embodiment. As shown in Fig. 5, the information extraction device 500 includes:

A detection module 502, configured to perform text detection on the image, and obtain a text area in the image, where the text area includes a plurality of text lines;

A construction module 504, configured to construct a graph network model according to the text region, where each text in the text region acts as a node of the graph network model;

A classification module 506, configured to classify nodes in the graph network model through a node classification model, and classify edges between nodes in the graph network model through an edge classification model;

The obtaining module 508 is configured to obtain at least one key-value pair in the image according to the classification results of the nodes and the classification results of the edges.

In a possible implementation manner, the device further includes an extraction module 510, and the extraction module 510 may be used to:

extracting features of the nodes, and extracting features of the edges;

Aggregating the characteristics of the neighbor nodes of the node according to the characteristics of the edge to obtain the embedded representation of the node;

The classification module 506 can be used for:

Classifying the nodes in the graph network model through a node classification model according to the embedded representation of the nodes;

The embedded representations of the two nodes corresponding to the edges are spliced, and according to the spliced embedded representations, the edges between the nodes in the graph network model are classified by an edge classification model.

In a possible implementation manner, the classification module 506 may be used to:

According to the classification result of the nodes, the edges between the nodes in the graph network model are classified by using an edge classification model.

In a possible implementation manner, the classification result of the node includes one of the following labels: key, value and others, and the classification result of the edge includes one of the following labels: key One of value edge, value-value edge, key-key edge, or other.

In a possible implementation manner, when the classification result of the node is a key, the classification result of the edge includes a key-value edge or a key-key edge; when the classification result of the node is a value, then the The edge classification results include key-value edges or value-value edges.

In a possible implementation manner, the features of the nodes include at least one of the image features, text features, and position features of the nodes, and the edge features include the relative position and the relative width between the text lines. At least one from high school.

In a possible implementation manner, the node classification model and the edge classification model are end-to-end models.

The functions of the above modules have been described in detail in the method steps in the previous embodiment, and will not be repeated here.

Referring now to FIG. 6 , it shows a schematic structural diagram of an electronic device 600 suitable for implementing an embodiment of the present disclosure. The terminal equipment in the embodiment of the present disclosure may include but not limited to such as mobile phone, notebook computer, digital broadcast receiver, PDA (personal digital assistant), PAD (tablet computer), PMP (portable multimedia player), vehicle terminal (such as mobile terminals such as car navigation terminals) and fixed terminals such as digital TVs, desktop computers and the like. The electronic device shown in FIG. 6 is only an example, and should not limit the functions and application scope of the embodiments of the present disclosure.

As shown in FIG. 6, an electronic device 600 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which may be randomly accessed according to a program stored in a read-only memory (ROM) 602 or loaded from a storage device 608. Various appropriate actions and processes are executed by programs in the memory (RAM) 603 . In the RAM 603, various programs and data necessary for the operation of the electronic device 600 are also stored. The processing device 601 , ROM 602 and RAM 603 are connected to each other through a bus 604 . An input/output (I/O) interface 605 is also connected to the bus 604 .

Typically, the following devices can be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speaker, vibration an output device 607 such as a computer; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While FIG. 6 shows electronic device 600 having various means, it should be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a non-transitory computer readable medium, where the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 609, or from storage means 608, or from ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

It should be noted that the above-mentioned computer-readable medium in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, the client and the server can communicate using any currently known or future network protocols such as HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol), and can communicate with digital data in any form or medium The communication (eg, communication network) interconnections. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: performs text detection on the image, obtains a text area in the image, and the text The region includes a plurality of text lines; a graph network model is constructed according to the text region, and each text behavior in the text region is a node of the graph network model; nodes in the graph network model are processed through a node classification model Classify, and classify the edges between the nodes in the graph network model through the edge classification model; according to the classification results of the nodes and the classification results of the edges, at least one key value in the image is obtained right. Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, or combinations thereof, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages - such as "C" or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, using an Internet service provider to connected via the Internet).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The modules involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Wherein, the name of the module does not constitute a limitation on the module itself under certain circumstances.

The functions described herein above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System on Chips (SOCs), Complex Programmable Logical device (CPLD) and so on.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, Example 1 provides an information extraction method, the method comprising: performing text detection on an image, and obtaining a text area in the image, the text area including a plurality of text lines ; Construct a graph network model according to the text region, each text behavior in the text region is a node of the graph network model; classify the nodes in the graph network model through the node classification model, and classify the nodes through the edge classification The model classifies the edges between the nodes in the graph network model; according to the classification results of the nodes and the classification results of the edges, at least one key-value pair in the image is obtained.

According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, the method further includes: extracting the features of the nodes, and extracting the features of the edges; aggregating the The characteristics of the neighbor nodes of the node are obtained to obtain the embedded representation of the node; the node classification model is used to classify the nodes in the graph network model, and the edge classification model is used to classify the nodes in the graph network model Classifying the edges includes: according to the embedded representation of the nodes, classifying the nodes in the graph network model through a node classification model; splicing the embedded representations of the two nodes corresponding to the edges, and according to the spliced The embedding represents that edges between nodes in the graph network model are classified by an edge classification model.

According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, and classifying the edges between nodes in the graph network model through an edge classification model includes:

According to one or more embodiments of the present disclosure, Example 4 provides the method of any one of Example 1 to Example 3, the classification result of the node includes one of the following labels: key, value and others, and The classification result of the edge includes one of the following labels: key-value edge, value-value edge, key-key edge or one of others.

According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 4. When the classification result of the node is a key, the classification result of the edge includes a key-value edge or a key-key edge; for all When the classification result of the node is a value, the classification result of the edge includes a key-value edge or a value-value edge.

According to one or more embodiments of the present disclosure, Example 6 provides the method of any one of Examples 1 to 5, wherein the features of the node include at least one of image features, text features, and location features of the node, The feature of the side is at least one of the relative position between the text lines, the relative width and the high height.

According to one or more embodiments of the present disclosure, Example 7 provides the method of any one of Example 1 to Example 5, wherein the node classification model and the edge classification model are end-to-end models.

According to one or more embodiments of the present disclosure, Example 8 provides an information extraction device, the device includes: a detection module, configured to perform text detection on an image, and obtain a text area in the image, the text area Including a plurality of text lines; a building module for constructing a graph network model according to the text region, each text behavior in the text region is a node of the graph network model; a classification module for classifying the model through the node Classify the nodes in the graph network model, and classify the edges between the nodes in the graph network model through the edge classification model; the acquisition module is used to classify the nodes according to the classification results of the nodes and the A classification result of the image to obtain at least one key-value pair in the image.

According to one or more embodiments of the present disclosure, Example 9 provides the device of Example 8, the device further includes an extraction module, the extraction module is configured to: extract the feature of the node, and extract the feature of the edge; Aggregating the characteristics of the neighbor nodes of the node according to the characteristics of the edge to obtain the embedded representation of the node; the classification module can be used to: classify the graph network model through a node classification model according to the embedded representation of the node The nodes in the graph network model are classified; the embedded representations of the two nodes corresponding to the edges are spliced, and according to the spliced embedded representations, the edges between the nodes in the graph network model are classified by an edge classification model.

According to one or more embodiments of the present disclosure, Example 10 provides the apparatus of Example 8, and the classification module may be configured to: use an edge classification model to classify nodes in the graph network model according to the classification result of the nodes classify the edges between them.

According to one or more embodiments of the present disclosure, Example 11 provides the device of any one of Example 8 to Example 10, the classification result of the node includes one of the following labels: key, value and others, and The classification result of the edge includes one of the following labels: key-value edge, value-value edge, key-key edge or one of others.

According to one or more embodiments of the present disclosure, Example 12 provides the apparatus of Example 11. When the classification result of the node is a key, the classification result of the edge includes a key-value edge or a key-key edge; for all When the classification result of the node is a value, the classification result of the edge includes a key-value edge or a value-value edge.

According to one or more embodiments of the present disclosure, Example 13 provides the apparatus of any one of Example 8 to Example 12, wherein the characteristics of the node include at least one of image characteristics, text characteristics and location characteristics of the node, The feature of the side is at least one of the relative position between the text lines, the relative width and the high height.

According to one or more embodiments of the present disclosure, Example 14 provides the apparatus of any one of Example 8 to Example 12, wherein the node classification model and the edge classification model are end-to-end models.

The above description is only a preferred embodiment of the present disclosure and an illustration of the applied technical principles. Those skilled in the art should understand that the disclosure scope involved in this disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, but also covers the technical solutions formed by the above-mentioned technical features or Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

In addition, while operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or performed in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while the above discussion contains several specific implementation details, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the foregoing embodiments, the specific manner in which each module executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

Claims

An information extraction method, the method comprising:

Perform text detection on the image to obtain a text area in the image, where the text area includes a plurality of text lines;

Constructing a graph network model according to the text area, each text behavior in the text area is a node of the graph network model;

classifying nodes in the graph network model by a node classification model, and classifying edges between nodes in the graph network model by using an edge classification model;

Obtain at least one key-value pair in the image according to the classification results of the nodes and the classification results of the edges.
The method according to claim 1, wherein the method further comprises:

extracting features of the nodes, and extracting features of the edges;

Aggregating the characteristics of the neighbor nodes of the node according to the characteristics of the edge to obtain the embedded representation of the node;

The classifying the nodes in the graph network model through the node classification model, and classifying the edges between the nodes in the graph network model through the edge classification model include:

Classifying the nodes in the graph network model through a node classification model according to the embedded representation of the nodes;

The embedded representations of the two nodes corresponding to the edges are spliced, and according to the spliced embedded representations, the edges between the nodes in the graph network model are classified by an edge classification model.
The method according to claim 1, wherein said classifying edges between nodes in said graph network model through an edge classification model comprises:

According to the classification result of the nodes, the edges between the nodes in the graph network model are classified by using an edge classification model.
The method according to claim 1, wherein the classification result of the node includes one of the following labels: key, value and others, and the classification result of the edge includes one of the following labels : One of key-value edge, value-value edge, or key-key edge.
The method according to claim 4, wherein, when the classification result of the node is a key, the classification result of the edge includes a key-value edge or a key-key edge; when the classification result of the node is a value, Then the classification result of the edge includes a key-value edge or a value-value edge.
The method according to any one of claims 1 to 5, wherein the features of the nodes include at least one of image features, text features, and position features of the nodes, and the text features of the edges are At least one of the relative position of the space, the relative width and high height.
The method according to any one of claims 1 to 5, wherein the node classification model and the edge classification model are end-to-end models.
An information extraction device, the device comprising:

A detection module, configured to perform text detection on the image to obtain a text area in the image, the text area including a plurality of text lines;

A building block for constructing a graph network model according to the text region, where each text in the text region acts as a node of the graph network model;

A classification module, configured to classify nodes in the graph network model through a node classification model, and classify edges between nodes in the graph network model through an edge classification model;

An obtaining module, configured to obtain at least one key-value pair in the image according to the classification results of the nodes and the classification results of the edges.
The device according to claim 8, wherein the device further comprises an extraction module, the extraction module is used for:

extracting features of the nodes, and extracting features of the edges;

Aggregating the characteristics of the neighbor nodes of the node according to the characteristics of the edge to obtain the embedded representation of the node;

The classification model is specifically used for:

Classifying the nodes in the graph network model through a node classification model according to the embedded representation of the nodes;

The embedded representations of the two nodes corresponding to the edges are spliced, and according to the spliced embedded representations, the edges between the nodes in the graph network model are classified by an edge classification model.
The device according to claim 8, wherein the classification module is specifically used for:

According to the classification result of the nodes, the edges between the nodes in the graph network model are classified by using an edge classification model.
The apparatus according to claim 8, wherein the classification result of the node includes one of the following labels: key, value and others, and the classification result of the edge includes one of the following labels : One of key-value edge, value-value edge, or key-key edge.
The device according to claim 11, wherein when the classification result of the node is a key, the classification result of the edge includes a key-value edge or a key-key edge; when the classification result of the node is a value, Then the classification result of the edge includes a key-value edge or a value-value edge.
The device according to any one of claims 8 to 12, wherein the feature of the node includes at least one of an image feature, a text feature and a position feature of the node, and the feature of the edge is that the text line At least one of the relative position of the space, the relative width and high height.
The apparatus according to any one of claims 8 to 12, wherein the node classification model and the edge classification model are end-to-end models.
An apparatus comprising a processor and a memory;

The processor is configured to execute instructions stored in the memory, so that the device executes the method according to any one of claims 1-7.
A computer-readable storage medium comprising instructions instructing a device to perform the method according to any one of claims 1-7.
A computer program product, which causes the computer to execute the method according to any one of claims 1 to 7 when the computer program product is run on a computer.