TWI701620B - Document information extraction and archiving system - Google Patents

Abstract

A document information extraction and archiving system is electrically connected to a database, and the Document information extraction and archiving system includes an input module, a text segmentation area detection module, a character recognition module, a semantic segmentation module, and a database connection module. The input module accepts a document image including a plurality of characters, and the text segmentation area detecting module frames the text in the document image by a first type of neural network model to form a text segmentation area. The character recognition module recognizes the text in the text segmentation area by a second type of neural network model to obtain an editable string. The semantic segmentation module breaks the string to form a plurality of word segments, and assigns each word segment a symbol. Moreover, the database connection module serially links the word segment to each field of the database according to the symbol thereof.

Description

Document information extraction and filing system

The invention refers to a file information extraction and archiving system, in particular to a file information extraction and archiving system involving text images.

At present, in order to evaluate the existing insurance policies of potential customers, the insurance company’s brokers must enter the information on the existing insurance policies into the insurance company’s evaluation system in order to evaluate the potential customers’ existing insurance policies and make further recommendations to the potential customers . However, there are many information on existing insurance policies and potential customers often only have written information without electronic information. Brokers must manually enter the information on existing insurance policies into the insurance company’s evaluation system, which will take a lot of time , Reduce the efficiency of developing new customers.

Therefore, how to automatically retrieve the information on the existing insurance policy and input it into the evaluation system of the insurance company to which the broker belongs is a subject worthy of consideration by those with ordinary knowledge in the field.

The purpose of the present invention is to provide a document information extraction and filing system that can automatically extract text information on document images. The document information extraction and filing system is electrically connected to a database. The document information extraction and filing system includes an input module, a text segmentation area detection module, a text recognition module, a semantic segmentation module, and a database Docking module. The input module accepts a document image, and the document image includes multiple characters. The text segmentation area detection module uses a first-type neural network model to frame the text in the document image to form at least one text segmentation area. The text recognition module has a second The neural network-like model recognizes the text in the text segmentation area to obtain at least one editable character string, and the character string includes at least one editable character. The semantic segmentation module hyphenates the word string to form multiple word segments, and assigns a part of speech to each word segment. Moreover, the database docking module concatenates the word segmentation with each field of the database based on the part of speech.

As in the above-mentioned document information extraction and archiving system, the first type of neural network model includes a first convolutional neural network model and a target detection neural network model. The first convolutional neural network model performs processing on the document image The feature extraction is to output a feature vector, and the target detection neural network model performs frame selection on the serial number type text according to the input of the feature vector to form a serial number segmentation area. Among them, the first convolutional neural network model is a VGG model, a ResNet model, or a DenseNet model. In addition, the target detection neural network model is a YOLO model, a CTPN model, or an EAST model.

As in the above-mentioned document information extraction and filing system, the second type of neural network model includes a second convolutional neural network model and a recursive neural network model. The second convolutional neural network model divides the text area The image in is processed to output a text sequence, and the recursive neural network model outputs the editable text string according to the input of the text sequence. Among them, the recurrent neural network model implements the Connectionist Temporal Classification algorithm.

As in the above-mentioned document information extraction and filing system, the second type of neural network model is the Seq2Seq model.

As in the above-mentioned document information extraction and filing system, the semantic segmentation module further includes a vocabulary and a rule module. The vocabulary stores multiple proper nouns in a specific field, and the rule module is used to combine various Participles give different parts of speech.

As in the above-mentioned document information extraction and filing system, the semantic segmentation module vectorizes these word segments, and uses conditional random fields or hidden Markov models to assign a part of speech to each word segment.

As in the above-mentioned document information extraction and filing system, the semantic segmentation module includes a third-type neural network model. The semantic segmentation module converts each character of the string into a fixed-dimensional feature vector, and inputs the feature vector to The third type of neural network model is to give each word segmentation a part of speech. Among them, the third type of neural network model is a recurrent neural network. Moreover, the third type of neural network model includes a conditional random field layer.

As in the above-mentioned document information extraction and filing system, the database docking module includes a word classifier, which further classifies the word segmentation of the same part of speech.

With the document information extraction and filing system in this case, the text in the document image can be automatically input into the corresponding field of the database, without the need for manual input by humans, which greatly improves the efficiency of administrative operations.

In order to make the above-mentioned features and advantages of the present invention more obvious and understandable, a detailed description is given below of preferred embodiments in conjunction with the accompanying drawings.

10: File image

12: Text segmentation area

12a: Picture sequence

30: Database

40: Video input device

100: Document information extraction and filing system

110: Input module

115: image pre-processing module

120: Text segmentation area detection module

122: The first type of neural network model

1221: The first convolutional neural network model

1223: Neural network model for target detection

130: text recognition module

132: The second type of neural network model

1321: The second convolutional neural network model

1323: Recurrent neural network model

140: Semantic Segmentation Module

141: The third type of neural network model

1411: Embedding vector layer

1413: Recurrent neural network layer

1415: excitation function layer

1417: Conditional Random Field Layer

142: Thesaurus

143: Rule Module

150: Database docking module

151: text classifier

Hereinafter, various embodiments will be described based on the accompanying drawings, which are used for illustration rather than limiting the scope in any way, wherein similar reference numerals indicate similar components, and among them: FIG. 1 is a drawing of the present invention. An embodiment of a file information extraction and filing system.

Figure 2A shows the document image of the insurance policy.

Figure 2B shows the document image of the insurance policy after image preprocessing.

FIG. 2C shows a document image with text division areas.

Figure 3 shows the architecture diagram of the first type of neural network model.

Figure 4 shows the architecture diagram of the second type of neural network model.

FIG. 5 shows a schematic diagram of disassembling the text segmentation area into multiple picture sequences.

FIG. 6 shows the structure diagram of the semantic segmentation module.

Figure 7 shows the architecture diagram of the third type of neural network model.

The present invention is best understood with reference to the detailed content set forth herein and the description of the drawings. Various embodiments will be discussed below with reference to the drawings. However, those skilled in the art will easily understand that the detailed description given here with respect to the drawings is only for explanatory purposes, because these methods and systems may exceed the described embodiments. For example, the given teachings and specific application requirements may produce a variety of alternative and suitable methods to implement Now this describes the function in any detail. Therefore, any method can extend beyond the specific implementation options described and illustrated in the following embodiments.

In the specification and subsequent patent applications, certain words are used to refer to specific elements. Those with ordinary knowledge in the field should understand that different manufacturers may use different terms to refer to the same components. The scope of this specification and subsequent patent applications does not use differences in names as a way to distinguish elements, but uses differences in functions of elements as a criterion for distinguishing. The "include" or "include" mentioned in the entire specification and the subsequent request items is an open term, so it should be interpreted as "include but not limited to". In addition, the term "coupled" or "connected" herein includes any direct and indirect electrical connection means. Therefore, if it is described that a first device is coupled to a second device, it means that the first device can be directly electrically connected to the second device, or indirectly electrically connected to the second device through other devices or connection means. Device.

Please refer to FIG. 1. FIG. 1 illustrates an embodiment of the document information extraction and filing system of the present invention. The document information extraction and filing system 100 includes an input module 110, an image pre-processing module 115, a text segmentation area detection module 120, a text recognition module 130, a semantic segmentation module 140, and a database docking module 150. The database docking module 150 is connected to a database 30. In this embodiment, the database 30 is, for example, an insurance company's database. The database includes multiple fields, such as name, ID number, insurance type, insurance amount, etc. In addition, the input module 110 is, for example, electrically connected to an image input device 40. The image input device 40 is, for example, a scanning device, a digital camera, or a smartphone with a camera function. With this image input device 40, a document image (for example, an insurance policy, the photo shown in FIG. 2A) can be imported into the image pre-processing module 115. The image pre-processing module 115 can perform image pre-processing on the document image, such as: orientation correction, curved surface correction, image denoising, binarization, etc., so that the document image has high contrast characteristics (as shown in FIG. 2B) File image 10) to facilitate subsequent processing. In this embodiment, the input module 110, the image pre-processing module 115, the text segmentation area detection module 120, the text recognition module 130, the semantic segmentation module 140, and the database docking module 150 are installed in the server For example, the server side is composed of one or more servers. It should be noted that, in order to protect personal privacy, the insured and the insured in Figure 2A The name of the person and the policy number are covered, while the names of the insured and the insured and the policy number in Figure 2B are changed.

The pre-processed document image 10 will be transmitted to the text segmentation area detection module 120. The text segmentation area detection module 120 includes the first type of neural network model 122. The first type of neural network model 122 can The text in the document image 10 is frame-selected to form at least one text segmentation area 12 (a plurality of texts are shown in FIG. 2C). It should be noted that the text in the text segmentation area 12 exists in the form of images, that is, the text in the text segmentation area 12 cannot be edited at this stage. In order to convert these texts into editable texts, the text recognition module 130 can be used to complete. Hereinafter, a more detailed operation mechanism of the text segmentation area detection module 120 and the text recognition module 130 will be introduced.

3, the first type of neural network model 122 includes a first convolutional neural network model 1221 and a target detection neural network model 1223, this first convolutional neural network model 1221 belongs to convolution Convolutional neural network (convolutional neural network) includes convolutional layer and pooling layer (both convolutional layer and sampling layer are not shown in the figure). The convolutional layer is mainly used for feature extraction, and sampling The layer is used to reduce the parameters required by the first convolutional neural network model 1221 to avoid overfitting. The first convolutional neural network model 1221 generates a feature vector according to the input document image 10, and then the feature vector is input to the target detection neural network model 1223. In this embodiment, the first convolutional neural network model 1221 may be a VGG model, a ResNet model, or a DenseNet model. In addition, the target detection neural network model 1223 may be a YOLO model, preferably a CTPN model or an EAST model. After the calculation of the target detection neural network model 1223, the text in the document image 10 will be framed to form the aforementioned text segmentation area 12 (as shown in FIG. 2C).

After the text in the document image 10 is framed to form the text segmentation area 12, the text recognition module 130 will recognize the text in the text segmentation area 12 through a second-type neural network model 132. 4, the second type of neural network model 132 includes a second convolutional neural network model 1321 and a recurrent neural network model 1323, the second convolutional neural network model 1321 and the first The convolutional neural network model 1221 belongs to the same convolutional neural network (convolutional neural network). network), this second convolutional neural network model 1321 can pre-judge the text in the text segmentation area 12. Although the second convolutional neural network model 1321 can make preliminary judgments on the text in the text segmentation area 12, it is better to add a recursive neural network model 1323 to the second convolutional neural network model 1321. In order to better recognize the text in the text segmentation area 12, the relevant detailed mechanism will be described later.

When the second convolutional neural network model 1321 recognizes the text in the text segmentation area 12, it first disassembles the text segmentation area 12 into a plurality of picture sequences 12a (as shown in FIG. 5). For example, if the text segmentation area 12 includes the character "S", these picture sequences 12a may be the left part of "S" or the right part of "S", so that the second convolutional neural network The network model 1321 may recognize the character "S" as these two "S" characters. Or, conversely, it is also possible to recognize multiple characters as one, such as the string "llc.", and the second convolutional neural network model 1321 may treat two ls ("ll") among them as one l ("1"). The recursive neural network model 1323 belongs to the recurrent neural network (Recurrent Neural Network, RNN). Because the recurrent neural network refers to the previous input, that is to say, it has the function of short-term memory, so it can perform the second convolution The possible output error of the neural network model 1321 is corrected, and the text in the text segmentation area 12 is correctly recognized.

In this embodiment, the recursive neural network model 1323 uses the Connectionist Temporal Classification algorithm (hereinafter referred to as the CTC algorithm), for example. At present, the CTC algorithm is mainly used for speech recognition. For detailed operating principles, please refer to the following webpage: "Sequence Modeling With CTC" (https: //distill.pub/2017/ctc/)

After research, the creator of this case found that the CTC algorithm can also be applied to the text recognition of this case and has good results. The main reason is that the context of speech recognition and the context of text recognition in this case have some in common. Some of the more common situations in speech recognition are: some people speak faster, some people speak slower, or some people stretch longer in certain phonemes; and the CTC algorithm is officially developed for these situations. In the text recognition in this case, the spacing between characters in some documents will be relatively open (corresponding to some people speaking slowly in voice recognition), and the spacing between characters in some documents will be widened. Should be more compact (corresponding to some people speaking faster in speech recognition), and Moreover, the document image in this case may be obtained by taking a photo, so it is more likely that the distance between characters in the document will change due to the angle or distance of the photographer. Therefore, the creator of this case uses the CTC algorithm to solve such problems and obtain good results.

In addition, the second type of neural network model 132 may also be a Seq2Seq model. The Seq2Seq model generally includes an encoder (Encoder) and a decoder (Decoder), where the encoder can be a convolutional neural network, which will also first disassemble the text segmentation area 12 into multiple image sequences 12a (as shown in the figure) 5), and convert the picture sequence 12a into a context vector, and then input the context vector to the decoder, and the decoder converts the context vector into an editable string.

It is worth noting that because capturing document images (as shown in Figure 2A) involves taking pictures, different people have different shooting angles. Therefore, the first type of neural network model 122 and the second type of neural network The model 132 can input file images under different angles and various light environments during training. These file images under different angles and various light environments can be directly obtained by computer simulation.

After the editable character string is obtained through the text recognition module 130, the word string can be segmented through the semantic segmentation module 140 to form a plurality of word segments, and a part of speech is assigned to each word segment. In this embodiment, the word segmentation program jieba can be used for word segmentation to form the aforementioned word segmentation. In addition, referring to FIG. 6, the semantic segmentation module 140 includes a vocabulary 142 and a rule module 143. The vocabulary 142, for example, stores multiple proper nouns in a specific field, and the rule module 143 is used to combine various Participles are assigned different parts of speech. For example, "Kong Yiji" is assigned the part of speech of a person's name, "Taipei City Dazhi" is assigned the part of speech of place name, "Nanshan Life Insurance" is assigned the part of speech of company name, and "102/12/31" is assigned to the date. This part of speech etc. In addition to assigning part of speech according to the characteristics of the word segmentation, the rule module 143 can also determine the position of the word segmentation in the string. For example, using the CYK algorithm (Cocke-Younger-Kasami algorithm, abbreviated as CYK algorithm) ).

In other embodiments, the semantic segmentation module 140 vectorizes the aforementioned word segmentation, and then uses Conditional Random Field (CRF) or Hidden Markov Model (Hidden Markov Model) to assign a part of speech to each word segmentation.

In a preferred embodiment, as shown in FIG. 7, the semantic segmentation module 140 includes a third type of neural network model 141, and the third type of neural network model 141 includes an embedding layer (Embedding layer) 1411, a recurrent neural network layer (RNN layer) 1413, an activation function layer (activation layer) 1415, and a conditional random field layer (CFR layer) 1417. In this embodiment, the semantic segmentation module 140 converts each character of the character string into a feature vector of a fixed dimension, and these feature vectors constitute the embedded vector layer 1411. Here, the recurrent neural network layer 1413 can be a basic recurrent neural network or a bidirectional recurrent neural network (Bi-RNN, as shown in Figure 7), and can also include long and short-term memory recurrent neural networks. Network (LSTM-RNN, LSTM is the abbreviation of Long Short-Term Memory), bidirectional long and short-term memory recurrent neural network (BLSTM-RNN, BLSTM is the abbreviation of Bidirectional Long Short-Term Memory), GRU recurrent neural network (GRU-RNN, GRU is the abbreviation of Gated Recurrent Unit). In addition, the excitation function of the excitation function layer 1415 is, for example, a tanh function, and the reason why the third type of neural network model 141 is added to the conditional random field layer 1417 is that the conditional random field has advantages in sequence labeling. After the third type of neural network model 141, each text in the string will be given a part of speech. Related matters are as follows, where 1 represents the name of the person, 2 represents the name of the place, 3 represents the name of the company, 4 represents the date, 5 is others.

Please refer back to FIG. 1. After completing the part-of-speech tagging of each word segment in the string, the database docking module 150 will concatenate the word segmentation with each field of the database 30 according to the part-of-speech of each word segment. For example, "Kong Yiji" in Figure 2B will be classified into 30 fields related to the names of people in the database. In a preferred embodiment, the database docking module 150 further includes a text classifier 151, which can further classify the word segmentation of the same part of speech. For example, different names of persons may appear in a document, and the word classifier 151 can determine which of these names is the insured person and which is the non-insurable person, and the rule is based on the rule, for example, the word closest to the insured person The person is called the guarantor.

When the text in the document image in FIG. 2A is classified into the corresponding fields in the database 30 by the document information extraction and filing system 100 of this embodiment, the brokers of the insurance company can use their insurance company The evaluation system 20 evaluates the existing insurance policies of potential customers and makes further recommendations. Compared with the conventional practice, brokers do not need to manually enter the information on existing insurance policies into the insurance company's evaluation system, which will save a lot of time and increase the efficiency of developing new customers.

It is worth noting that although the above example uses an existing insurance policy as the document image, those with ordinary knowledge in the field should know that the document information extraction and filing system in this case can also be applied to other types of documents, such as : Appointments, contracts, judgments, etc., without the use of human manual input, greatly improving the efficiency of administrative operations.

Although the present invention has been disclosed as above in preferred embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be subject to those defined by the attached patent scope.

30: Database

40: Video input device

100: Document information extraction and filing system

110: Input module

115: image pre-processing module

120: Text segmentation area detection module

122: The first type of neural network model

130: text recognition module

132: The second type of neural network model

140: Semantic Segmentation Module

150: Database docking module

151: text classifier

Claims (6)

A document information extraction and archiving system is electrically connected to a database. The document information extraction and archiving system includes: an input module that accepts a document image, the document image includes a plurality of text; a text segmentation area detection module, by A first type neural network model is used to frame the text in the document image to form at least one text segmentation area; a text recognition module uses a second type neural network model for the text segmentation area The character is recognized to obtain at least one editable character string, the character string includes at least one editable character; a semantic segmentation module, which hyphenates the character string to form a plurality of word segments, and assigns each word segmentation A part of speech; and a database docking module, according to the part of speech, the word segmentation and each field of the database are concatenated; wherein, the first type of neural network model includes a first convolutional neural network model And a target detection neural network model, the first convolutional neural network model performs feature extraction on the document image to output a feature vector, the target detection neural network model is based on the input of the feature vector for the serial number type The text is framed to form the serial number segmentation area; wherein, the first convolutional neural network model is VGG model, ResNet model, or DenseNet model; wherein, the target detection neural network model is YOLO model, CTPN Model, or EAST model; wherein the second type of neural network model includes a second convolutional neural network model and a recursive neural network model, and the second convolutional neural network model segmented the text Image in Processing is performed to output a text sequence, and the recurrent neural network model outputs the editable text string according to the input of the text sequence; wherein the recurrent neural network model implements the Connectionist Temporal Classification algorithm. A document information extraction and archiving system is electrically connected to a database. The document information extraction and archiving system includes: an input module that accepts a document image, the document image includes a plurality of text; a text segmentation area detection module, by A first type neural network model is used to frame the text in the document image to form at least one text segmentation area; a text recognition module uses a second type neural network model for the text segmentation area The character is recognized to obtain at least one editable character string, the character string includes at least one editable character; a semantic segmentation module, which hyphenates the character string to form a plurality of word segments, and assigns each word segmentation A part of speech; and a database docking module, according to the part of speech, the word segmentation and each field of the database are concatenated; wherein, the first type of neural network model includes a first convolutional neural network model And a target detection neural network model, the first convolutional neural network model performs feature extraction on the document image to output a feature vector, the target detection neural network model is based on the input of the feature vector for the serial number type The text is framed to form the serial number segmentation area; wherein, the first convolutional neural network model is VGG model, ResNet model, or DenseNet model; wherein, the target detection neural network model is YOLO model, CTPN Model, or EAST model; The second type of neural network model is the Seq2Seq model. For example, the document information extraction and filing system described in item 1 or item 2 of the scope of patent application, wherein the semantic segmentation module further includes a vocabulary and a rule module, wherein the vocabulary stores a plurality of proprietary domains Nouns, and the rule module is used to assign different parts of speech to the participle according to the characteristics of the participle. For example, the document information extraction and filing system described in item 1 or item 2 of the scope of patent application, wherein the semantic segmentation module vectorizes the word segmentation, and uses conditional random field or hidden Markov model to give each word segmentation a part of speech . For example, the document information extraction and filing system described in item 1 or item 2 of the scope of patent application, wherein the semantic segmentation module includes a third type neural network model, and the semantic segmentation module converts each character of the string into a Fix the dimensional feature vector, and input the feature vector into the third type neural network model to assign a part of speech to each word segmentation. In the document information extraction and filing system described in item 5 of the scope of patent application, the third type of neural network model is a recurrent neural network.

Images