KR20220031097A - Document processing methods, devices, devices and computer-readable storage media - Google Patents

Abstract

The present invention provides a document processing method, an apparatus, a device, and a computer-readable storage medium. The method includes: obtaining semantic characteristics and visual characteristics of a document to be processed; determining a currency characteristic of the to-be-processed document based on the semantic characteristic and the visual characteristic; and determining a category of the to-be-processed document based on a common characteristic of the to-be-processed document.

Description

Document processing methods, devices, devices and computer-readable storage media

[Cross Citation of Related Applications]

This application is filed on June 29, 2020, the application number is 202010610080.8, and claims priority to the Chinese application entitled "Document Processing Method, Apparatus, Apparatus and Computer-readable Storage Medium", the application All contents of are incorporated herein by reference.

[Technical field]

FIELD OF THE INVENTION The present invention relates to computer vision technology, and more particularly to a document processing method, apparatus, apparatus and computer readable storage medium.

Currently, documents are generally recognized by applying OCR (Optical Character Recognition) technology. When recognizing using this technology, the category of the document should be accurately obtained and the corresponding template should be used, but the classification result of the document by the related technology is not accurate.

Therefore, how to correctly classify documents is an urgent task.

An embodiment of the present invention provides a document classification scheme.

According to one aspect of the present invention, there is provided a document processing method, the method comprising: acquiring semantic characteristics and visual characteristics of a document to be processed; determining a currency characteristic of the to-be-processed document based on the semantic characteristic and the visual characteristic; and determining a category of the to-be-processed document based on a common characteristic of the to-be-processed document.

Combining any one embodiment provided in the present invention, the obtaining of the semantic characteristics of the to-be-processed document may include: obtaining a text recognition result of the to-be-processed document; and obtaining a semantic characteristic of the document to be processed based on the text recognition result.

Combining any one embodiment provided in the present invention, the step of obtaining a text recognition result of the to-be-processed document includes: determining a target text box in the to-be-processed document and text content included in the target text box; step; and obtaining a word separation processing result of text content in each of the target text boxes; and obtaining a feature vector corresponding to a result of the word separation processing.

Combining any one embodiment provided in the present invention, the step of determining a common characteristic of the to-be-processed document based on the visual characteristic and the semantic characteristic may include normalizing ( regularization) processing; and performing weighted summation on the normalized visual feature and the normalized semantic feature to obtain a common feature of the to-be-processed document.

Combining any one embodiment provided in the present invention, the document processing method is performed using a neural network, wherein the neural network includes a feature extraction sub-network for extracting common features of the document to be processed; a first classification subnetwork for determining the category of the to-be-processed document based on comparing the characteristics to determine a degree of similarity between a common characteristic of the document to be processed and a standard characteristic of the document of the at least one category; A category of the document to be processed is determined based on the obtained at least one degree of similarity.

Combining any one of the embodiments provided in the present invention, determining the category of the document to be processed based on the obtained at least one degree of similarity may include: acquiring a highest degree of similarity among the at least one degree of similarity; and determining, as the category of the document to be processed, a category of a document to which a standard feature corresponding to the highest similarity belongs in response to the highest similarity being greater than or equal to a preset similarity threshold.

Combining any one embodiment provided in the present invention, the method further comprises training a feature extraction subnetwork in the neural network, specifically, inputting a sample document into the feature extraction subnetwork, wherein the obtaining a currency characteristic of a sample document, wherein the sample document is labeled with a category; inputting the common feature into a second classification sub-network to obtain a prediction category of the sample document; and adjusting a network parameter of the feature extraction subnetwork based on a difference between the prediction category of the sample document and the labeling category of the sample document.

Combining any one embodiment provided in the present invention, the standard feature of the document of the at least one category is obtained by performing feature extraction on the document of the at least one category using a trained feature extraction sub-network. is obtained

Combining any one embodiment provided in the present invention, the method includes: in response to the highest similarity being less than the preset similarity threshold, adding the to-be-processed document as a standard template; The method further includes determining the prevailing characteristic of the document as a standard characteristic of a category corresponding to the newly added standard template.

Combining any one embodiment provided in the present invention, the method further includes, in response to the selection command, selecting at least one category from among preset document categories to be regarded as a target category; determining a similarity between the common feature of the document to be processed and the standard feature of the document of the at least one category by comparing the common feature of the to-be-processed document with the standard feature of the document of at least one category; Comparing a common characteristic of a document to be processed with a standard characteristic of a document of at least one preset target category, determining a degree of similarity between the common characteristic of the to-be-processed document and a standard characteristic of a document of the at least one target category do.

Combining any one embodiment provided in the present invention, the method may further include: obtaining a corresponding preset standard template based on the category of the document to be processed; and performing layout recognition processing on the document to be processed based on the standard template to obtain a layout recognition result of the document.

According to an aspect of the present invention, there is provided a document processing apparatus, the apparatus comprising: an acquiring module for acquiring semantic features and visual features of a document to be processed; a currency module for determining a currency characteristic of the to-be-processed document based on the semantic characteristic and the visual characteristic; and a classification module for determining a category of the to-be-processed document based on a common characteristic of the to-be-processed document.

According to one aspect of the present invention, there is provided a document processing device, the document processing device comprising a non-volatile storage medium and a processor, the storage medium storing computer instructions executable by the processor, the processor comprising: perform the method according to any one embodiment of

According to one aspect of the present invention, there is provided a computer readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method according to any one embodiment of the present invention is implemented.

According to an aspect of the present invention, there is provided a computer program, and when the computer program is executed by a processor, the method according to any one embodiment of the present invention is implemented.

The document processing method, apparatus, device, computer-readable storage medium and computer program of one or more embodiments of the present invention determine a currency characteristic of a document based on a visual characteristic and a semantic characteristic of the obtained document, and based on the category of the document. The document processing method of the present invention can implement accurate classification for any document; By combining semantic characteristics and visual characteristics to obtain common characteristics of documents, the accuracy of classification results of documents of different categories having similar visual characteristics is improved, and robustness of document classification is also improved.

The general description above and the description of the details in the back are illustrative and for the purpose of interpretation and not limitation of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The drawings herein are incorporated in and constitute a part of this specification, and these drawings illustrate embodiments consistent with the present invention and, together with the specification, serve to interpret the principles of the present specification.
1 is a flowchart of a document processing method shown in an embodiment of the present invention.
2 exemplarily shows a partial network structure of a neural network for extracting visual features according to an embodiment of the present invention.
3 exemplarily shows a partial network structure of a neural network for extracting semantic features according to an embodiment of the present invention.
4 is a schematic diagram of a text recognition process of a form shown in an embodiment of the present invention.
5 is a schematic diagram of a user selection interface shown in an embodiment of the present invention.
6 is a schematic diagram of a document processing apparatus shown in the embodiment of the present invention.
7 is a structural schematic diagram of the document processing apparatus shown in the embodiment of the present invention.

Examples will be described in detail herein, examples of which are shown in the drawings. When the following description includes drawings, like numbers in different drawings refer to the same or similar elements unless otherwise indicated. The embodiments described in the following examples do not represent all embodiments consistent with the present disclosure. Rather, these are merely examples of devices and methods consistent with some aspects of the present disclosure and as detailed in the appended claims.

The terminology used in the present invention is only for describing the purpose of a specific embodiment, and does not limit the present invention. As used in the present invention and the appended claims, the odd-numbered forms “a,” “the,” and “therefore” also include the plural forms, unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein includes any or all possible combinations including one or more correlated listed items.

Although various information has been described using the terms first, second, third, etc. in the present invention, it should be understood that such information is not limited to these terms. These terms are only used to distinguish between the same categories of information. For example, without departing from the scope of the present invention, the first information may be referred to as second information, and similarly, the second information may be referred to as first information. Depending on the context, the word “if” as used herein may be interpreted as “if…”, “when…” or “…in response to a decision.”

Currently, documents are generally recognized by applying OCR (Optical Character Recognition) technology. When recognizing using the above technique, the category of the document is accurately obtained and a corresponding template is used, but the classification result of the document using the related technique is not accurate.

Based on this, at least one embodiment of the present invention provides a document processing method. Referring to FIG. 1 , the process of the document processing method is illustrated, including steps S101 to S103.

Here, the document may include one or more of books, documents, forms, tickets, identification documents, and radio frequency (RF) cards, and specifically, for example, common texts, identification cards, bank cards, driving permits, Driver's licenses, passports, forms, receipts, business licenses, and handwritten documents. The document processing method may automatically recognize the category of the document, for example, automatically recognize one bank card as the bank card category, or automatically recognize one ID card as the ID category, or A receipt can be automatically recognized as a receipt category. It should be noted that, in the implementation process, there may be more than one document to be processed. In other words, the user can select batch processing or single processing of documents to be processed according to their own needs. In the implementation process, the processing process of each to-be-processed document during batch processing is similar to the processing process of the single to-be-processed document, and the processing process of the single to-be-processed document may be referred to. For convenience of description in the present invention, a single document to be processed has been described as an example, but this is not a limitation on the technical solution of the present invention.

In step S101, semantic characteristics and visual characteristics of the document to be processed are acquired.

In this step, there is no specific limitation on the precedence order of acquiring the semantic feature and acquiring the visual feature, that is, acquiring the semantic feature first and then acquiring the visual feature, or acquiring the visual feature first, then acquiring the semantic feature Alternatively, the semantic feature and the visual feature may be acquired at the same time.

In this step, a neural network can be used to extract the visual features of the document to be processed. Specifically, first, an initial feature of a document to be processed is extracted using a convolutional kernel (for example, a convolutional kernel of 3*3), and then the initial feature is converted into a plurality of (for example, 7) inverse residuals. Residuals) blocks to extract intermediate features in turn, and the intermediate features output by the last inverse residual block are again convolved through one convolution kernel (for example, a 1*1 convolution kernel), so that the specified dimension By outputting the features of , it can be used as a visual feature of the document to be processed. Each backward residual block consists of one 1*1 convolution kernel and an up channel module (extending the number of channels of the input feature) composed of an activation function (eg Relu6), one depth separable convolution An extraction module composed of a layer and an activation function (extracts the features of each channel and connects the features of each channel), and a descending channel module consisting of one 1*1 convolutional kernel (restores the number of channels of the input features) ) is included. Each reverse residual block is used as the output of the reverse residual block by summing its input and the output of the falling channel module. All outputs of each reverse residual block except for the last reverse residual block are used as inputs to the next reverse residual module.

In one example, FIG. 2 exemplarily shows a part of a network structure for extracting visual features of a document to be processed. Some network structures shown in FIG. 2 include two reverse residual blocks, a first reverse residual block 201 and a second reverse residual block 202 . The first reverse residual block 201 includes a first rising channel module 2011 , a first extraction module 2012 , and a first falling channel module 2013 connected in turn. Here, the first up channel module 2011 may be composed of, for example, one 1*1 convolution kernel (Conv1*1) and an activation function (eg, Relu6), and the first extraction module ( 2012) may consist, for example, of a deeply separable 3*3 convolutional layer (Dwise3*3) and an activation function (eg, Relu6), and the first falling channel module 2013 is, for example, It may be composed of one 1*1 convolutional kernel (Conv1*1). A first input of the first backward residual block 201 is an initial feature of a document to be processed, and may be obtained by extraction with, for example, a 3*3 convolution kernel. The first output of the first reverse residual block 201 is the sum of the first input and the output of the first falling channel module, and the first output is the second input of the second reverse residual block 202 . The second reverse residual block 202 includes a second rising channel module 2021 , a second extraction module 2022 and a second falling channel module 2023 connected in turn. Here, the second up-channel module 2021 may be composed of, for example, one 1*1 convolutional kernel (Conv1*1) and an activation function (eg, Relu6), and a second extraction module ( 2022) may be composed of, for example, a depth separable convolutional layer (Dwise3*3) and an activation function (eg Relu6), and the second falling channel module is, for example, one 1*1 It can be composed of a convolutional kernel (Conv1*1). The second output of the second reverse residual block 202 is the sum of the second input and the output of the second falling channel module.

In this step, the semantic characteristics of the document to be processed may be obtained using the following method. First, a text recognition result of the document to be processed is obtained. Then, based on the text recognition result, a semantic characteristic of the document to be processed is acquired.

Here, the text recognition result may be a result of extracting text content from a document to be processed and displaying it using a specific method. In one example, OCR technology may be used to obtain a text recognition result of a document to be processed.

Here, a semantic feature of the text recognition result can be extracted using a neural network. Specifically, it is possible to first extract features of different levels of the text recognition result, then connect and extract features of different levels, and finally obtain semantic features of the text recognition result.

Referring to FIG. 3 , in one example, intermediate features of text recognition results are first extracted using at least one third extraction module 301 , wherein each third extraction module 301 is a receptive field. field) may be different convolutional kernels. For example, using a convolution kernel with receptacle 1, a convolution kernel with receptacle 3, and a convolution kernel with receptacle 5, three different levels of features (e.g., convolutional After extracting (through manipulation and/or pooling, etc.), three different level features can be connected to obtain an intermediate feature. Then, using the fourth extraction module 302 (eg, a convolution kernel of 1*1), additional feature extraction (eg, through manipulations such as convolution and/or pooling) is performed on the intermediate features. to obtain the semantic characteristics of the text recognition result.

The feature extraction process corresponding to FIG. 3 is merely an example of extracting semantic features, and is not a specific limitation on a method of extracting semantic features of a text recognition result, and includes a convolutional kernel with a larger quantity or a smaller quantity and Different receptive field combinations can be used to extract different levels of features.

Here, the semantic characteristics of the document to be processed can be used to distinguish various documents having similar visual characteristics but different text contents, and the above-described various documents cannot be accurately classified into related technologies. The example solves this problem of the related art by adding a semantic feature.

In step S102, a current characteristic of the document to be processed is determined based on the semantic characteristic and the visual characteristic.

Here, in step S101, when the visual feature is extracted and the semantic feature is extracted, the visual feature and the semantic feature having the same dimension are outputted, thereby facilitating the fusion of the two features. Of course, this embodiment does not limit the dimensional relationship between the visual feature and the semantic feature extracted in step S101.

Here, step S101 may output visual features and semantic features of different dimensions when the visual features are extracted and the semantic features are extracted. In this case, after comparing the dimensions of two features, dimensionality reduction is performed on the higher-dimensional feature among the two features to make the dimensions of the two features the same, and then the fusion of the two features can be performed. there is. As the dimensionality reduction method, for example, linear dimensionality reduction and non-linear dimensionality reduction may be used.

In one example, first, normalization processing is performed on the visual feature and the semantic feature, respectively. Then, weighted summing is performed on the normalized visual feature and the normalized semantic feature to obtain a common feature of the document to be processed.

Other methods may be used to obtain common characteristics of the document to be processed, for example, after performing normalization or normalization on visual characteristics and semantic characteristics, weighted summing, or points By fusing semantic features and visual features using a bit-by-bit addition or vector splicing method, a common feature of a document to be processed is obtained, and the like.

In an embodiment of the present invention, by fusing the semantic feature and the visual feature of the to-be-processed document, it is possible to obtain the prevailing characteristic of the to-be-processed document. Here, the common feature of the document to be processed may be used for document classification in step S103, and may be used for document comparison to match document images.

In step S103, a category of the to-be-processed document is determined according to a common characteristic of the to-be-processed document.

In an embodiment of the present invention, a currency characteristic of the document is determined based on the obtained visual characteristics and semantic characteristics of the document, and a category of the document is determined based on the currency characteristic. The document processing method of the present invention can implement accurate classification for any document; By combining semantic characteristics and visual characteristics to obtain common characteristics of documents, the accuracy of classification results of documents of different categories having similar visual characteristics is improved, and robustness of document classification is also improved.

In some embodiments, a text recognition result of the to-be-processed document may be obtained through the following method.

First, a target text box in the document to be processed and text content included in the target text box are determined.

Then, a word separation processing result of the text content in each of the target text boxes is obtained.

Finally, a feature vector corresponding to the result of the word separation processing is obtained.

Referring to FIG. 4 , it shows a text recognition process of one to-be-processed document (ie, format). Through text recognition, target text boxes in the document to be processed, that is, 15 text boxes of 401 to 415, and text content included in each target text box are determined. For example, text box 401 contains an office supply billing table, text box 402 contains table creation time, year, month, and day, and text box 415 contains general manager's opinion. Word separation processing is performed on the text content in each text box to obtain a plurality of word separation processing results, for example, 11 word separation processing results of 416 to 426, that is, text content in the 15 text boxes. A partial word separation processing result obtained after performing word separation processing on . The word separation processing result may include letters or words, for example, the word separation processing results 416 (office work), 417 (goods), 418 (purchase request) and 419 (table) are the text in the text box 401 . The content is the result of four word separation processing obtained through word separation processing, and the word separation processing results 420 (table creation), 421 (hour), 422 (year), 423 (month) and 424 (day) are text boxes ( The text contents in 402) are the five word separation processing results obtained through word separation processing, and the word separation processing results 425 (general manager) and 426 (opinion) are the text contents in the text box 415 obtained through word separation processing. It is the result of two word separation processing. 427 to 438 are 12 feature vectors, and each feature vector is a result obtained by performing a feature vector display of a single word separation processing result.

In an embodiment of the present invention, a target text box in a document and a text content in the target text box are determined, and word separation processing and feature vector display are performed on the text content to obtain a text recognition result. In addition to extracting text content from a document (for example, part or all text content of a document), it is possible to obtain the minimum character/word unit in the text through text box segmentation and word separation processing, so that the semantic characteristics It is very accurate to make a decision about documents, and further improves the accuracy of document classification. In addition, since the text recognition result is a feature vector display, it is convenient to extract semantic features, and the efficiency of document classification is further improved.

In some embodiments, the document processing method may be performed using a neural network, wherein the neural network includes a feature extraction sub-network for extracting a common feature of the to-be-processed document, and a feature extraction sub-network of the to-be-processed document based on the common feature. and a first classification sub-network for determining a category. Here, the first classification sub-network specifically compares the common feature of the to-be-processed document with the standard feature of the document of at least one preset category, and compares the common feature of the to-be-processed document with the document of the at least one category. determine a degree of similarity of at least one standard feature of The category of the document to be processed may be determined based on at least one degree of similarity.

Here, since the dimensions of the common feature and the standard feature of the document to be processed may be the same, comparison of the common feature and the standard feature is convenient. The similarity between the common feature and the standard feature may be obtained by calculating the Euclidean distance between the two, or may be obtained through a single neural network that can output the similarity between the two, and the neural network is obtained through training.

In an embodiment of the present invention, standard characteristics of documents of each category are preset in the neural network. The category of the document to be processed is determined using the similarity of the standard characteristics different from the common characteristics of the document to be processed. The degree of similarity indicates the relationship between the document to be processed and the standard document of each category, that is, whether it is similar and the degree of similarity, so that the accuracy of the classification result is improved, the operation is simple, and the classification efficiency is further improved.

In some embodiments, determining the category of the document to be processed based on at least one degree of similarity specifically uses the following manner.

Then, the highest similarity among the at least one similarity is acquired.

Next, in response to the highest similarity being greater than or equal to a preset similarity threshold, the category of the document to which the standard feature corresponding to the highest similarity belongs is determined as the category of the document to be processed.

Here, each degree of similarity is compared to determine the highest degree of similarity. When at least two identical highest similarities appear, it may return to the step of calculating the similarities, recalculate the similarities with higher precision, and then compare the calculation results again to obtain one highest similarity. After one or several iterations, if at least two identical highest similarities are still included, iterative calculations are continued until only one highest similarity remains.

It should be explained that during the implementation process, first, by comparing the similarity with a preset similarity threshold, and selecting one or a plurality of similarities that are greater than or equal to the similarity threshold, the highest similarity among the selected similarities can be obtained. . As can be seen from this, the implementation method for determining the unique and highest similarity includes, but is not limited to, the above two situations, and in the implementation process, other implementation methods that can achieve the same or similar effect may be used. , which will not be explained here.

In this embodiment, only the similarity higher than the similarity threshold is recognized as a valid similarity, that is, only when the similarity between the common feature and the standard feature of the document to be processed is higher than or equal to the similarity threshold value, the document to be processed and the standard document is recognized as similar, and further, as the degree of similarity is greater than the similarity threshold, the degree of similarity between the document to be processed and the standard document is recognized as higher, and if the similarity between the common feature and the standard feature of the document to be processed is lower than the similarity threshold, However, it is recognized that the document to be processed and the standard document are not similar.

In an embodiment of the present invention, a similarity threshold is preset in the neural network. The highest similarity is compared with a similarity threshold, and only when the highest similarity is higher than the similarity threshold, the document to be processed is classified into a category corresponding to the standard document. Thereby, it is prevented that a classification error is generated when the degree of similarity between the general characteristics of the document to be processed and the overall standard characteristics is low, that is, when the document to be processed does not belong to a category corresponding to any one standard document. Furthermore, by improving the classification accuracy, the problem of misclassifying documents other than the preset category is prevented.

In some embodiments, the following scheme is used to train a feature extraction sub-network in the neural network.

First, a sample document is input to the feature extraction subnetwork to obtain a common feature of the sample document, wherein the sample document is labeled with a category.

Then, the common feature is input to a second classification sub-network to obtain a prediction category of the sample document.

Finally, according to the difference between the prediction category of the sample document and the labeling category of the sample document, the network parameter of the feature extraction subnetwork is adjusted.

Here, the network structure of the feature extraction sub-network allows to extract common features of an input document, and training of the feature extraction sub-network improves the accuracy of feature extraction.

Here, the second classification subnetwork is one classifier, and for example, it may be composed of at least one fully connected layer and a normalization layer. The number of categories classified by the second classification subnetwork is fixed, and corresponds to the category quantity of the sample document, for example, 5, 8, or 10, in other words, the output of the second classification subnetwork is each is the probability of the preset category of , and the one category with the highest probability is the classification result. For example, there are sample documents of all 10 categories, A, B, C, D, E, F, G, H, I, J, respectively, the output dimension of the second classification subnetwork is 10, the 10 It corresponds to each category. After the common feature of one sample document extracted by the feature extraction subnetwork is input to the second classification subnetwork, 10 probabilities are output by the second classification subnetwork, respectively, 83%, 2%, 1%, 3%, 0.5%, 0.2%, 0.3%, 5%, 4%, 1%, and the ten probabilities are that the sample document is A, B, C, D, E, F, G, H, I, Since it is a probability of category J, the prediction category for which the second classification subnetwork outputs sample probability is A.

Here, when the network loss value is less than a preset loss value threshold, the adjustment of the network parameters for the feature extraction sub-network may be stopped, and/or when the number of adjustments exceeds the preset number of times threshold, the Adjustment of network parameters for the feature extraction subnetwork may be stopped.

Here, a set of sample documents may be prepared in advance. First, a plurality of sample documents are acquired. Then, each category of each of the sample documents is marked. Finally, a sample document set is determined based on the plurality of sample documents marked with categories. In addition, one of the sample documents may be selected and used as a standard template of the category document, and the standard features may be subsequently stored and used.

In an embodiment of the present invention, the extraction capability of the feature extraction sub-network determines the accuracy of the extracted common features, and the accuracy of the common features also determines the accuracy of the classification result, so that the prediction category output by the second classification subnetwork is The accuracy of may indicate the strength or weakness of the feature extraction subnetwork extraction capability. By implementing the indication of the extraction capability of the feature extraction subnetwork by the second classification subnetwork, feedback and adjustment of the network parameters of the feature extraction subnetwork, and continuously optimizing the network parameters, improve the extraction capability of the feature extraction subnetwork In addition, the accuracy of the extracted common features is improved, and the accuracy of document classification is improved.

In some embodiments, the standard feature of the document of the at least one category is obtained by processing the standard template of the document of the at least one category, using a trained feature extraction subnetwork.

Here, after the feature extraction subnetwork training is completed, it has the ability to accurately extract common features of documents input into the interior. First, the standard template of the document of each category can be determined, the layout of the standard template is clear, the boundaries of text boxes and/or text blocks are clear, the text content is complete, and the general characteristics of the standard template of the document of each category are clear. After extracting, it is stored as a standard feature of the document of the above category. Labeling may be performed on a standard template, ie by performing labeling on attributes such as each position, text box and/or text block of the standard template, the standard template will be used to perform document layout recognition. can

In the embodiment of the present invention, since both the standard template and the common document of the document to be processed are extracted using the feature extraction subnetwork, the common feature and the standard feature have the same origin, and the rules and standards match, so that the The similarity is more accurate, further improving the accuracy of document classification.

Standard features stored through the above method are limited and cannot include categories of all documents. Further, according to the description of some embodiments described above, only when the highest similarity threshold is greater than or higher than the similarity threshold, the document to be processed may be classified into a document category corresponding to the highest similarity. Based on the causes of the above two aspects, if the category of one document is not included in the preset standard template, the classification cannot be completed.

Accordingly, in some examples, standard features are added using the following scheme.

In response to the highest similarity being smaller than a preset similarity threshold, the document to be processed is added as a standard template, and a common characteristic of the to-be-processed document is determined as a standard characteristic of a corresponding category of the newly added standard template.

Here, that the highest similarity is smaller than the similarity threshold value indicates that the document to be processed does not belong to any one preset document category, that is, the document to be processed is a new document category. When classification fails, the document to be processed for which classification is not completed is stored as a new category in the neural network, that is, the document to be processed is stored as a standard template, and the extracted common feature is stored as a standard feature of the new category document. In addition, after the category is stored, it is possible to generate notification information, perform labeling on a standard template of the category, and inform the user so that it can be used for layout recognition.

In an embodiment of the present invention, since the feature extraction subnetwork can accurately extract common features of the document to be processed, the first classification subnetwork can automatically expand the classification dimension or quantity.

In an embodiment of the present invention, by storing and setting a document to be processed that has failed to be classified as one new category, it is possible to automatically expand the quantity of the preset document category, and to continuously improve the classification capability.

In some embodiments, the method further comprises, in response to the selection command, selecting at least one category from among the preset document categories to be regarded as the target category. Here, the selection command may be triggered by a user's selection operation, or may be automatically triggered when a trigger condition is preset and the trigger condition is satisfied.

A degree of similarity between a common characteristic of the document to be processed and a standard characteristic of a document of the at least one category is determined using the following scheme. A common feature of the to-be-processed document is compared with a standard feature of a document of at least one target category to determine a degree of similarity between the common feature of the to-be-processed document and the standard feature of the document of the at least one target category.

In one example, referring to FIG. 5 , it shows some contents of one user selection interface, and document categories preset from the drawing are common characters, identification cards, bank cards, driving licenses, driver's licenses, passports, common forms, additions It can be seen that VAT receipts, business registration certificates and handwritten texts are included, and the user selects identification cards, bank cards, currency forms, VAT receipts and handwritten texts as target categories through manipulation. In this case, in the process of performing processing based on a document to be subsequently recognized, a plurality of categories selected by the user are used as references.

It should be explained that the content shown in FIG. 5 is only one possible implementation form, and in an actual application process, the user actively creates a template, builds a new target category, and processes a document to recognize the new target category It can also be used as a reference for In addition, the target category may include at least some of the various categories shown in FIG. 5 , that is, more or fewer than those shown in FIG. 5 , but the present disclosure is not limited thereto.

The present invention further provides a document processing apparatus, referring to FIG. 6 , which shows the structure of the apparatus, the apparatus comprising: an acquiring module 601 for acquiring semantic features and visual features of a document to be processed; a currency module (602) for determining a currency characteristic of the to-be-processed document based on the semantic characteristic and the visual characteristic; and a classification module 603 for determining a category of the to-be-processed document based on a common characteristic of the to-be-processed document.

In some embodiments, the acquiring module is specifically configured to: acquire a text recognition result of the document to be processed; Based on the text recognition result, a semantic characteristic of the document to be processed is acquired.

In some embodiments, obtaining a text recognition result of the to-be-processed document may include: determining a target text box in the to-be-processed document and text content included in the target text box; obtaining a word separation processing result of text content in each of the target text boxes; and obtaining a feature vector corresponding to a result of the word separation processing.

In some embodiments, the currency module specifically performs normalization processing on the visual feature and the semantic feature, respectively; Weighted summing is performed on the normalized visual feature and the normalized semantic feature to obtain a common feature of the document to be processed.

In some embodiments, the document processing apparatus includes a neural network, wherein the neural network is configured to: a feature extraction sub-network for extracting a common feature of the to-be-processed document; and to determine a category of the to-be-processed document based on the common feature and a first classification sub-network for: specifically, comparing the common characteristics of the to-be-processed document with a standard characteristic of a document of at least one preset category, the common characteristics of the to-be-processed document and a degree of similarity of standard features of documents of the at least one category; A category of the document to be processed is determined based on the obtained at least one degree of similarity.

In some embodiments, when the first classification sub-network determines the category of the document to be processed based on the obtained at least one degree of similarity, specifically, obtain a highest degree of similarity among the at least one degree of similarity; In response to the highest similarity being greater than or equal to a preset similarity threshold, the category of the document to which the standard feature corresponding to the highest similarity belongs is determined as the category of the document to be processed.

In some embodiments, the apparatus further comprises a training module for training a feature extraction subnetwork in the neural network, which inputs a sample document to the feature extraction subnetwork to obtain common features of the sample document; The sample document is labeled with categories; inputting the common feature into a second classification sub-network to obtain a prediction category of the sample document; According to a difference between the prediction category of the sample document and the labeling category of the sample document, the network parameter of the feature extraction sub-network is adjusted.

In some embodiments, the standard feature of the document of the at least one category is obtained by performing feature extraction on the document of the at least one category, using a trained feature extraction subnetwork.

In some embodiments, the device, in response to the highest similarity being less than the preset similarity threshold, adds the to-be-processed document as a standard template, and adds a common characteristic of the to-be-processed document to the newly added standard template. and an extension module for determining as a standard feature of a corresponding category.

In some embodiments, the apparatus further comprises: a target module for, in response to the selection command, selecting at least one category from among preset document categories to be regarded as a target category; The first classification sub-network compares a common feature of the to-be-processed document with a standard feature of a document of at least one preset category, and the degree of similarity between the common feature of the to-be-processed document and the standard feature of the document of the at least one category Specifically, by comparing the common characteristics of the document to be processed with the standard characteristics of the document of at least one target category, the common characteristics of the document to be processed and the standard characteristics of the document of the at least one target category are specifically compared. determine the similarity of

In some embodiments, the device is configured to: obtain a corresponding preset standard template according to the category of the document to be processed; and a recognition module for performing layout recognition processing on the document to be processed based on the standard template to obtain a layout recognition result of the document.

The present invention further provides a document processing device, and with reference to FIG. 7 , which shows the structure of the document processing device, the document processing device includes a non-volatile storage medium (701) and a processor (702), the The storage medium 701 stores computer instructions executable by the processor 702, and the processor 702, when the computer instructions are executed, implements the method according to any one embodiment of the present invention.

The present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method according to any one embodiment of the present invention is implemented.

In an embodiment of the present invention, when a document to be processed is classified using the classification method of the present embodiment based on documents of various categories that are already known, at least one target category among these various categories is selected as a reference, so that the degree of similarity It is possible to reduce the computational load of the step of determining and the computational load of the step of comparing the similarity, and improve the efficiency of classification.

In some embodiments, obtaining a corresponding preset standard template according to the category of the document to be processed; The method further includes performing layout recognition processing on the document to be processed based on the standard template to obtain a layout recognition result of the document.

Here, by performing layout recognition by automatically and accurately calling a corresponding standard template through the classification result, not only the accuracy of layout recognition is improved, but also the efficiency of layout recognition is improved.

Those of ordinary skill in the art should understand that one or more embodiments of the present specification may be provided as a method, a system, or a computer program product. Accordingly, one or more embodiments of the present specification may use a complete hardware embodiment, a complete software embodiment, or a combination of software and hardware. In addition, one or more embodiments of the present specification may be stored in one or more computer-usable storage media (including but not limited to magnetic disk memory, CD-ROM and optical memory, etc.) including computer-usable program code. The form of an implemented computer program product may be used.

Each embodiment of the present specification is described gradually, and the same or similar parts between each embodiment may be referred to each other, and the point mainly described in each embodiment is a point to be distinguished from other embodiments. In particular, in the case of the apparatus embodiment, since it is almost similar to the method embodiment, the description is relatively simple, and relevant parts may refer to some descriptions of the method embodiment.

The foregoing has described specific embodiments of the present specification. Other embodiments are included within the scope of the appended claims. In some cases, acts or steps recited in the claims may be performed in a different order than the embodiments and still achieve desired results. In addition, the processes described in the drawings do not necessarily have to be performed in the specific order shown or in a continuous order to achieve a desired result. In some embodiments, multi-task processing and parallel processing are also possible and may be advantageous.

Examples of the subject matter and functional operations described herein are implemented in digital electronic circuits, tangible computer software or firmware, computer hardware, which may include the structures disclosed herein and structural equivalents thereof, or implementation in one or more of these. can be Examples of subject matter described herein are as one or more computer programs, ie, one or more units in computer program instructions executed by a data processing device or encoded in a tangible non-transitory program carrier for controlling the operation of the data processing device. can be implemented. Alternatively or in addition, the program instructions may be generated by an artificially generated, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information and transmit it to a suitable receiver device for execution by a data processing apparatus. It can be encoded into a radio signal. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof.

The processing and logic flows described herein may be executed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flows may also be executed by dedicated logic circuits such as Field Programmable Gate Arrays (FPGAs) or Application Specific Integrated Circuits (ASICs), and the device may also be implemented as dedicated logic circuits.

A computer suitable for executing a computer program may include, for example, a general purpose and/or special target microprocessor, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory and/or a random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, magneto-optical disk, or optical disk, or the like, or the computer receives data or transmits data to it, or both. will be operatively combined with this mass storage device for However, the computer need not have such equipment. A computer may also be a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or other device such as a universal serial bus (USB) flash drive, to name a few. can be embedded in

A computer-readable medium suitable for storing computer program instructions and data includes all forms of non-transitory memory, media and memory devices, such as semiconductor memory devices (eg, EPROMs, EEPROMs and flash memory devices). , magnetic disks (eg, internal hard disks or removable disks), magneto-optical disks, CD ROMs and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into dedicated logic circuitry.

Although this specification contains many specific implementation details, these should not be construed as limiting any disclosed or claimed scope, but are primarily used to describe features of the specific examples disclosed. Certain features described in multiple examples herein may also be implemented in combination in a single example. On the other hand, various features described in a single example may also be implemented in a plurality of examples individually or in any suitable sub-combination. Also, although features may function in certain combinations as described above and even as originally claimed, one or more features from a claimed combination may in some cases be eliminated from the combination, and the claimed combination is - may refer to a variant of a combination or sub-combination.

Similarly, although acts are shown in a particular order in the figures, this is to be interpreted as requiring that such acts be performed in the specific order or sequentially shown, or requiring all illustrated acts to be performed to achieve a desired result. shouldn't be In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system units and components in the examples should not be construed as requiring such separation in all examples, and the described program components and systems are generally integrated into a single software product, or a plurality of software products. It should be understood that they can be packaged into

Accordingly, specific examples of the subject matter have been described. Other examples are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desired results. Moreover, the processes depicted in the figures need not be in the particular order shown or sequential order to achieve a desired result. In some implementations, multitasking and parallel processing may be advantageous.

Claims (21)

A document processing method comprising:
obtaining semantic characteristics and visual characteristics of the document to be processed;
determining a currency characteristic of the to-be-processed document based on the semantic characteristic and the visual characteristic; and
determining a category of the to-be-processed document based on a common characteristic of the to-be-processed document;
Document processing method, characterized in that. According to claim 1,
Obtaining the semantic characteristics of the document to be processed includes:
obtaining a text recognition result of the document to be processed; and
based on the text recognition result, acquiring a semantic characteristic of the document to be processed
Document processing method, characterized in that. 3. The method of claim 2,
The step of obtaining a text recognition result of the document to be processed includes:
determining a target text box in the document to be processed and text content included in the target text box;
obtaining a word separation processing result of text content in each of the target text boxes; and
obtaining a feature vector corresponding to the result of the word separation processing;
Document processing method, characterized in that. According to claim 1,
The step of determining a common characteristic of the document to be processed based on the semantic characteristic and the visual characteristic,
performing regularization processing on the visual feature and the semantic feature, respectively; and
performing weighted summation on the normalized visual feature and the normalized semantic feature to obtain a common feature of the document to be processed
Document processing method, characterized in that. 5. The method according to any one of claims 1 to 4,
The document processing method is performed using a neural network, wherein the neural network includes a feature extraction sub-network for extracting a common feature of the to-be-processed document, and a first for determining a category of the to-be-processed document based on the common feature. a classification subnetwork, wherein the first classification subnetwork comprises:
comparing the common feature of the to-be-processed document with a standard feature of a document of at least one category to determine a degree of similarity between the common feature of the to-be-processed document and the standard feature of the document of the at least one category;
determining the category of the document to be processed based on the obtained at least one degree of similarity;
Document processing method, characterized in that. 6. The method of claim 5,
Determining the category of the document to be processed based on the obtained at least one degree of similarity comprises:
acquiring the highest similarity among the at least one similarity; and
In response to the highest similarity being greater than or equal to a preset similarity threshold, determining a category of a document to which a standard feature corresponding to the highest similarity belongs as the category of the document to be processed
Document processing method, characterized in that. 7. The method according to claim 5 or 6,
The method further comprises training a feature extraction sub-network in the neural network, wherein training a feature extraction sub-network in the neural network comprises:
inputting a sample document into the feature extraction sub-network to obtain a common feature of the sample document, wherein the sample document is labeled with a category;
inputting the common feature into a second classification sub-network to obtain a prediction category of the sample document; and
adjusting a network parameter of the feature extraction subnetwork based on a difference between a prediction category of the sample document and a labeling category of the sample document
Document processing method, characterized in that. 8. The method of claim 7,
The standard feature of the document of the at least one category is obtained by performing feature extraction on the document of the at least one category using a trained feature extraction sub-network.
Document processing method, characterized in that. 9. The method according to any one of claims 6 to 8,
In response to the highest similarity being smaller than the preset similarity threshold, adding the to-be-processed document as a standard template, and determining a common feature of the to-be-processed document as a standard feature of a category corresponding to the newly added standard template further comprising
Document processing method, characterized in that. 10. The method according to any one of claims 5 to 9,
in response to the selection command, further comprising selecting at least one category from among the preset document categories to be regarded as a target category;
Comparing the common feature of the document to be processed with the standard feature of the document of at least one category to be processed, and determining the similarity between the common feature of the document to be processed and the standard feature of the document of the at least one category,
Comparing a common characteristic of the to-be-processed document with a standard characteristic of a document of at least one preset target category to determine a degree of similarity between the common characteristic of the to-be-processed document and a standard characteristic of the document of the at least one target category doing
Document processing method, characterized in that. 11. The method according to any one of claims 1 to 10,
obtaining a corresponding preset standard template according to the category of the document to be processed; and
Based on the standard template, performing layout recognition processing on the document to be processed, further comprising the step of obtaining a layout recognition result of the document
Document processing method, characterized in that. A document processing apparatus comprising:
an acquiring module for acquiring semantic features and visual features of the document to be processed;
a currency module for determining a currency characteristic of the to-be-processed document based on the semantic characteristic and the visual characteristic; and
a classification module for determining a category of the to-be-processed document based on a common characteristic of the to-be-processed document;
Document processing apparatus, characterized in that. 13. The method of claim 12,
The acquisition module is
obtain a text recognition result of the document to be processed;
to obtain a semantic characteristic of the document to be processed based on the text recognition result
Document processing apparatus, characterized in that. 14. The method of claim 13,
Obtaining the text recognition result of the document to be processed comprises:
determining a target text box in the document to be processed and text content contained in the target text box;
obtaining a word separation processing result of text content in each of the target text boxes; and
acquiring a feature vector corresponding to the word separation processing result
Document processing apparatus, characterized in that. 13. The method of claim 12,
The common module is
perform normalization processing on the visual feature and the semantic feature, respectively;
performing weighted summation on the normalized visual feature and the normalized semantic feature to obtain a common feature of the document to be processed
Document processing apparatus, characterized in that. 16. The method according to any one of claims 12 to 15,
The document processing apparatus includes a neural network, wherein the neural network includes a feature extraction sub-network for extracting a common feature of the document to be processed, and a first classification sub-network for determining a category of the document to be processed based on the common feature. a network, wherein the first classification sub-network comprises:
comparing the common feature of the to-be-processed document with a standard feature of a document of at least one category to determine a degree of similarity between the common feature of the to-be-processed document and the standard feature of the document of the at least one category;
determining the category of the document to be processed based on the obtained at least one degree of similarity;
Document processing apparatus, characterized in that. 17. The method of claim 16,
When the first classification sub-network determines the category of the document to be processed based on the obtained at least one degree of similarity,
acquire a highest similarity among the at least one similarity;
In response to the highest similarity being greater than or equal to a preset similarity threshold, determining the category of the document to which the standard feature corresponding to the highest similarity belongs as the category of the document to be processed; or
In response to the highest similarity being smaller than the preset similarity threshold, adding the to-be-processed document as a standard template, and determining a common feature of the to-be-processed document as a standard feature of a category corresponding to the newly added standard template
Document processing apparatus, characterized in that. 18. The method of claim 16 or 17,
a target module for selecting at least one category from among preset document categories to be regarded as a target category in response to the selection command;
The first classification sub-network compares a common feature of the to-be-processed document with a standard feature of a document of at least one preset category, and the degree of similarity between the common feature of the to-be-processed document and the standard feature of the document of the at least one category If you decide
determining a degree of similarity between the common feature of the document to be processed and the standard feature of the document of the at least one target category by comparing a common feature of the to-be-processed document with a standard feature of a document of at least one preset target category
Document processing apparatus, characterized in that. A document processing device comprising:
The document processing device includes a non-volatile storage medium and a processor, wherein the storage medium stores computer instructions executable by the processor, wherein the processor executes the computer instructions. to implement a method according to
Document processing device, characterized in that. A computer-readable storage medium having a computer program stored therein, comprising:
12. When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented
A computer-readable storage medium, characterized in that. A computer program comprising:
12. When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented
A computer program characterized in that.

Images