RU2744769C1 - Method for image processing using adaptive technologies based on neural networks and computer vision - Google Patents

Abstract

FIELD: image processing.

SUBSTANCE: group of inventions relates to image processing, in particular to recognition of the type of document and its content from a stream of arbitrary images. The method includes the steps of extracting a document image using machine vision algorithms from a set of arbitrary images; determining the proposed type of document using a convolutional neural network based on the extracted document image, while the convolutional network sets the degree of similarity of the document image with known document types based on the specified document samples; determining the document type using a recurrent neural network based on the proposed document type received from the convolutional neural network, while the recurrent neural network evaluates the degree of similarity of the document with the intended document type based on machine learning algorithms. Data contained in a document are recognized based on a specific type of document; the recognized data are saved.

EFFECT: group of inventions ensures correct recognition of the document and restoring its content from the stream of images, even in the presence of illegible text or its absence on a part of the document.

40 cl, 2 dwg, 3 tbl

Description

The technical field to which the invention relates

The present invention relates to image processing by an unskilled user. In particular, the invention relates to the recognition and classification of documents from a stream of arbitrary images based on computer vision algorithms and neural networks.

State of the art

Modern smartphones allow you to quickly capture images and save them in large quantities. Often in quantities much larger than the user is able to sort, process. The functionality of such devices is constantly increasing, the number of raw photos is growing, and it becomes almost impossible for the user to find a photo that he took a few months or a year ago.

A separate task is to take into account photographs of any documents or information materials. The need for stricter classification and information extraction from such photographs is obvious.

For example, a receipt for a product becomes unreadable after a while due to the fact that it is printed on thermal paper, and in order to avoid situations, it is recommended to make copies of receipts, both paper and electronic, which can be printed out at any time. However, due to the large number of images, finding a photo of a receipt taken more than a year ago seems to be a time consuming and often insoluble task.

In addition, the likelihood of using a copy of each receipt or other document or material is small, so the user is not ready to independently enter various attributes and parameters of the photographed document.

In this regard, there is a need for a virtual assistant on the phone, implemented, for example, in the form of a mobile application, which would independently, without involving the user, detect documents, determine their type, recognize text and other important characteristics (QR code and others) ...

Such an assistant could not only select documents and materials from the stream of images, but also make a decision for the user on a number of fields found in the document: set reminders without his participation by the expiration date of the CTP, foreign passport and others.

At the moment, the user has a set of various programs and technical means that are capable of performing some of the indicated actions. In particular, it is known to use neural networks either for recognizing an image in an image, or for text recognition. However, there is no way to instruct the virtual assistant to perform these actions.

In addition, technologies for classifying and extracting information from images are in demand not only on a smartphone, but also on other mobile personal devices, such as augmented reality glasses, and, in addition, on non-mobile and non-personal devices such as a corporate scanner, CCTV cameras ` carrying out the capture and fixation of documents.

Disclosure of the essence of the invention

The subject of the invention is an information system that processes the image of documents. At the input, this information system receives an image in electronic form, and at the output - a block of information containing: an improved image with corrected geometry and color correction; document type determined based on machine learning algorithms; the content of the fields of the document, depending on the previously defined type of document, supplemented with information from other sources, if necessary. The information system has the ability to determine the type of document based on neural network algorithms, as well as correct text recognition errors through cascade image processing. This ability is realized using the neural network training method; in particular, the neural network independently deduces a formula by which it determines the degree of similarity of the analyzed image with a document of one type or another.

The essence of this invention lies in the use of a cascade of recognition technologies to ensure complete automation of the process and autonomy from the user. These technologies are adaptive and decisions are made based on many parameters, including but not limited to:

- the general appearance of documents of one kind or another;

- ways of photographing or fixing, in a broader sense, the type of documents of one type or another;

- the text of the document amenable to recognition;

- barcodes, QR codes, other machine-readable information on the document;

- information extracted from documents of other types, but belonging to the same user;

- the context in which the image was taken, including previous images;

- various verification databases obtained from open sources, including the concept of open access to government data.

Brief Description of Drawings

The accompanying drawings illustrate how the virtual assistant works and how to detect documents in an image stream. In the drawings:

1 is a block diagram depicting a virtual assistant and some examples of context sources that it can handle;

Fig. 2 is a block diagram of document detection and processing;

Implementation of the invention

Preferred embodiments of the invention are presented below.

In one embodiment, the claimed invention is implemented on a smartphone as a virtual assistant. The general operating principles of the virtual assistant are shown in FIG. one.

The virtual assistant processes the images of documents received either directly from the user (photographing), or independently with the appropriate permission from the user. After processing, the virtual assistant classifies the processed images by the type of document they contain (checks, tickets, contracts, insurance policies, etc.).

FIG. 2 shows a general algorithm for recognizing the type and parameters of a document in one embodiment of the invention.

Document image processing begins in step 1.

First, preprocessing of the document is performed. At stage 2, it is checked that this image is not a previously recognized document. If the image is a duplicate of an already recognized document, the document is ignored.

If the document is not a duplicate, then at stage 3, the boundaries of the document, that is, a paper sheet or other base on which the document is printed / depicted, will be determined. After clipping the image outside the bounds of the document base, in step 4, the geometry of the image is corrected, for example, trapezoidal distortion correction or angle correction.

In step 5, it stores the initial key color parameters so that in step 6 it can perform color corrections, if necessary, as well as optimize contrast and brightness for better text recognition. The sharpness of the image is also adjusted.

Thereafter, at step 7, an initial attempt is made to recognize text on the image, and the up-down orientation is determined (step 8). In step 9, it is determined whether the top and bottom of the document was detected. In case of failure in text recognition, attempts are made to find lines of text without trying to read them (step 10), and a decision is also made about whether the previous processing of the document from the same user is similar in parameters and what decision about the orientation of the document was made as a result of the previous processing (step 11). Further, based on the collected information, the document is rotated in accordance with a certain orientation (step 12).

Then, in step 13, the convolutional neural network performs primary recognition of the document type. The convolutional network sets the degree of similarity of a given image with all known types of documents on the basis of typical images of this type of documents known to it in advance.

The convolutional network then tries to find images it knows (non-textual features) on the document, including a face image (step 14), a QR code or barcode (step 15), a coat of arms, a logo (step 16), and other non-textual information, as well as extract information about coordinates and proportions of such information.

At the next stage, all information received about the document is packed and transmitted to a recurrent neural network, including

- "opinion" of a convolutional network in the form of a tuple with certain values of the likelihood of similarity,

- extracted unstructured text obtained during primary processing,

- the presence of stop words and their potential modifications associated with low image quality,

- information about the date, time, place of photographing,

- information about the key color parameters of the image

- information about the presence of non-text elements and their characteristics

At stage 17, the recurrent neural network exposes its estimates of the degree of similarity of the document to a certain type of document based on machine learning algorithms.

In step 18, a template corresponding to the document type is applied to recognize the document.

At step 19, it is determined whether the quality of the recognized text is sufficient. If the quality is not enough, then at step 20, the document is rotated and the quality of the recognized text is checked again.

At step 21, a refined template for high-quality text recognition is applied, based on the document type, taking into account the relative position of the text elements, their color, font, and other features. This stage, in particular, allows you to separate the font from the background image, which was not possible in the previous stage. The extracted blocks of text are saved as a field-value structure.

Further, for some types of documents, the information extracted from the document is enriched (step 22). Additional information can be obtained from external sources. So, for example, to obtain information about a cashier's check, an appeal is made to the Open API of the Federal Tax Service of Russia. For other documents, for example SNILS, the full name of the owner of the phone is compared with the full name on the document, and if the differences are in a small number of characters, and which, according to statistics for this font, are similar, then the full name of the owner of the smartphone is added as a candidate for correction.

Further, at step 22, the document is analyzed for whether it is multi-page due to the presence of features characteristic of the type of documents, such as page numbers, text concatenation, the presence of the same passport number on images of different passport pages. For this, previously entered images are analyzed.

If it is determined that the images refer to the same document, then at step 24 it is analyzed if this image is the same page of the document. If it is found that this is the same page of a multi-page document, then the page is selected with better text recognition and higher overall image sharpness. The images from which the multi-page document was composed are deleted (step 26).

If it is found that this is another page of a multi-page document, then at step 25 it is determined whether to create a multi-page document or to merge several images into one. For example, a sales receipt can be very long and the user needs to take several photos in order to fill in the entire receipt. In the first case, a new page is added to the multipage document (step 27), and in the second case, several images are glued into one.

CNN trained convolutional neural network uses InceptionV3 architecture. This network analyzes the image and, based on the weighting factors stored in it, determines the visual similarity of the current image with an array of document images of one type or another. The network produces a tuple consisting of scalar values of the probabilities of the image coincidence. These probabilities range from 0% to 100%.

In one embodiment, a convolutional neural network can determine the probability that a document is of a particular type using the following parameter values shown in Table 1:

Table 1

Parameter Parameter description Value Epoch Number of complete iterations over the entire dataset 200 BatchSize Dataset size with mini training ten LearningRate Learning rate ratio 0.01

A recurrent neural network (RNN), also based on TensorFlow Inception, analyzes many parameters of a given image, including the visual similarity decisions made by the CNN neural network. The input of this neural network is a set of numeric and string values, including the data obtained as a result of the first pass of the text recognition module. Further, through the use of a multi-class classifier and a multinomial logistic regression algorithm, a comparison is made with the coefficients previously obtained by machine learning.

In one embodiment, the recurrent neural network determines the probabilistic document type using the following formula:

,

,

Where

- T - the probabilistic value of the priority document type

- Max U [] - function of finding the maximum in an array of scalar values, each of which is represented by the sum of N weights.

- W _ik * F _ik - scalar probability value for the k-th parameter of the i-th document type

- W _ik - the weighting factor for the k-th parameter of the i-th document type

- F _ik - the k-th parameter (factor) of the i-th document type, represented by the convolution function of the corresponding input parameter

An example of the parameters of the weighting factors is presented in Table 2.

table 2

Parameter Description A type cnn_document_type The result of processing a document via CNN Line b_pattern_rf The presence of the phrase ‘RUSSIAN FEDERATION’ Boolean b_pattern_birthdate Presence of the phrase ‘Date of birth’ Boolean b_pattern_birthplace The presence of the phrase ‘Place of birth’ Boolean b_pattern_sign Presence of the phrase 'Signature of the owner' Boolean b_pattern_lname The presence of the phrase ‘Last name’ Boolean b_pattern_fname The presence of the phrase ‘Name’ Boolean b_pattern_mname The presence of the phrase ‘Patronymic’ Boolean b_pattern_insurpolice Presence of the phrase 'Insurance policy' Boolean b_psprt_issued The presence of the phrase ‘Passport issued’ Boolean b_psprt_branch The presence of the phrase ‘DEPARTMENT OF THE FMS OF RUSSIA’ Boolean b_psprt_branch The presence of the phrase ‘DEPARTMENT OF THE INTERIOR’ Boolean b_psprt_branch The presence of the phrase ‘DEPARTMENT OF THE FMS OF RUSSIA’ Boolean b_psprt_branch The presence of the phrase ‘OUFMS RUSSIA’ Boolean b_psprt_isdate The presence of the phrase ‘Date of issue’ Boolean b_psprt_issuer The presence of the phrase ‘Subdivision code’ Boolean b_psprt_code The presence of the phrase ‘Personal code’ Boolean b_psprt_sign The presence of the phrase ‘Personal signature’ Boolean b_psprt_gendate The presence of the phrase ‘Gender + date pattern’ Boolean b_face Having a face image Boolean n_pattern_rf Location of the phrase ‘RUSSIAN FEDERATION’ in relation to the document Integer n_pattern_birthdate Location of the phrase ‘Date of birth’ in relation to the document Integer n_pattern_birthplace Location of the phrase ‘Place of birth’ in relation to the document Integer n_pattern_sign Location of the phrase 'Owner's signature' in relation to the document Integer n_pattern_lname Location of the phrase ‘Last name’ in relation to the document Integer n_pattern_fname Location of ‘Name’ in relation to document Integer n_pattern_mname Location of the phrase ‘Patronymic’ in relation to the document Integer n_pattern_insurpolice Location of the phrase ‘Insurance policy’ in relation to the document Integer n_psprt_issued Location of the phrase 'Passport issued' in relation to the document Integer n_psprt_branch Location of the phrase ‘DEPARTMENT OF THE FMS OF RUSSIA’ in relation to the document Integer n_psprt_branch Location of the phrase ‘DEPARTMENT’ in relation to the document Integer n_psprt_branch Location of the phrase ‘DEPARTMENT OF THE FMS OF RUSSIA’ in relation to the document Integer n_psprt_branch Location of the phrase ‘OUFMS RUSSIA’ relative to the document Integer n_psprt_isdate Location of the phrase ‘Date of issue’ in relation to the document Integer n_psprt_issuer Location of the phrase ‘Department code’ in relation to the document Integer n_psprt_code Location of the phrase ‘Personal code’ in relation to the document Integer n_psprt_sign Location of the phrase ‘Personal signature’ in relation to the document Integer n_psprt_gendate Location of the phrase ‘Gender + date pattern’ relative to the document Integer n_face Location of the face image relative to the document Integer

An example of the values of the weighting factors is presented in Table 3.

Table 3

Parameter Document types T1 T2 T3 T4 T5 T6 T7 b_pattern_rf five% 0% 0% 0% 0% 0% 0% b_pattern_birthdate 43% 0% 0% 0% 0% 0% 0% b_pattern_birthplace 85% 0% 0% 0% 0% 0% 0% b_pattern_sign 82% 0% 0% 0% 0% 0% 0% b_pattern_lname 47% 0% 0% 0% 0% 0% 0% b_pattern_fname 81% 0% 0% 0% 0% 0% 0% b_pattern_mname 61% 0% 0% 0% 0% 0% 0% b_pattern_insurpolice 35% 0% 0% 0% 0% 0% 0% b_psprt_issued fifty% 0% 0% 0% 0% 0% 0% b_psprt_branch 26% 0% 0% 0% 0% 0% 0% b_psprt_branch 52% 0% 0% 0% 0% 0% 0% b_psprt_branch five% 0% 0% 0% 0% 0% 0% b_psprt_branch 60% 0% 0% 0% 0% 0% 0% b_psprt_isdate 71% 0% 0% 0% 0% 0% 0% b_psprt_issuer 37% 0% 0% 0% 0% 0% 0% b_psprt_code 66% 0% 0% 0% 0% 0% 0% b_psprt_sign 2% 0% 0% 0% 0% 0% 0% b_psprt_gendate 55% 0% 0% 0% 0% 0% 0% b_face 42% 31% 88% 14% 0% 0% eleven% n_pattern_rf 16% 0% 0% 0% 0% 0% 0% n_pattern_birthdate 9% 0% 0% 0% 0% 0% 0% n_pattern_birthplace 48% 0% 0% 0% 0% 0% 0% n_pattern_sign 20% 0% 0% 0% 0% 0% 0% n_pattern_lname thirty% 0% 0% 0% 0% 0% 0% n_pattern_fname 19% 0% 0% 0% 0% 0% 0% n_pattern_mname fifty% 0% 0% 0% 0% 0% 0% n_pattern_insurpolice 65% 0% 0% 0% 0% 0% 0% n_psprt_issued five% 0% 0% 0% 0% 0% 0% n_psprt_branch 62% 0% 0% 0% 0% 0% 0% n_psprt_branch 55% 0% 0% 0% 0% 0% 0% n_psprt_branch 2% 0% 0% 0% 0% 0% 0% n_psprt_branch 94% 0% 0% 0% 0% 0% 0% n_psprt_isdate 43% 0% 0% 0% 0% 0% 0% n_psprt_issuer 57% 0% 0% 0% 0% 0% 0% n_psprt_code 39% 0% 0% 0% 0% 0% 0% n_psprt_sign 29% 0% 0% 0% 0% 0% 0% n_psprt_gendate 35% 0% 0% 0% 0% 0% 0% n_face 39% 20% 19% eleven% 0% 0% 8%

In one embodiment, the training of the neural network occurs not in real time, but during maintenance periods. In this case, the following steps can be performed:

- Perform initial markup in the database of documents and images, in respect of which during the work there were complaints about incorrect recognition or a number of indirect parameters are knocked out of the statistical error.

- Study problem documents in manual mode

- Prepare the cleaned dataset

- Carry out training of a neural network on a cleaned dataset

- Analyze learning outcomes

- Upload the updated coefficients to the database and switch the user flow to the updated recognition logic

Example of application of the invention

The user launches the application on a smartphone and photographs the cashier's receipt using the means built into the application. The application independently processes the images according to the method in accordance with the present invention. If after some time the user needs this check, he can launch the application and give, for example, a voice command "find a check on the iron". The application searches for all receipts with an iron specified as the item. The user prints out the receipt and attaches a copy of the receipt, for example, to a claim sent to a store. In this case, the user does not perform any other actions, except for photographing the receipt and giving a command to the application. The application does the rest of the processing itself.

The present invention makes it possible to reliably recognize and classify a document from an arbitrary image stream.

Claims (67)

1. A method for recognizing and classifying documents from a stream of arbitrary images: i) extracting a document image using machine vision algorithms from a set of arbitrary images; ii) determining an assumed document type using a convolutional neural network based on the extracted document image, wherein the convolutional network exposes the degree of similarity of the document image with known document types based on the specified document samples; iii) determining the document type using the recurrent neural network based on the intended document type obtained from the convolutional neural network, while the recurrent neural network evaluates the degree of similarity of the document with the intended document type based on machine learning algorithms; iv) recognize the data contained in the document based on a certain type of document; v) save the recognized data. 2. The method according to claim 1, further comprising the step of extracting key non-textual features of the document, however, in step iii) the document type is additionally determined based on the extracted non-textual characteristics of the document. 3. The method according to claim 1, further comprising the step of extracting key text parameters of the document, however, in step iii) the document type is additionally determined based on the extracted key text parameters of the document. 4. A method according to any one of claims. 1-3, further comprising the step of recognizing the text contained in the document based on the recognized document type and storing the recognized text. 5. The method according to claim 4, wherein at the stage of recognizing the text contained in the document, based on the recognized document type, a template corresponding to the document type is applied. 6. The method according to any one of claims. 1-3, further comprising the step of enriching the recognized data with additional information extracted from external sources based on the recognized data. 7. A method according to any one of claims. 1-3, additionally containing the stages at which: determine whether the recognized data belongs to a previously recognized document, and in the case where it is determined that the recognized data belongs to the previously recognized document, it is determined whether to create a multi-page document or to glue the recognized data with the previously recognized data. 8. The method according to claim. 7, in which in the case when the recognized data belongs to the previously recognized document, it is checked whether the recognized data matches the previously recognized data, and, if the recognized data coincides with the previously recognized data, the previously recognized data or the recognized data are deleted. 9. The method according to any one of claims. 1-3, additionally containing the stages at which: before the step of determining the intended type of document by the convolutional neural network, it is determined whether the document image is already a recognized document image, and, if the document image is already a recognized document image, the method is terminated. 10. The method according to any one of claims. 1-3, further comprising a stage at which; before step ii) the boundaries of the document are determined and the geometry of the document image is corrected. 11. The method according to any one of claims. 1-3, further comprising a stage at which; before step ii) perform color correction of the document image, and also optimize the contrast and brightness of the image for better text recognition. 12. The method according to any one of claims. 1-3, in which before step ii) the orientation of the document is determined and, if necessary, the document is rotated. 13. The method according to claim 1, in which at step ii) the image is analyzed using a convolutional neural network, the visual similarity of the current image with an array of images of documents of various types is determined based on the weight coefficients stored in it, and a tuple consisting of scalar values of the membership probabilities images to different types of document. 14. The method according to claim 1, wherein in step iii) a plurality of document image parameters are determined and, using a multi-class classifier and a multinomial logistic regression algorithm, these parameters are compared with the coefficients obtained in the result of machine learning. 15. The method of claim 1, wherein the set of arbitrary images required in step i) is obtained by photographing documents. 16. The method of claim 15, wherein documents are photographed using a smartphone camera. 17. The method of claim 15, wherein the set of arbitrary images is contained in a photo bank specified by the user. 18. The method of claim 1, wherein the steps of the method are performed using software. 19. The method of claim 18, wherein the software is a mobile application installed on a smartphone. 20. The method of claim 18, in which the software is located in the cloud on the servers of the operating organization. 21. A system for recognizing and classifying documents from a stream of arbitrary images, containing: a preprocessing module configured to extract a document image using machine vision algorithms from a set of arbitrary images; a convolutional neural network configured to determine the intended type of document based on the extracted document image, while the convolutional neural network is configured to set the degree of similarity of the document image with known document types based on the specified document samples; a recurrent neural network configured to determine the type of document based on the intended type of document received from the convolutional neural network, while the recurrent neural network is configured to assess the degree of similarity of the document with the intended type of document based on machine learning algorithms; and recognizing data contained in a document based on a specific type of document; a storage module configured to store the recognized data. 22. The system of claim. 21, in which the recurrent neural network is configured to extract key non-text features of the document and determining the document type additionally based on the extracted non-text characteristics of the document. 23. The system according to claim. 21, in which the recurrent neural network is additionally configured to recognize the key text parameters of the document and determining the document type additionally based on the recognized key parameters of the document. 24. The method according to any one of paragraphs. 21-23, in which the recurrent neural network is configured to recognize the text contained in the document based on the recognized type of the document, while the storage module is configured to store the recognized text. 25. The system of claim 24, wherein the recurrent neural network is configured to apply a template corresponding to the type of document when recognizing the text contained in the document. 26. The system according to any one of paragraphs. 21-23, in which the recurrent neural network is configured to enrich the recognized data with additional information extracted from external sources based on the recognized data. 27. System according to any one of paragraphs. 21-23, in which the recurrent neural network is additionally configured to: determining whether the recognized data belongs to a previously recognized document, and in the case where it is determined that the recognized data belongs to a previously recognized document, determining whether to create a multi-page document or to glue the recognized data with the previously recognized data. 28. The system of claim. 27, in which the recurrent neural network, in the case when the recognized data belongs to a previously recognized document, is configured to check whether the recognized data matches the previously recognized data, in this case, the storing module, if the recognized data coincides with the previously recognized data, is configured to delete the previously recognized data or the recognized data. 29. The system according to any one of paragraphs. 21-23, in which the preprocessing module is configured to: determining if the document image is already a recognized document image, and, if the document image is already a recognized document image, stop processing the document. 30. The system of claim 20, wherein the preprocessing module is further configured; defining the boundaries of the document and performing correction of the geometry of the document image. 31. The system according to any one of paragraphs. 21-23, in which the preprocessing module is further configured to perform color correction on the document image, and to optimize the contrast and brightness of the image to provide better text recognition. 32. The system according to any one of paragraphs. 21-23, in which the preprocessing module is further configured to determine the orientation of the document and, if necessary, perform rotation of the document. 33. The system according to claim 21, in which the convolutional neural network is additionally configured to analyze the image, determine, based on the weight coefficients stored in it, the visual similarity of the current image with an array of images of documents of various types, and issue a tuple consisting of scalar values of the probabilities of the image belonging to different types of document. 34. The system according to claim 21, in which the recurrent neural network is configured to determine a plurality of document image parameters and by using a multi-class classifier and a multinomial logistic regression algorithm for comparing said parameters with coefficients, obtained as a result of machine learning. 35. The system of claim. 21, wherein said set of arbitrary images is obtained by photographing documents. 36. The system according to claim 35, wherein the photographing of documents is carried out using a smartphone camera. 37. The system of claim 35, wherein the set of arbitrary images is contained in a bank of photographs specified by the user. 38. The system of claim 21, wherein the preprocessing module, convolutional neural network, recurrent neural network, and storage module are implemented as software. 39. The system of claim 38, in which the software is a mobile application installed on a smartphone. 40. System p. 38, in which the software is in the cloud on the servers of the operating organization.

Images