EA040560B1 - METHOD AND SYSTEM FOR INTELLIGENT DOCUMENT PROCESSING - Google Patents

Description

Technical field

The presented technical solution relates in general to the field of image analysis, and in particular to methods and systems for intelligent processing of an electronic set of documents, for example, scanned documents of bank customers.

State of the art

Currently, there is a problem of prompt and high-quality processing of data from an electronic set of scanned documents in order to check the presence of mandatory filled document fields from structured and unstructured documents, as well as signer attributes, such as a signature. In the prior art, various solutions are known that are made with the ability to process documents, for example, a Bank client, implemented on the basis of ABBYY FlexiCapture software, etc. Also known is a solution for checking a set of documents, disclosed in the application US 2011134494 A1, publ. 06/09/2011, in which a document having a plurality of pages is read; checking the image data of each page of a document having a plurality of pages, wherein certain areas of the document image are checked for the presence or absence of information. This solution is the closest analogue.

A significant drawback of the known solutions is the low efficiency in detecting errors when checking documents for the correctness of their filling, since in a very large number of cases, the known solutions give the result there is an error, although it actually does not exist, all fields are filled in correctly, but the well-known solution simply could not find them in the text due to the fact that the text is poorly structured. Also, in the known solutions, there is no mechanism for automated decision-making based on the results of the mentioned check.

Disclosure of invention

The technical problem or task posed in this technical solution is the creation of a new effective, simple and reliable method for automated intellectual processing of all types of documents for the correctness of their filling.

The technical result is to increase efficiency and ensure high accuracy in detecting errors during automated intellectual processing of documents.

The specified technical result is achieved due to the implementation of the method of intellectual processing of documents, performed by at least one computing device and containing the steps at which at least one document image is obtained;

recognize characters in the document image and convert them into text information;

on the basis of text information determine the type of document;

extracting from the text information a set of entities, taking into account the type of the document;

comparing the entity set with a reference entity set for the document;

on the basis of the results of comparison of the mentioned sets of entities, the results of processing the document are formed.

In one of the particular examples of the implementation of the method, the document is an agreement on individual lending conditions (ICC) or a guarantee agreement (DP).

In another particular example of the implementation of the method, the steps are additionally performed, at which the signer's attribute is detected on the incoming image of the document; determining the location of at least one signer attribute on the document page; wherein the results of document processing are formed taking into account information about the location of at least one attribute of the signer on the document page.

In another particular embodiment of the method, the step is additionally performed, at which the status of the person to whom the detected signer attribute belongs is determined.

In another particular embodiment of the method, the steps are additionally performed, at which a process identifier is obtained; determining a set of text classification models based on the process ID; converting the received text information into a set of vectors; processing the set of vectors with the previously defined set of text classification models to determine the document type.

In another particular embodiment of the method, the steps are additionally performed at which the entity set is divided into simple entities consisting of 1-3 words and complex entities consisting of at least four words; moreover, if as a result of comparing said sets of entities, the threshold values of matching words for simple and complex entities are reached, then reconciliation results are generated, which include information about the successful completion of data reconciliation; if said threshold values of matching words for simple and complex entities are not reached, then reconciliation results are generated, which include information about entities in the set of entities that have not passed reconciliation; in this case, the results of processing the document are formed taking into account the results of reconciliation.

In another particular embodiment of the method, an additional step is performed, at which the quality of scanning a document is determined; moreover, the results of processing the document are formed taking into account the quality of scanning the document.

- 1 040560

In another preferred embodiment of the claimed solution, an intelligent document processing system is provided, comprising at least one computing device, and at least one memory device containing machine-readable instructions, which, when executed by at least one computing device, perform the above method.

Brief description of the drawings

The features and advantages of the present technical solution will become apparent from the following detailed description of the invention and the accompanying drawings, in which in Fig. 1 shows the general scheme of interaction between the elements of the intelligent document processing system;

in fig. 2 shows an example of a scanned document;

in fig. 3 shows an example of a general view of an intelligent document processing system.

Implementation of the invention

The concepts and terms necessary for understanding this technical solution will be described below.

In this technical solution, the system means, among other things, a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, well-defined sequence operations (actions, instructions).

A command processing device is an electronic unit, a computing device, or an integrated circuit (microprocessor) that executes machine instructions (programs).

An instruction processing device reads and executes machine instructions (programs) from one or more data storage devices. The role of a storage device can be, but not limited to, hard drives (HDD), flash memory, ROM (read only memory), solid state drives (SSD), optical drives.

Program - a sequence of instructions intended for execution by a computer control device or a command processing device.

Database (DB) - a collection of data organized in accordance with a conceptual structure that describes the characteristics of this data and the relationship between them, and such a collection of data that supports one or more areas of application (ISO / IEC 2382: 2015, 2121423 database).

In accordance with the diagram shown in Fig. 1, the intelligent document processing system 10 includes interconnected data conversion module 11; a signature detection module 12, a data extraction module 13, a package document classification module 17, and a business rule module 18, consisting of a data verification module 14, a document property analysis module 15, a decision module 16, and a legal validity analysis module 19.

These modules can be implemented on the basis of the firmware of the system 10 of intelligent document processing, for example, on the basis of at least one computing device, in particular a microprocessor, and at least one memory device containing machine-readable instructions written in the Python programming language, for implementation of the functions performed by the modules. For example, the data conversion module 11 may be implemented on the basis of an optical character recognition (OCR) tool. The signature detection module 12 can be implemented on the basis of a neural network of the YOLOv3 architecture, pre-trained on a typical set of signatures and seals. The package document classification module 17 may be implemented in system 10 firmware configured to represent text as vectors (eg, TFIDF) and include a set of text classification models, such as SVM or Random Fields. The data extraction module 13 can be implemented on the basis of the firmware of the system 10 and include a set of models for analyzing the semantics of natural languages word2vec, a pre-trained mathematical model - conditional random fields (Conditional Random Fields) and computing tools for natural language processing (Natural Language Processing , NLP). The business rules module 18, consisting of a data verification module 14, a document properties analysis module 15, a decision module 16, and a legal validity analysis module 19, can be implemented on the basis of the firmware of the system 10 configured in the firmware in this way to perform the functions assigned to them below.

At the first stage of the system 10 operation, the data conversion module 11 and the signature detection module 12 receive at least one image of a document, in particular a scanned document, for example, a file in multipage PDF, JPEG, TIFF or any other known format that can be used to store It contains images of the scanned document. The document image may come from an image data source 1, in particular directly from a document scanning device such as a scanner, or may be retrieved from an appropriate image database in which the document image data is stored in advance.

Also, in accordance with a predetermined software and hardware algorithm, the package document classification module 17 and the business rules module 18 receive data on the process identifier from the automated system (AS) 2 of the Bank. The process ID from the Bank's AS 2 may be supplied to said modules by methods well known in the art, for example, before the document is fed to the scanner or before the document image is retrieved from the database, according to the process in which the document is checked. Based on the process ID data, a set of possible types of documents is subsequently determined, which may be on the image of the document received by the data conversion module 11; a set of entities to be retrieved by module 13, and data on the location of signatures in documents. For example, data on the process identifier may indicate that 2 types of documents can come to the input of the document classification module 17: an agreement on individual lending conditions (ICC) or a surety agreement (DP), therefore, the corresponding classifier is triggered.

The document whose image is supplied to the data conversion module 11 may be any document consisting of at least one page, which may contain the attributes of the signer, and filled in accordance with a known template. The document can be, for example, an ITP document/agreement signed by a bank client or a suretyship agreement (SA). The document may contain fields that contain information about the signer, such as the signer's full name, signer's address, signer's card number, passport details, etc., as well as information about the terms of the agreement, such as credit conditions. In particular, according to the scheme shown in Fig. 2, in area 101 of document 100 may contain a field with information about the number of the mentioned application, in area 102 a field with the name of the city, in area 103 - a field with the date of the application, in area 104 - fields with information about the signatory and credit conditions, in area 105 or document 106 - signer attribute images, such as a signature image.

The data conversion module 11 performs character recognition on the document image and converts them into text information. Along with this, the signature detection module 12 detects the signer's attribute on the incoming document image, determining its location on the document page. The signer attribute may not be present on the page, and this information is also passed on in the manner shown in FIG. 1. For example, module 12 can determine that the signer attribute image is a signature image in the document area 105 or 106 (see FIG. 2) by automatically indicating the coordinates of the found boxes 105 and 106. Accordingly, data about the location of the signer attributes on the document page or about their absence, module 12 sends to module 19 the analysis of legal validity.

To detect signer attribute images, the well-known algorithms of the YOLOv3 architecture neural network, trained on a selected set of signatures and seals data, are used, disclosed, for example, in an article published on the Internet at https://pireddie.com/media/files/papers/YOLOv3 .pdf.

If the document image contains the attributes of more than one signer, for example, a signature image of a Bank client and a signature image of a Bank employee, then the legal validity analysis module 19 can be configured to determine the status of the person who owns the detected attributes of the signer. To do this, in the memory of module 19, the user of system 10 can pre-set a list of statuses of persons and information about the location of their attributes of the signer on the document image based on the process identifier, data about which was received from AS 2 of the Bank to module 18, and information about the status of persons can indicate to which person the attribute of the signer belongs, in particular, for example, a client of the Bank or an employee of the Bank. For example, for the status of a Bank client person, the location data may indicate that its signer attributes should be located in the document area 105, and for the Bank employee person status, the location data may indicate that its signer attributes are located in the document area 106.

Accordingly, the legal validity analysis module 19 compares the location data of the signer attribute image on the document page received from the module 12 with said stored data, in particular the location data of the signer attributes according to the process type determined by the module 19 based on the identifier data obtained previously. process and, based on the result of the comparison, determines the status of the person who owns the detected signer attribute, i.e. based on information about the location of the signer attribute on the document page image. The data on the status of the person and the data on the location of the images of the attributes of the signers on the page of the document module 19 sends to the decision module 16 . If module 19 receives information about the absence of the signer's attributes in the image, then module 19 redirects this information to module 16.

As for the textual information, the data conversion module 11 sends it to the data extraction module 13 and to the document classification module 17 of the package. Module 17, based on the process ID data received from AS 2, determines a set of text classification models that can be pre-defined in said module 17 for each type of process by the user

- 3 040560 system 10, after which the module 17 converts the received text information into a set of vectors, which is processed by a previously defined set of text classification models to determine the type of document. Module 17 sends data about the document type to module 13, which extracts from the received text information from module 11 a set of entities in accordance with the document type. An entity set may include a full name, address, card number, document date, card number, passport data, credit conditions, etc. To extract a set of entities from the received textual information, module 13 performs tokenization of textual information and supplies tokenized textual information as input to a set of word2vec models, at the output of which module 13 receives a sequence of vectors.

Next, within module 13, a trained CRF (Conditional Random Fields) machine learning model is defined based on document type data, and a sequence of vectors is processed by said trained model that defines a set of entities. The trained CRF machine learning models for each type of document can be pre-defined in the mentioned module 13 by the user of the system 10. Machine learning models trained by the CRF method are widely used in various areas of AI, in particular in speech and image recognition, text information processing, and also in other subject areas: bioinformatics, computer graphics, etc.

In an alternative implementation of the claimed solution, the entities can be extracted using natural language processing technology (Natural Language Processing, NLP). This technology is widely known from the prior art (see, for example, the article NLP. Fundamentals. Techniques. Self-development. Part 2: NER, published on the Internet at https://habr.com/ru/company/abbyy/blog/449514/ ), and will not be further disclosed in this document. The vector sequence processing algorithm can also be selected depending on the document type.

The resulting set of entities module 13 data extraction sends to the module 14 data reconciliation. Also, module 14 is supplied with a reference set of entities by module 18 of business rules. The reference set of entities is determined by module 18 on the basis of previously received data about the process identifier from AS 2 of the Bank. The reference set of entities for each type of process can be pre-specified in the mentioned module 18 by the user of the system 10. The module 14 divides the received data of entity sets into simple entities consisting of 1-3 words, and complex entities consisting of at least four words. For example, if system 10 received an ICC document as input, then simple entities will be, for example, full name, loan amount, contract start date, passport number, date of issue of a passport, etc., and complex entities will be, for example, address, place of issue passports, etc. Next, the data verification module 14 proceeds to the stage of comparing the entity set received from the module 13 with the reference entity set. Data of simple entities module 14 data verification leads to one format, and then compares them. In these complex entities, before they are compared, generally recognized abbreviations are deciphered, words that do not contain names are excluded. If the threshold values of matching words for simple and complex entities set by the user of the system 10 are reached, then the set of entities received from the module 13 passes the data verification. If the matching word thresholds for simple and/or complex entities are not reached, then the entity set fails the test. As a result of the comparison of entity sets, the data verification module 14 generates the results of the verification, which include information about the successful completion of the verification, or if the entity set did not pass the verification, information about the entities in the entity set that did not pass the verification. The information about the entity set received from the module 13, together with the textual information and the results of the verification by the data verification module 14, are sent to the document properties analysis module 15.

All information collected by module 15 during the work of all previous modules, in particular textual information and the results of reconciliation from module 14 and the image of the document from source 1, is checked by module 15 to ensure that all the necessary paragraphs of the document (or fields of the document) are contained in the text of the document. To do this, module 15 processes the received text information using NLP methods (fuzzy occurrence of keywords for each paragraph), the results of which module 15 determines the integrity of the document. The NLP processing algorithm can also be selected based on the process ID data that was previously received by module 18 from AS 2 of the Bank. To process the received textual information using NLP methods, a set of typical documents was analyzed for the distribution of words in paragraphs of the document and characteristic words and / or phrases were found for each paragraph of the document, and from its different parts (beginning, middle, end). Thus, for each significant (which must be present in the document to check the integrity) paragraph of the document, its characteristic words have become known. Further, a rule was created according to which: if a certain proportion of words or phrases occurs (fuzzy search) in a paragraph of the document, then this significant paragraph is found. If all the necessary paragraphs (points) of the document are found in the text, then the integrity was checked successfully. In an alternative implementation of the claimed solution, the integrity of the document can be verified using the tools and methods disclosed in the application US 2011134494 A1. Based on the document integrity data and the verification data, module 15 determines the scan quality of the document image.

- 4 040560

For example, if the data verification is successful and the document integrity data indicates that the document contains all items, then module 15 assigns a high scan quality score to the document image. If the reconciliation results indicate that the matching word thresholds for simple and/or complex entities are not met, and the document integrity data indicate that the document does not contain all items, then module 15 assigns a low scan quality score to the document image. Information about the indicator of the quality of the scan module 15 transmits to the module 16 decision.

Also, the document properties analysis module 15 is configured to check whether the document is attached from another person. Said verification is performed on the basis of document integrity data and entity set unique entity data, which are different for different clients or that can be the same for different clients with a very small probability (for example, entities that identify the signer). Analysis of only unique entities allows you to exclude those entities that may be repeated for different clients, for example, the loan currency, which is most often in rubles, and other entities depending on the type of document. For example, for an ITP or PD document, the name of the borrower is a unique entity. Also unique entities can be TIN, SNILS, passport serial number, etc.

If the unique entities do not match (for example, in relation to the ICC document - the full name of the borrower), while the data on the integrity of the document indicates that all items in the document are present, then module 15 determines that the document whose image entered the system 10 belongs to to another person. If the module 15 has determined that the integrity of the document is incomplete, and the unique entities of the entity set, for example, identifying the signer, indicate that the document whose image entered the system 10 is a document of this person, then the module 15 generates a list of entities that did not pass the test. Accordingly, if the unique entity set entities identifying the signer match the reference entity set and the document integrity data indicates that all items in the document are present, then module 15 determines that said document is that person's document. The algorithms of the property analysis module 15 are parameterized by the process ID.

All information collected during the work of all previous modules of the document, with the exception of the images of the document, is sent to the decision module 16 . If the reconciliation results received from module 14 are positive and the data received from module 19 indicate that all the necessary attributes of signers are present on the document image in its respective areas (i.e. the rule for the location of all signatures is satisfied; in this case is determined by the number of signatures found, according to their relative position, excluding places where there certainly cannot be a signature), then the module 16 writes information about the successful completion of the document verification in the results store of the document processing web service 20. For example, if the package of documents contained only the ICC document and the DP was not required, then the module 16 writes to the mentioned storage of the web service 20 information about the successful completion of the document verification, as well as information about the decision, in particular, that it is possible to issue a loan. Additionally, information about the set of entities and the results of reconciliation are entered into the results of document processing generated and recorded in the storage by module 16. If the data received from module 19 indicates that the signer attribute is not present on the image of the document in the corresponding area, then the decision module 16 generates information that the document should be checked by a human, which also includes information about the results of the reconciliation.

The corresponding areas (permissible range of coordinates for the attributes of the signer) can be determined by module 18 based on the type of document, which is determined on the basis of the process ID data received from AS 2 of the Bank, and then goes to module 16. If the reconciliation results are negative, then the decision module 16 extracts from the received data information about all entities from the set of entities that have not passed data validation, and determines the types of these entities. If the entity type indicates that the entity is a simple entity, and the scan quality information received from module 15 indicates that a document image has been assigned a high scan quality score, then decision module 16 generates information that the document is not passed a review, which also includes information about the results of the reconciliation, and that the loan should be denied. At the same time, if the scan quality information indicates that the document image is assigned a low scan quality score, then the module 16 generates and writes to the web service 20 document processing results store information that the document should be checked by a person, which also includes information about the reconciliation results.

If the entity that has not passed data verification is a complex entity, then the decision module 16, regardless of the document scanning quality index, generates and writes to the web service 20 document processing results store information that the document should be checked by a person, in which also includes information about the results of the reconciliation. The generated results of document processing with negative reconciliation results also include information

- 5 040560 information about the presence or absence of the signer's attributes.

The results of document processing generated by the decision module 16 can be obtained via the web service 20 interface or its API. Web service 20 generates a response in the form of json with the results of document processing. These results of document processing can be displayed on a data display device, for example, a display of a computing device, such as a portable or stationary computer, communication terminal, mobile phone or smartphone, tablet, etc. For example, if the document was an ICC document, then the data display device additionally a decision can be made to grant a loan, to refuse to issue a loan, or to check the document manually.

Thus, due to the fact that the results of document processing are formed on the basis of the results of comparing the set of entities extracted from textual information, taking into account the type of document, with the reference set of entities for this document, high accuracy is ensured in detecting errors during automated intellectual processing of documents, and also its efficiency, i.e. the specified technical result is achieved. Also, through the use of machine learning algorithms and NLP methods disclosed in this application, and data typing, the efficiency and accuracy in detecting errors in automated document processing are further improved. In addition, the presented technical solution has enhanced functionality compared to known solutions, in particular: it provides the possibility of automated decision-making on issuing a loan, identifying the reason for refusal or justifying the transfer of a document for verification to a person; provides a mechanism for checking the legal validity and completeness of documents. In general terms (see Fig. 3), the system (200) of intellectual document processing contains one or more processors (201), memory facilities, such as RAM (202) and ROM (203), input / output interfaces connected by a common information exchange bus (204), input/output devices (205) and a device for networking (206).

The processor (201) (or multiple processors, multi-core processor, etc.) can be selected from a range of devices currently widely used, for example, manufacturers such as: Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™ , Qualcomm Snapdragon™, etc. Under the processor or one of the processors in use in the system (200), it is also necessary to take into account the graphics processor, for example, NVIDIA GPU with a CUDA-compatible software model, or Graphcore, the type of which is also suitable for full or partial execution of the method, and can also be used for training and application of machine learning models in various information systems.

RAM (202) is a random access memory and is designed to store machine-readable instructions executable by the processor (201) to perform the necessary operations for logical data processing. The RAM (202) typically contains the executable instructions of the operating system and associated software components (applications, program modules, etc.). In this case, the RAM (202) may be the available memory of the graphics card or graphics processor.

The ROM (203) is one or more persistent storage devices such as a hard disk drive (HDD), a solid state data drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R/ RW, DVD-R/RW, BlueRay Disc, MD), etc.

Various types of I/O interfaces (204) are used to organize the operation of system components (200) and organize the operation of external connected devices. The choice of the appropriate interfaces depends on the specific implementation of the computing device, which can be, but not limited to, PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

To ensure user interaction with the computer system (200), various means (205) of I/O information are used, for example, a keyboard, a display (monitor), a touch screen, a touchpad, a joystick, a mouse manipulator, a light pen, a stylus, a touch panel, trackball, speakers, microphone, augmented reality tools, optical sensors, tablet, indicator lights, projector, camera, biometric identification tools (retinal scanner, fingerprint scanner, voice recognition module), etc.

The networking means (206) provides data transmission via an internal or external computer network, such as an Intranet, Internet, LAN, and the like. As one or more means (206), an Ethernet card, a GSM modem, a GPRS modem, an LTE modem, a 5G modem, a satellite communication module, an NFC module, a Bluetooth and/or BLE module, a Wi-Fi module, and others

Additionally, satellite navigation tools as part of the system (200) can also be used, such as GPS, GLONASS, BeiDou, Galileo. The specific choice of elements of the device (200) for the implementation of various software and hardware architectural solutions may vary while maintaining the required functionality provided.

Modifications and improvements to the above embodiments of this technical

- 6 040560 solutions will be clear to those skilled in the art. The foregoing description is provided by way of example only and is not intended to be limiting in any way. Thus, the scope of the present technical solution is limited only by the scope of the appended claims.

Claims (8)

1. A method of intellectual processing of documents to check for the correctness of their filling, performed by at least one computing device, comprising the steps that receive at least one image of the document; recognize characters in the document image and convert them into text information; based on the text information obtained in the previous step, determine the type of document; extracting from the text information a set of entities, taking into account the type of the document; comparing the entity set with a reference entity set for the document; on the basis of the results of comparison of the mentioned sets of entities, the results of processing the document are formed, containing information about the successful passing of the document check or information that the document did not pass the check. 2. The method according to claim 1, characterized in that the document is an agreement on individual lending conditions (ICC) or a guarantee agreement (DP). 3. The method according to claim 1, characterized in that it further comprises the steps, which carry out the detection on the received image of the document attribute of the signer; determining the location of at least one signer attribute on the document page; wherein the results of document processing are formed taking into account information about the location of at least one attribute of the signer on the document page. 4. The method according to claim 3, further comprising the step of determining the status of the person who owns the detected signer attribute. 5. The method according to claim 1, characterized in that the stage at which the type of document is determined based on textual information, contains the steps at which a process identifier is obtained; determining a set of text classification models based on the process ID; converting the received text information into a set of vectors; processing the set of vectors with the previously defined set of text classification models to determine the document type. 6. The method according to claim 1, characterized in that the step at which the entity set is compared with the reference entity set comprises the steps at which the entity set is divided into simple entities consisting of 1-3 words, and complex entities consisting of at least least four words; moreover, if as a result of comparing the said sets of entities, the threshold values of matching words for simple and complex entities are reached, then reconciliation results are generated, which include information about the successful completion of the data reconciliation; if said threshold values of matching words for simple and complex entities are not reached, then reconciliation results are generated, which include information about entities in the set of entities that have not passed reconciliation; in this case, the results of processing the document are formed taking into account the results of reconciliation. 7. The method according to claim 1, characterized in that it further comprises the step of determining the quality of scanning the document; moreover, the results of processing the document are formed taking into account the quality of scanning the document. 8. An intelligent document processing system, comprising at least one computing device and at least one memory device containing machine-readable instructions, which, when executed by at least one computing device, performs the method according to any one of claims 1-7.