RU2774653C1 - Methods and systems for identifying fields in a document - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: invention relates to the field of computer technology and can be used to detect fields on document images. The method involves obtaining a training dataset containing several documents, and each of several documents correlates with several fields marked up by the user; for a given field from several user-marked fields in this document from several documents, determining whether a specific combination of relative positions of one or more additional user-marked fields relative to this field is repeated in one or more additional documents in this document; indicating that the selection of this field is incorrect if it is established that a specific combination is not repeated in any additional documents; establishing the presence in two or more other documents of another combination of relative positions of additional one or more fields marked up by the user, relative to this field, if it is established that a specific combination is repeated in one or more additional documents, and specifying this field as marked up correctly if it is established that another combination does not exist in two or more other documents; specifying this field as marked up is contradictory if it is established that this combination exists in two or more other documents.

EFFECT: determining the accuracy of user markup of fields in documents.

1 cl, 9 dwg

Description

FIELD OF TECHNOLOGY

[001] The present invention generally relates to computer systems, and more specifically to systems and methods for detecting fields in a document.

BACKGROUND OF THE INVENTION

[002] The image may show a document or part of it. A document can be made up of various types of elements, including fields. An image processing engine can be tasked with detecting margins in a document image.

DISCLOSURE OF THE INVENTION

[003] In accordance with one or more embodiments of the present invention, the method described in the example may include: obtaining a training data set containing multiple document images, each of the multiple document images being matched with the corresponding metadata that defines a field of the document, containing variable text; the formation by processing several images of the document of the first heat map, represented by a data structure containing several elements of the heat map corresponding to several pixels of the document image, each element of the heat map storing a counter of a certain number of document images, in which the document field contains a document image pixel associated with heat map element; obtaining an image of the input document; and determining, within the input document image, a candidate region containing the document field, the candidate region containing a plurality of input document image pixels corresponding to heat map elements that satisfy the threshold condition.

[004] In accordance with one or more embodiments of the present invention, the system described in the example may include: a memory device for storing instructions; a processing device coupled to the storage device, the processing device for executing instructions for: obtaining a training data set containing multiple document images, each of the multiple document images being matched with corresponding metadata defining a document field containing variable text; the formation by processing several images of the document of the first heat map, represented by a data structure containing several elements of the heat map corresponding to several pixels of the document image, and in each element of the heat map a counter of a certain number of document images is stored, in which the document field contains a document image pixel associated with heat map element; obtaining an image of the input document; and determining, within the input document image, a candidate region containing the document field, wherein the candidate region contains a plurality of input document image pixels corresponding to heat map elements that satisfy the threshold condition.

[005] In accordance with one or more embodiments of the present invention, the non-volatile computer-readable medium described in the example may contain instructions, when executed by the data processing device, the data processing device: obtains a training data set containing several document images, each of several images of the document is compared with the corresponding metadata that defines the field of the document containing the variable text; generates, by processing several document images, the first heat map represented by a data structure containing several heat map elements corresponding to several document image pixels, each heat map element storing a counter of a certain number of document images, in which the document field contains a document image pixel associated with heat map element; receives an image of the input document; defines, within the input document image, a candidate region containing the document field, wherein the candidate region contains several input document image pixels corresponding to heat map elements that satisfy the threshold condition.

[006] In accordance with one or more embodiments of the present invention, the method described in the example may include: obtaining a training data set containing multiple documents, each of the multiple documents being associated with multiple user-labeled fields; for a given field of multiple user-marked fields in a given document of multiple documents, determining whether one or more additional documents repeat a particular combination of relative positions of one or more additional user-marked fields relative to the given field in the given document; an indication of the incorrect selection of this field, if it is established that a particular combination is not repeated in any additional documents; establishing the presence in two or more other documents of another combination of relative positions of one or more additional fields marked up by the user, relative to this field, if it is determined that a particular combination is repeated in one or more additional documents, and indicating this field as marked up correctly, if installed, that another combination does not exist in two or more other documents; and indicating a given field as tokenized inconsistently if the given combination is found to exist in two or more other documents.

BRIEF DESCRIPTION OF THE DRAWINGS

[007] The present invention is illustrated by way of example, without any limitation; its essence becomes clear when considering the following detailed description of the invention in combination with the drawings, in which:

[008] In FIG. 1 is a high-level component diagram of an exemplary system architecture in accordance with one or more embodiments of the present invention.

[009] In FIG. 2 is a high-level flow diagram of an exemplary method for detecting fields in a document in accordance with one or more embodiments of the present invention.

[0010] In FIG. 3 is a block diagram of various components of an exemplary document field detection system in accordance with one or more embodiments of the present invention.

[0011] In FIG. 4A-4B show examples of heat maps used in performing field detection on a document in accordance with one or more embodiments of the present invention.

[0012] In FIG. 5 shows an illustrative example of an internal field format evaluation in accordance with one or more embodiments of the present invention.

[0013] In FIG. 6 shows an illustrative example of field detected documents in accordance with one or more embodiments of the present invention.

[0014] In FIG. 7 is a flow diagram of an exemplary method for detecting fields in a document in accordance with one or more embodiments of the present invention.

[0015] In FIG. 8 depicts an exemplary computer system that can perform any of the methods described herein in accordance with one or more embodiments of the present invention.

DESCRIPTION OF THE INVENTION

[0016] Applications for document field detection are described. "Field" refers to an area within a document (e.g., a rectangular area), such that this area is designated as a placeholder where variable data can be entered, thereby creating a new instance of the document type (e.g., invoice, order, etc.). d.). Fields are usually found in form documents. A document can have various fields such as text fields (containing numbers, numbers, letters, words, sentences), graphic fields (containing a logo or any other image), tables (containing rows, columns, cells) and so on.

[0017] In this application, "electronic document" (also referred to simply as "document" in this application) can mean any document whose image can be accessed by a computing system. An image can be a scanned image, a photographed image, or any other representation of a document that can be converted into a computer-accessible data form. For example, an "electronic document" may refer to a file containing one or more digital content items that can be visually displayed to provide a visual representation of the electronic document (eg, on a display or in print). In accordance with various applications of the present invention, the document may conform to any suitable electronic file format such as PDF, DOC, ODT, JPEG, etc.

[0018] A "document" can be a financial, legal, or any other document, such as a document created by filling fields with alphanumeric characters (eg, letters, words, numbers) or images. A "document" may be a document that is printed, typed, or handwritten (eg, by filling out a standard form). A "document" may be a form document that has various fields such as text fields (containing numbers, numbers, letters, words, sentences), a graphic field (containing a logo or any other image), tables (having rows, columns, cells) and so on. Examples of different types of documents that have fields include contracts, invoices, financial documents, business cards, identification cards, loan applications, orders/statements, accounting documents, reporting documents, patent search reports, various company letterheads, and etc.

[0019] Fields can be found in various types of documents such as invoices, financial documents, business cards, identity cards, loan applications, etc. Optical Character Recognition (OCR) of such a document may include a preliminary step of determining all fields contained in a document, which can be performed by neural networks trained on a large number (for example, thousands) of document samples that include fields. However, this approach does not provide field detection with sufficient accuracy for various types of documents using universal document templates, since documents can differ significantly from each other. Thus, a large number of documents of each type may be required to train neural networks. These field discovery methods require lengthy and extensive training with many manual steps, lack flexibility, and potentially expose sensitive data. Moreover, these methods also require precise markup for each document. However, when manual operations are used to mark up documents, users often skip or mark up fields incorrectly, making the documents unsuitable for use as training samples. In addition, in order to correct the consequences of such incorrect markup, the user may need to repeat the markup or markup additional documents of the same type in order to start learning again. In some cases, the user also cannot markup correctly because they may not know where a particular field is in the document, or the field is not defined by a descriptive word. For example, the user may have the intention to mark up a "total" field that is expected to be filled with a number. However, if the field is not defined by the word "total", or the user cannot find the word "total", or for other reasons, the user may instead mark up another field containing characters visually similar to the expected content of the "total" field, such as another field containing numbers.

[0020] Embodiments of the present invention address the aforementioned and other disadvantages by providing mechanisms for detecting fields in a document without having to mark up a large number of documents manually to train a neural network. Mechanisms can provide fast training of a trainable model on a small data set, for example, a data set that includes no more than ten documents of a particular type with labeled fields. After the model has been trained for a particular class of documents, it is used to detect fields in other custom documents of the same class.

[0021] In one application, embodiments of the present invention provide training to a neural network using a small number of markup documents to be used as training documents. These documents may contain metadata that defines one or more document fields based on custom markup that indicates the location of the corresponding fields in the document. Field detection is based on determining the spatial distribution of fields in relation to visual references in training documents. After receiving the document images, the text is obtained from the document images, as well as various characters, including words, from the text in the document images. Reference elements on the document image can be used to determine the location of marked fields. Any structural element related to the layout of the document can be used as a reference. An anchor element may include predefined visual elements such as a predefined word (eg, keywords, custom dictionary words), a predefined graphic element (eg, visual divider, logo), etc. on document images. Anchors in document images can be determined by matching words from a user dictionary and/or words that appear on a given document (or body of documents) at a frequency greater than a predetermined threshold frequency. For example, an invoice might contain the company name, totals, due dates, etc. as pivots, depending on the frequency with which these keywords may appear in this type of document. The location of the various fields of the document can be determined relative to the reference element.

[0022] For each field in the training dataset, a heat map can be generated for each pivot. "Heat map" is a set of numerical elements, compiled in such a way that the value of each element is determined by a specific function, calculated from the image coordinates that reflect the position of the element. In some applications, the heat map may be represented by a rectangular matrix, each element of which corresponds to a particular pixel in the vicinity of the reference element, such that the value associated with each pixel reflects the number of training documents in which that field contains that pixel. The numeric values of the heatmap elements can be color-coded for visualization (hence the name of the term), but this step would be redundant for training neural networks that use numeric values instead of colors.

[0023] Accordingly, at the training stage, heat maps of a relatively small set of training documents can be generated, which are accompanied by metadata ("markup") indicating the positions and names of the fields of the document. Then the generated heat maps can be used to determine the positions of fields in other documents. In some applications, a system operating in accordance with embodiments of the present invention may determine a candidate region for each field of interest in an input document image based on heat maps generated for that field against one or more reference elements. Each defined candidate area will include input document image pixels corresponding to heatmap elements that satisfy a threshold condition (e.g., when their respective values exceed a threshold, when a predetermined proportion of pixels having the highest values is selected, etc.).

[0024] The selected candidate regions can then be considered as the positions of the respective fields, for example, by applying optical character recognition (OCR) techniques to the image fragments lying within the candidate regions. In some applications, the extracted content of each document field can be evaluated using BPE (Byte Pair Encoding) tokens, evaluating the differences (e.g., Euclidean distances) between the BPE token representing the extracted content of a given field in the input document and the BPE tokens computed for same field in the tutorials, the BPE token refers to a numeric vector representing the input text. In some applications, a vector may be represented by embedding an intermediate representation of the input text such that the intermediate representation may use an artificial alphabet, each character of which may encode a substring of one or more characters of the input text, as described in more detail below. Embedded data is formed in such a way that semantically similar input data will produce numerically close embedded data.

[0025] Accordingly, if the calculated distance between the BPE token representing the content extracted from the candidate field and the BPE token(s) representing the same field in the training dataset is less than a threshold value, then the probability that this the field will be detected correctly, relatively large, and the candidate field can be taken to extract information. The technologies described in this application allow automatic field detection in documents using artificial intelligence. The systems and methods described herein represent significant improvements in providing more accurate and efficient field detection in documents. The methods employ trainable models that can be trained on a small number (eg, less than ten) of sample documents to detect and classify fields while maintaining high quality. These methods allow you to speed up and improve the quality of data verification. In addition, the methods may also serve as a guide to users if the user may have inaccurately labeled a field or forgotten to label it. The methods make it possible to detect erroneous document markup performed by the user, as well as to effectively correct and restore the missing field markup. In addition, these methods allow you to select a subset of markup documents containing complete and consistent markup, which, in turn, can allow you to train additional, more accurate models.

[0026] Various embodiments of the aforementioned methods and systems are detailed below by way of example and not limitation.

[0027] In FIG. 1 is a high-level component diagram of an exemplary system architecture 100 in accordance with one or more embodiments of the present invention. System architecture 100 includes computing device 120, storage 160, and server 150 connected to network 130. Network 130 may be a public network (eg, the Internet), a private network (eg, a local area network (LAN), or a wide area network (WAN) ), or a combination of them.

[0028] Computing device 120 may perform field detection on a document image. In one application, computing device 120 may be a desktop computer, laptop, smartphone, tablet computer, server, scanner, or any suitable computing device capable of implementing the technologies described herein. Computing device 120 may receive one or more images. For example, image 110 may be received by computing device 120. Image 110 may include an image of a document, a page of a document, or a portion of a page of a document. The document page or portion of a document page shown in image 110 may include one or more variable text fields. The example might want to detect various document fields within a document. Image 110 may be provided as input to computing device 120.

[0029] In one application, computing device 120 may include a field detector 122. Field detector 122 may include instructions stored on one or more tangible computer-readable storage media of computing device 120 and executed by one or more processing devices of computing device 120. In one application, field detector 122 may generate as output a number of detected fields. , content extracted from the detected fields, and/or an output document with some number of detected fields and content matching the detected fields. In one application, field detector 122 may use a trained machine learning model 140 that is trained to detect fields in image 110. Machine learning model 140 may be trained with a training set of images. In some cases, the machine learning model 140 may be part of the field detector 122, or it may be accessed on another machine (eg, server 150) through the field detector 122. Based on the output (eg, heat maps corresponding to image pixels) of the trained machine learning model 140, the field detector 122 may determine a candidate region in the input image 110 that is detected as a particular field. Field detector 122 may also extract words related to the detected field.

[0030] The server 150 may be a rack-mounted server, a router computer, a personal computer, a personal digital assistant, a mobile phone, a laptop, a tablet computer, a camera, a camcorder, a netbook, a desktop computer, a media center, or any combination of the above. The server 150 may include a field detection learner 151 . The machine learning model 140 may denote model instances created by the field detection learning device 151 using training data that includes training inputs and corresponding given outputs (correct responses to corresponding training inputs). During training, patterns can be found in the training data that tie the training input to a given output (the answer to be predicted) and are subsequently used by the machine learning model 140 for future predictions. As described in more detail below, the machine learning model 140 may consist of, for example, a single layer of linear or non-linear operations (e.g., a Support Vector Machine [SVM]) or may be a deep network, i.e., a machine learning model consisting of from several levels of nonlinear operations). Examples of deep networks are neural networks, including convolutional neural networks, recurrent neural networks with one or more hidden layers, and fully connected neural networks.

[0031] The machine learning model 140 can be trained to determine the probability that image pixels belong to a certain document field, as will be described below. Once machine learning model 140 has been trained, it can be connected to field detector 122 to analyze image 110. For example, field detector 122 may request heat maps for a number of keywords in image 110. In some examples, model 140 may consist of a convolutional neural network. The field detector 122 may receive one or more outputs from the trained machine learning model 140. The output result can be a set of hypotheses about the location of document fields based on heat maps.

[0032] Store 160 may be a persistent store capable of storing image 110, heat maps, pivots and points, document margin hypotheses, detected margins and output images, and data structures for labeling, organizing, and indexing image 110. Store 160 may reside on one or more storage devices such as main memory, magnetic or optical disks, tapes or hard drives, NAS, SAN, and so on. Although storage 160 is depicted separately from computing device 120, in one application, storage 160 may be part of computing device 120. In some applications, storage 160 may be a network file server, while in other applications, storage 160 may be another type of persistent storage, such as an object-oriented database, relational database, etc., that can be hosted on a server or on one or more different machines connected to or connected to the network 130.

[0033] In FIG. 2 shows a high-level block diagram of an exemplary method 200 for detecting fields in a document, in accordance with one or more embodiments of the present invention. From a high level, the operation of the system can be logically divided into 2 stages. The first step is the learning step 220, the second step is the input document field detection step 230. The learning step 220 may take as input different documents 210 containing different fields. On the example of Fig. 2, documents 210 include a bank account template 212, a savings deposit 214, a receipt 216, an invoice 218, and others. Each of the documents 210 contains several fields. For example, the bank account template 212 includes fields in the form of a table 212a with two columns and several rows, the account 218 includes a graphical logo field 218a, a text field with numbers 218b, and others. documents is processed so that the system can learn from the markup of fields in these documents. One or more models may be generated in training step 220 to detect fields in documents. In step 230 of detecting input document fields, the system processes the input document to detect the structure of the input document, detects the field(s) within the input document based on the models obtained in learning step 220, and extracts fields with its content.

[0034] In FIG. 3 is a block diagram of various components of an exemplary document field detection system 300 in accordance with one or more embodiments of the present invention. In some applications, a user, such as a human-user or user-computer system, may define a small number of documents containing one or more fields. The user may define each type of document in which field detection is performed by the system 300. In one application, the user may mark up fields in certain documents. To mark a field, a user may draw lines, circles, frames, rectangles, or other shapes, highlight, or otherwise create markup on or around a portion of a document to designate an area as a particular field. The user can mark several fields that need to be trained for their detection for each document, for example, the fields "Total" and "Address". In addition, custom markup can also include the definition of marked up fields (for example, "Date", "Total", etc.). A "small amount" as used in this application may be, for example, 3-5 documents. In the example, the user has marked up no more than ten documents of a certain type. The user can mark all fields in the document or mark selective fields on the document. Each field is defined and marked up independently of the other fields in the document.

[0035] In some applications, the user may then upload certain documents to the system 300 as electronic documents. Electronic documents can be scanned images, photographed images, or any other representation of a document that can be converted into a computer-accessible data form. Downloaded documents are referred to as document images 310 in this application. In one use case, the user may download documents that already contain marked up fields. In another application, a user may electronically mark up document images 310 using the user interface of system 300. word, numbers, etc., and the system can then automatically mark the surrounding field boundaries. In some applications, received documents can be automatically grouped into different pre-clusters so that each cluster has similar documents, which in turn can help the user to correctly label the fields. The system 300 associates each document image of the document images 310 with metadata identifying a particular field of the document from the markup in the document image. In some examples, the metadata defines a field in the document that contains the text of the variable. In some applications, the system 300 assigns each document image 310 to a specific document class using the document classifier 320 . For example, document images may be assigned to a particular class based on the similarity of document characteristics. In one example, images of a document may be classified based on the name of the merchant associated with the document. For each class, the system 300 collects a small selection of document images (eg, 2-6 document images).

[0036] In some applications, the word selection sub-unit 330 of the system 300 is a sub-module that uses a heuristic algorithm to parse the text of a document. Text can be analyzed for selected words in the document layout based on character types such as letters, numbers, delimiters, etc. Heuristics can include problem solving by experimentation and/or trial and error. A typical heuristic algorithm is derived by some function that is included in the system to find a solution, often using decision trees. The algorithm may include steps to adjust the weights of the decision tree branches depending on the probability that the branch will lead to the final goal node. Here heuristics can be used to split lines of text into groups of the same character type. For example, a cascading classification of text fragments in a document can be represented using a graph. Text fragments are graph nodes. Graph nodes are connected by edges (logical connections between text fragments). The graph can be parsed and modified to further break down the text chunks that were originally defined at each node. For example, a piece of text containing both letters and numbers can be split into two new nodes to separate letters from numbers. In one application, block 330 receives text from document image 310 and separates the document text into a contiguous subsequence of characters. Subsequences of characters can refer to the same character type. For example, character types may include letters, numbers, and delimiters. In subblock 330, you can split the text into individual words. In subblock 330, all possible words in document image 310 can be obtained.

[0037] In some applications, system 300 uses field component classifier 340 to classify each word in document image 310 based on the likelihood of that word being included in a particular field. In some applications, anchors on a document image are used to define the location of a document margin. Any structural element related to the layout of the document can be used as a reference. The datum may include a predefined word (eg, a keyword), a predefined graphic element (eg, visual divider, logo), etc. In some applications, a set of datums may be derived from a training set of document images. In some examples, a special dictionary of "frequent words" can be formed based on the training documents. Frequently occurring words is a list of words grouped by frequency of occurrence in a corpus of documents (for example, training sample documents). For example, frequently occurring words can be grouped into a ranking list. In some examples, pivots on document images may be determined by matching frequently occurring words that appear on one or more document images with a frequency greater than a predetermined threshold frequency. In some examples, pivots may be defined by custom word dictionaries, various word character separators, stamps, and other predefined text and/or visual elements in a document image.

[0038] The reference element may act as a focal point with respect to which the location of the document margin is determined. For example, the center of a rectangular area enclosing a frequently occurring word in a document image may be defined as the location of the reference element in that document image. In other examples, any other location corresponding to the anchor may be designated as the location of the anchor. On the example of Fig. 2, the location (e.g. center) of the rectangular area 218c surrounding the word "totaltotal" in document 218 can be used as the location of the "totaltotal" pivot.

[0039] In some applications, the location of a document margin (also referred to herein as a "margin area") may be determined in relation to an anchor element. On the example of Fig. 2, the location of the document field (eg, location of numbers 218b) corresponding to the "total" anchor element may be determined with respect to the location of the "totaltotal" anchor element in the document. For each document image in the training set of document images (eg, document image 310), the location of the document field may be derived based on the document image markup of the training sample. In one application, system 300 associates metadata with a particular field in a document based on the user's markup on the document. The metadata may define the location of a document field relative to a reference element. For example, the metadata for a particular markup document image may indicate that the position of the document margin is 50 pixels to the right of the anchor element position on the document image. The location of a document field may be expressed as a range of locations, number of pixels, and so on. The location of a document field may include an area in a document image. A region can be a region within a document image. The region may have a specific geometric shape, including, for example, a rectangle, quadrilateral, ellipse, circle, other polygon, etc. The location of a document field may refer to an area on the image of the document contained in the document field.

[0040] In some applications, a heat map is used to determine the likelihood of a word in a document image being included in a particular field. For each given field in the training dataset, a heat map can be generated for each pivot. "Heat map" is a set of numerical elements, compiled in such a way that the value of each element is determined by a specific function, calculated from the image coordinates that reflect the position of the element. In some applications, a heat map may be represented by a rectangular matrix in the form of a table, grid, etc. Each element of the heat map corresponds to a specific pixel in the vicinity of the reference element, such that the value associated with each pixel reflects the number of training documents in which this field contains that pixel. A heat map can be represented using various data structures. For example, a heat map can be represented using bar charts, charts, grids, graphs, drawings, etc. A heat map is a data visualization technology that shows the magnitude of a phenomenon using color in two dimensions. The numeric values of the heatmap elements can be color-coded for visualization (hence the name of the term), but this step would be redundant for training neural networks that use numeric values instead of colors.

[0041] In some applications, a heat map may be generated for each reference element in the image set 310 of the training document. The heat map is generated using the location of the document field relative to the reference element based on the metadata for images of the training document. For example, the location of a document field may be represented by specific pixels in an image included in a box surrounding the document field, as determined by the markup on the document image. A heat map is represented by a data structure that includes several heat map elements. For example, a heat map can be created by dividing an image into a rectangular grid with a given cell size in pixels. In this example, the grid is a heatmap data structure and the cells are heatmap elements. The image used for the heatmap may correspond to each of the training document images, and each of the multiple heatmap elements may correspond to each of a number of document image pixels of the corresponding training document image. In the example, for each pair of pivots and document field location in the training set of document images, the cell is filled with a value equal to the portion of the area occupied by the area for the document field contained in the cell.

[0042] In one application, for a selected reference element for which a heat map is being built, the relative position of the field corresponding to the reference element is set in each of the images of the training document. For example, in the hypothetical image of the first training document, 50 pixels to the right of the location of the “total” reference element, you can find the numerical value “$1000”. For the heatmap data structure, for the reference element "total", it is set whether each pixel of the document image in the first image corresponding to each element of the heatmap (for example, a cell) is included in the location of the document field, determined by the markup on the document image. If any document image pixel is entirely contained within the document field location (eg, the area covered by the document field), the heatmap element corresponding to that document image pixel is set to "1". For example, the value of a cell is set to "1" when the cell corresponds to an image pixel in the document contained in the marked-up portion of the document image covering the area for "$1000". The cell value is set to "0" if it corresponds to an image pixel in the document that is not occupied by the "$1000" field area. In one application, the value set in a cell indicates the number of images in a document in which the field contains a pixel corresponding to a heatmap element. Thus, the heatmap element stores a counter of the number of document images, in which the document field contains the document image pixel associated with the heatmap element.

[0043] System 300 continues to update the heat map for the selected reference element using the next document image in the training set of document images. The values of the heatmap elements are updated so that new values can be added to reflect the next image in the document. For example, if the value of a particular cell in the heatmap has already been set to "1" and the cell corresponds to an image pixel in the next document that is contained in the field "$1000", then the value of the cell is incremented by the value "1" and equals the value "2" . The system 300 continues aggregating the heatmap element values for each of the document images in the document image training set to determine the image pixels contained in a particular document field. In some applications, the final histogram of the relative location of the selected reference element is the arithmetic mean of the values in the corresponding cells of the heat maps.

[0044] In some applications, the system 300 may update the heatmap for the selected reference element so that it can include heatmap element values that are related to a different document field. That is, for the selected reference element, for which the heat map is being built, the location of another field is determined from all images of the training document. The location is relative to the selected heatmap reference element. For example, the location of the "invoice date" entered relative to the "total" reference element can be determined on a heatmap by setting the value of the heatmap elements to "1", where the heatmap elements correspond to the image pixels contained in the "invoice date" field on the first training image. Similarly, heatmap element values are aggregated for each additional document image in the training set to position the "invoice date" field relative to the "totaltotal" reference element. Thus, a heat map for a selected pivot may determine the potential locations of each document field in relation to the selected pivot. Accordingly, during the training phase, heat maps can be generated for a relatively small set of training documents, which are accompanied by metadata indicating the location of the document's fields. The generated heat maps can later be used to determine the location of fields in other documents.

[0045] In FIG. 4A shows an example heatmap 401 for a selected reference element. Anchor 410 is a predefined keyword "date" that occurs in the training set of document images. The heat map 401 defines the location of the various fields of the document relative to the element 410. The reference element 410 is shown in dotted lines because the reference element is not part of the grid data structure 420 that is the heat map 401. Rather, the reference element 410 is a position on the grid that corresponds to the location of the "date" keyword on the training set of document images. Heatmap elements, such as cells 431 and 432, correspond to image pixels in training document images that are contained in various document fields. In the example, cell 432 is shown darker than cell 433, indicating that the count for cell 432 is greater than the count for cell 433, which in turn indicates that more of the document's images have image pixels corresponding to cell 432 contained in the corresponding field rather than cell 433. Similarly, in FIG. 4B shows an example heatmap 402 for a selected pad 411 with the keyword "Total" and the locations of the various fields defined with respect to the pad 411 indicated by the shaded cells. In this example, cell 442 is shown darker than cell 443. In FIG. 4A and 4B show the grid data structure used for the heat maps depicted therein. In the example, the grid size is a hyperparameter, such as 64x64 pixels. The hyperparameter also has a different value, such as 32x32px, 16x16px, etc. The hyperparameter can be selected from values that depend on, for example, the document itself (number of marked up fields, text size, etc.), document layout, etc. . P.

[0046] In some applications, the system 300 uses the characteristics of the heat map to classify each possible word found in the document images 310 to determine the likelihood that the word will be contained in a particular region of the field. The classification consists of positive and negative examples. Good examples are words included in a specific area of a field according to the coordinates of the field (eg x-axis, y-axis) in the document. Negative examples are all words that are not included in a specific area of the field. As localizing features of the hypothesis generated by the field component classification block 340, the locations of specific areas of the fields defined on the heat maps in relation to the reference elements are used. At the output of block 340, one or more sets of hypotheses about the field components are generated. Hypotheses may indicate the likely location of a document field within the document in relation to the reference element. The probable location is established based on the positive examples identified by the heat maps.

[0047] In some applications, the system 300 evaluates the internal format of the extracted content of certain fields in the training set of document images 310 using BPE (Byte Pair Encoding) tokens. The BPE token refers to a numeric vector representing the input text. In some applications, a vector may be represented as an embedding of an intermediate representation of the input text such that the intermediate representation may use an artificial alphabet, each character of which may encode a substring of one or more characters of the input text, as described in more detail below. Embedded data is formed in such a way that semantically similar input data will produce numerically similar embedded data.

[0048] In FIG. 5 shows an illustrative example of an internal field format evaluation 500 in accordance with one or more embodiments of the present invention. In some use cases, system 300 uses BPE tokens to evaluate the internal format of a detected field. In traditional systems, BPE tokens are commonly used in natural language processing tasks. Embodiments of the present invention use BPE tokens to evaluate the internal format of a field for more accurate and confident field detection in a document. The use of BPE tokens for evaluation results in a significant improvement in the field discovery mechanism by improving the quality and speed of field discovery in documents.

[0049] In some applications, within the field internal format evaluation 500, the use of BPE tokens 510 allows obtaining characteristics describing the internal content format (eg, variable text, words) of detected fields in document images. The system 300 may use the string tokenization mechanism of the multilingual dictionary 520 of BPE tokens. The dictionary 520 may include pre-trained embeddings 530 and a dictionary 540 of a pre-trained specific word list by frequency. Both embeddings 530 and vocabulary 540 are pre-trained on the body of text fields of an existing markup database. In the example, the arithmetic mean of the embedded objects is taken as a vector of 550 features of the text string object of the tokens included in the string.

[0050] In some applications, BPE tokens are used for the content of detected fields in the training dataset (eg, an image of a training document). As noted above, an artificial character alphabet can be obtained for use as BPE tokens (eg in encodings). The alphabet includes individual characters and tokens of two characters, three characters, and so on. In the example, the alphabet may include a thousand or more characters representing various combinations of characters. Each word or characters in a word can be represented in the training document images using characters from a derived alphabet representing the word or characters to produce labeled content. BPE inserts are then output, which are a vector expression of BPE tokens. Words in the training document set that are semantically closer to each other (eg "1000" and "2000") will produce numerically close insertions (eg the Euclidean distance between two vectors is less than a predefined value). For example, one document might have "1000" in the "total" field, while another document might have "2000" in the "total" field. When applying BPE tokens to these values, the BPE inserts will be close to each other. As a result, it will be possible to confirm that the values are correctly defined as the contents of the "total" field. When the system processes the input document to detect fields, the sum (e.g. rms, mean, etc.) value of BPE tokens for the values "1000" and "2000" of the training document is taken into account for comparison with the values in the detected fields of the input document. If the detected field contains a value (eg "2500") whose BPE insertion is close to the total BPE value in the training documents, then the field detection can be more likely to be confirmed. You can specify a threshold range to determine whether the value in the input document is close to the aggregated value. Also, if the discovered field contains multiple words, BPE tokens can be used in the same way to compare against the discovered field's reference insert.

[0051] Returning to FIG. 3, in some applications, system 300 applies a component relationship classifier 350 to the derived hypotheses from block 340. For each pair of components (eg, words), block 350 computes an estimate of the pair's co-occurrence in a hypothesized document field. For example, a hypothetical field can include multiple words, such as an address field that includes street number, street name, city, state, country, and so on. In addition, the possible locations of fields that have been defined in heat maps, may include multiple words. Accordingly, each hypothesis includes a sequence of one or more words from several words corresponding to the location of the field.

[0052] In some applications, system 300 applies a filtering and hypothesis evaluation block 360 to the received data from block 350. Block 360 uses additional field parameters to filter and evaluate hypotheses obtained from block 350. For example, additional parameters may include a multi-page indicator (for example, to indicate that the field is on more than one page of the document), one-sided indicator (for example, to indicate that the content is on only one side of the document), two-sided indicator (for example, to indicate that the content is on both sides of the document ), the maximum and minimum size of the geometric field, as well as other features, both individually and in combination with each other. In one application, the options can be set by a user of the system 300. For example, the user can set the option to have the content of a field be multi-line or single-line. In the example, the user can set a parameter indicating that the "date" or "total" field in the invoice document can only be represented by one line. In another application, system 300 may define a parameter associated with a document type. For example, system 300 may define parameters such as field geometry, thresholds, etc. These parameters may be determined heuristically. A typical heuristic algorithm is obtained by using some function included in the system to find a solution, often using decision trees. The algorithm may include steps to adjust the weight of the decision tree branches depending on the probability that the branch will lead to the final goal node. System 300 may also use various combinations of parameters or methods. In addition, linear classifiers based on logistic regression (eg, trained classifiers based on gradient boosting) can be used in system 300 as a component classifier and a hypothesis classifier for a document field.

[0053] In some applications, the system 300 uses a hypothesis quality evaluation and analysis unit 370 to evaluate the overall quality of the hypotheses received. In some examples, the hypotheses are taken from block 360. In other examples, the hypotheses may be taken from another block of the system 300. Various thresholds can be defined to evaluate the quality of the hypotheses in the system. For example, a set of words may be determined to be in a "confident" group when the probability that the set of words belongs to any field in the classification output is above a predetermined threshold. For example, the predetermined threshold value may be 0.5, 0.4, or another value that has been established using an expert method such as evaluating a test sample document, or a value that is a single predetermined hyperparameter (for example, a parameter derived from a predistribution ).

[0054] Assessing the quality of the hypothesis may include determining the "confidence" of the word and the words that are "reliably" associated with the confident word. For evaluation, the starting component from the hypotheses is selected, and to limit the search for other words, the search area around the hypotheses is selected. For example, to start building one or more hypotheses, one or more "confident" words are selected. The search area is a rectangle describing the word with borders that lag behind the borders of the start component by the distance of the specified maximum field size. Then a "confident chain" of hypotheses is assembled, including all words "reliably" associated with the starting component lying in the search zone. Then the final one or more hypotheses are collected and evaluated, which include all words from the search area that are "reliably" associated with all components of the "sure chain". The "reliability" of a word association can be set using a heuristically configurable threshold value, such as determining which value above a configured threshold value is accepted as well connected, which value below a configured threshold value is accepted as not well connected, etc. For example, a predetermined threshold the value can be obtained by training on a large sample of data from the client using a suitable machine learning method, as well as by checking its value using cross-validation or other suitable method. In addition, further analysis of documents can be performed by comparing hypothesis fields with the highest quality compared to other hypotheses.

[0055] In some applications, system 300 uses a field detection and extractor 380 to detect and classify fields on other input document(s) 380. System 300 may determine fields according to selected potential field hypotheses with a quality value that meets a certain quality threshold. , and (or) the results of the analysis by the internal format of the content within the potential fields. For example, when system 300 receives input document 380 to detect and extract fields, system 300 can detect fields on input document 380, classify the fields, and extract content within the detected fields. In some examples, the system 300 may automatically upload a document image with the detected fields and retrieved content to the data store and/or software portal. On FIG. 6 shows an illustrative example of an input document 600 with detected fields 610 in accordance with one or more embodiments of the present invention. System 300 detects fields 610 based on hypotheses generated by the training set of document images.

[0056] Returning to FIG. 3, in some applications, system 300 may acquire an input document image and may determine, within the input document image, a candidate region for each field of interest based on heat maps generated for that field against one or more reference elements. Each defined candidate area will include input document image pixels corresponding to heatmap elements that satisfy a threshold condition (e.g., when their respective values exceed a threshold, when a predetermined proportion of pixels having the highest values is selected, etc.). The selected candidate regions can then be considered as the positions of the respective fields, for example, by applying optical character recognition (OCR) techniques to image fragments lying within the candidate regions.

[0057] In some examples, two or more heat maps can be used to detect fields in which each heat map is built for different reference elements. For example, to locate a particular field (eg, the field corresponding to the "Invoice #" reference element, hereinafter referred to as the "Invoice #" field) in the new input document image 380, keywords from the document image are first determined using a keyword dictionary. For example, specific keywords might include "Date", "Total", "Balance due", "Due date", etc. System 300 selects heatmaps for each of the keywords and determines the probable position for a particular field in a document image. . For example, system 300 selects a heat map for the "Date" pivot and determines the probable position of a particular field (eg, the "Invoice #" field) relative to the "Date" pivot. The probable position of the field is determined on the basis of hypotheses formed on the basis of the values of the elements of the heat map that are above the threshold values. Similarly, system 300 selects a heatmap for the "Total" pivot and determines the probable position of a particular field (e.g., the "Invoice #" field) relative to the "Total" pivot, etc. for heatmaps of the various keywords found in the input. document. The system 300 then compares the heat maps and determines the intersection of the possible points identified by the different heat maps. The system 300 selects one or more points on the input document that correspond to the maximum number of overlapping heatmaps and determines an area, including the points, as a candidate area for a particular field (eg, "Invoice #" field) in the input document image. A threshold number may be specified for the number of intersecting points. If the number of intersecting points on the heat maps meets or exceeds the threshold number, then the selected site should be included in the candidate area. Accordingly, the candidate region is defined as a specific field in the input document.

[0058] In some applications, the content extracted from each detected field of the document can be estimated using BPE tokens by determining differences (for example, Euclidean distances) between the BPE token representing the extracted content of a given field of the input document and the BPE tokens calculated for the same fields in training documents. If the estimated distance between the BPE token representing the content extracted from the candidate field and the population (e.g. rms, mean) of the BPE token(s) representing the same field in the training dataset is less than a threshold value, then the probability that that the field will be detected correctly is relatively large, and the candidate field can be taken to extract information.

[0059] In some applications, the output of various modules can be connected for a priori analysis of a document (for example, document types such as invoices, tables, receipts, key-values, etc.). Custom rules can also be used that describe knowledge about the nature of the input document (for example, country code, page number, etc.).

[0060] In some applications, after receiving a large set of documents (eg, several thousand documents), the system 300 may repeat the training process, but with errors identified in the field detection process. This can further improve the quality of field detection.

[0061] In some applications, the system 3400 may determine the accuracy of user markup on training documents and correct any inaccuracy found. Documents with custom margins are accepted as training input. The marking in the batch can be complete (correct), partial, inconsistent (the same fields are marked in different places), erroneous. This markup is exactly the markup that the user performed. For each labeled field, possible stable combinations of the relative position of other fields are determined based on the labeling of other fields, the search for these fields by the system, and various keywords (frequently occurring words included in the field area). The relative position of margins can be set by absolute location (for example, how it relates to the document on which the margin is located, for example, the line number or pixel designation on the document) or by relative location (for example, compared to a specific element on the document, for example, the "total" field is 100 pixels to the right of the "date" field), or by a zone (for example, a range) of an acceptable location (distribution) of specific fields or keywords (for example, an indication that the "client number" field should always be located to the left of field "client name" and no further than 100 pixels from it, otherwise it is not considered the value of this field). Fields for which stable combinations of other fields and keywords exist, and for which these combinations are repeated or correlated from document to document, are considered stable and possibly well-tagged. Fields for which no stable patterns were found are considered either incorrectly labeled or single labeled. Fields of the same type (for example, “total”) with different stable structures or combinations on different sets of documents are considered either inconsistent (if these are documents of the same cluster or seller) or show heterogeneity in the documents on which they are located. In this way, the system can validate the received markup and predict the markup with a high level of confidence at the first run of the system with a small number of documents required to start training the system, provided that the system contains a generic pre-trained markup machine learning model containing standard rules regarding intended types of user documents. Later, when collecting user markup statistics, the model is trained on user documents in the opposite direction, knowing stable combinations of fields and keywords, the system can determine the areas of possible location of unmarked fields or incorrectly marked fields and give the user hints. For example, the system may provide hints on how to properly mark up a particular document, or download a collection of documents where the markup is obviously wrong and needs to be corrected.

[0062] In FIG. 7 is a flow diagram of one illustrative example of a method for segmenting a document into blocks of various types, in accordance with one or more embodiments of the present invention. Method 700 and/or each of its individual functions, procedures, routines, or operations may be performed by one or more computer system processors (eg, computer system model 800 in FIG. 8) executing the method. In some applications, method 700 may be performed by a single processing thread. In another application, method 700 may be executed by two or more processing threads, each of which performs one or more separate functions, procedures, subroutines, or method steps. For example, the processing threads implementing method 700 may be synchronized (eg, using semaphores, critical sections, and/or other thread synchronization mechanisms). In another application, the processing threads that implement method 700 may execute asynchronously with respect to each other. Therefore, if in Fig. 7 and its associated description lists the operations of the method 700 in a particular order, various applications of the method may perform at least some of the described operations in parallel and/or in an arbitrary order. In one application, method 700 may be performed by one or more of the different components depicted in FIG. 1, such as the field detector 122, the field detection learning device 151, etc.

[0063] At 710, the computer system using the method may obtain a training data set. The training dataset can contain multiple document images. Each of the multiple images in a document can be associated with corresponding metadata that defines a field in the document. A document field can contain characters such as variable text. For each of the multiple images of a document, the metadata may define the marked-up location of a document field corresponding to that document field.

[0064] At step 720, the computer system may generate a first heat map. A heat map can be represented by a data structure (e.g. grid, graph, etc.). The data structure may include several heatmap elements (eg cells) corresponding to several pixels of the document image. In some examples, each heatmap element contains a counter. The counter can specify a number of document images where the document field contains the document image pixel associated with the heatmap element.

[0065] At step 730, the computer system may obtain an image of the input document. At 740, the computer system may determine a candidate region within the input document image. The candidate image may contain a document field. In some examples, the candidate region contains several pixels of the input document image. In some examples, pixels in an input document image correspond to heatmap elements that meet a threshold condition.

[0066] In some examples, a candidate region is determined using multiple heat maps. Multiple heat maps may contain a first heat map and one or more additional heat maps. Each of several heatmaps defines the potential location of a document field corresponding to that document field. Each of the multiple heatmaps defines the potential location of a document field relative to the corresponding reference element in each of the multiple heatmaps. In some examples, the corresponding anchor element contains a predefined word (for example, the keyword "Date") or a predefined graphic element (for example, a visual divider, logo, etc.).

[0067] In addition, during the training step, the computer system may retrieve the content of each image of the training document, where the content is included in the potential location of the document field (for example, the content of "1000", "2000", and "1500", respectively, in each document, where this content is contained in the potential location of the "total" field). The computer system can then parse the content of each document image using BPE tokens. In one application for content analysis, a computer system may represent the content of each document image using BPE tokens to obtain tagged content for each document image. The computer system may generate a vector representation (eg BPE embeddings) of the tagged content for each document image. The computer system may calculate a distance (eg, Euclidean distance) between a pair of embeddings (eg, embeddings representing "1000" and "2000") from two of the multiple document images. If the distance is determined to be less than a predetermined value, then the computer system indicates that the potential location of the document margin is likely to be correct. At the stage of discovering the fields of the input document, when a candidate field (for example, “total”) is detected on the input document, the content (for example, “2899”) of the detected field can be extracted using the training model, and BPE embeddings of the extracted content can also be generated. You can calculate the sum of the BPE value of embeddings to content (for example, content "1000", "2000", and "1500" in training documents) fields in a set of training documents. If the calculated distance between a BPE token representing the content extracted from a detected field in the input document and the aggregate BPE token(s) representing the same field in the training dataset is less than a threshold value, then the probability that the field was detected correctly is relatively is large, and the candidate field can be taken as the discovered field and selected for information extraction.

[0068] In FIG. 8 depicts an exemplary computer system 800 that can perform any of the methods described herein in accordance with one or more embodiments of the present invention. In one example, computer system 800 may be associated with a computing device capable of implementing method 700 of FIG. 7. Computer system 800 may be connected (eg, over a network) to other computer systems on a local area network, intranet, extranet, or the Internet. Computer system 800 may operate as a server in a client/server network environment. Computer system 800 may be a personal computer (PC), tablet computer, set-top box (STB), personal digital assistant (PDA), mobile phone, camera, camcorder, or any device capable of executing a set of instructions (sequential or otherwise) that specifies actions that must be undertaken by this device. In addition, although only one computer system is illustrated, the term "computer" also refers to any collection of computers that individually or collectively execute a set (or multiple sets) of instructions to implement one or more of the methods discussed in this application.

[0069] An exemplary computing system 800 includes a processor 802, a memory 804 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM), such as synchronous DRAM (SDRAM)), and a device 818 storage units that communicate with each other over the 830 bus.

[0070] The processor 802 is one or more general purpose processing devices such as a microprocessor, a central processing unit, etc. In particular, the processor 802 may be a complex instruction set microprocessor (CISC), a reduced instruction set microprocessor ( RISC), a microprocessor with extra long instruction words (VLIW), or a processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processor 802 may also be one or more dedicated processing devices such as an application specific integrated circuit (ASIC), a field-programmable logic integrated circuit (FPGA), a digital signal processor (DSP), a network processor, and the like. Processor 802 is configured to follow the instructions to perform the operations and steps described in this application.

[0071] Computer system 800 may also include a network interface device 822. Computer system 800 may also include a video display unit 810 (eg, a liquid crystal display (LCD) or cathode ray tube (CRT)), an alphanumeric input device 812 (eg, a keyboard), a cursor control device 814 (eg, a mouse), and a device 816 signal conditioning (eg network). In one illustrative example, the video display unit 810, the alphanumeric input device 812, and the cursor control device 814 may be combined into a single component or device (eg, an LCD touchscreen display).

[0072] Storage device 818 may include a computer-readable medium 824 that stores instructions 826 containing any of the techniques or functions described herein. Instructions 826 may also reside, in whole or at least in part, in memory 804 and/or processor 802 during their execution by computer system 800, memory 804, and processor 802, which are also computer-readable media. Further, the instructions 826 may be transmitted or received over the network via the network interface device 822.

[0073] While the computer-readable storage medium 824 is shown in the illustrative examples as a single medium, the term "machine-readable storage medium" should be understood as including one or more storage media (for example, a centralized or distributed database and (or) associated with caches and servers) that store one or more sets of instructions. The term "computer-readable medium" also means any medium that is capable of storing, encoding, or carrying a set of instructions for execution by a machine and that causes the machine to perform one or more of the techniques of the present invention. Accordingly, the term "computer-readable storage medium" includes, in particular, semiconductor memory, optical and magnetic media.

[0074] Although the procedures for performing the various methods presented herein are shown and described in a particular order, the order in which each method is performed can be changed so that some procedures can be performed in reverse order or so that some procedure can be performed, at least in part, along with other procedures. In some applications, statements or subroutines of individual procedures may be interrupted and/or interleaved.

[0075] It should be understood that the above description is to be illustrative and not restrictive. Many other uses will be apparent to those skilled in the art upon reading and understanding the foregoing description. Therefore, the scope of the invention must be established with reference to the appended claims, together with the full scope of the equivalents to which such claims may fall.

[0076] Numerous details are given in the above description. However, it will be apparent to those skilled in the art that embodiments of the present invention may be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form rather than described in detail so as not to obscure the present invention.

[0077] Some parts of the above detailed descriptions are presented in the form of algorithms and symbolic representations of operations on data bits in computer memory. These algorithmic descriptions and representations are the means used by data scientists to most effectively convey the substance of their work to others in the art. The algorithm here and as a whole is conceived as a self-consistent sequence of steps leading to the desired result. These are the steps that require physical manipulation of physical quantities. Usually, though not always, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise used. It has been shown that sometimes, mainly for reasons of common usage, it is convenient to refer to these signals as bits, values, elements, symbols, signs, terms, numbers, and the like.

[0078] However, it should be borne in mind that all these and similar terms should be associated with the corresponding physical quantities and be simply convenient designations applied to these quantities. Unless otherwise indicated, it will be clear from the discussion that follows that such frequently used terms in the discussion as "acquisition", "setting", "selection", "storage", "configuration", etc., refer to to the actions and processes of a computer system or similar electronic computing device that controls and converts data represented as physical (electronic) quantities in the registers and memory of a computer system into other data similarly represented as physical quantities in the memory or registers of a computer system, or in other similar devices for storing, transmitting or displaying information.

[0079] The present invention also relates to a device for performing the operations described in this application. This device may be specially designed for the required purposes, or may include a general purpose computer, selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a computer-readable storage medium such as floppy disks, optical disks, compact disks, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, or any other storage media suitable for storing electronic instructions, each of which is connected to the computer's system bus.

[0080] Presented in this application, the algorithms and displays are not inherently associated with any particular computer or other device. In accordance with the guidelines contained in this application, various general-purpose systems can be used with programs, but it may be appropriate to create a highly specialized device to perform the required methodological steps. For various such systems, the desired structure will meet the requirements set forth in the description. In addition, embodiments of the present invention are not considered in the context of any particular programming language. Obviously, various programming languages can be used to implement the present invention described in this application.

[0081] Embodiments of the present invention may be provided as a computer program product or software that may include a computer-readable medium with instructions stored thereon that may be used to program a computer system (or other electronic devices) to perform a process in in accordance with the present invention. A computer-readable medium includes any procedure for storing or transmitting information in a form that is readable by a machine (eg, a computer). For example, machine-readable (e.g., computer-readable) media includes machine-readable (e.g., computer-readable) storage media (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory, etc.).

[0082] The words "example" or "exemplary" are used in this application as an example, example or illustration. Any implementation or design described in this application with the characteristic "example" or "exemplary" should not necessarily be considered as preferred or advantageous over other implementations or designs. The use of the words "example" or "exemplary" is rather intended to present the concept as applied to a particular case. In the form in which the term "or" is used in this application, it means inclusive "or" and not exclusive "or". That is, unless otherwise indicated or clear from the context, "X includes A or B" is used to refer to any of the natural inclusive permutations. That is, if X includes A; X includes B; or X includes both A and B, then in any of the above cases, the wording "X includes A or B" is considered true. In addition, the indefinite articles "a" and "an" used in this application in English and the appended claims should generally be construed to mean "one or more" unless otherwise indicated or it is clear from the context that they must be given in the singular form. In addition, the use of the terms “use case”, “one use case”, or “one of the use cases” in the text should not mean the same use case, unless it is described as such. In addition, the terms "first", "second", "third", "fourth", etc., used in this application, mean marks to distinguish between various elements and do not necessarily have an ordinal value in accordance with their numerical designation.

Claims (7)

A method for determining the accuracy of a custom markup, including: obtaining a training dataset containing multiple documents, each of the multiple documents being associated with multiple user-labeled fields; for a given field of multiple user-marked fields in a given document of multiple documents, determining whether one or more additional documents repeat a specific combination of the relative positions of one or more additional user-marked fields relative to the given field in the given document; an indication of the incorrect selection of this field, if it is established that a particular combination is not repeated in any additional documents; and establishing the presence in two or more other documents of another combination of relative positions of additional one or more fields marked up by the user, relative to this field, if it is found that a particular combination is repeated in one or more additional documents, and: specifying a given field as properly tokenized if it is determined that another combination does not exist in two or more other documents; and specifying a given field as tokenized is inconsistent if the given combination is found to exist in two or more other documents.

Images