CN117396899A - System and method for extracting fields from unlabeled data - Google Patents

Abstract

Embodiments describe a field extraction system that does not require field-level annotations for training. Specifically, the training process is guided by mining pseudo-labels from unlabeled forms using simple rules. Then, a transformer-based structure is used to model the interactions between text tokens in the input form and predict the field labels for each token accordingly. Pseudo labels are used to supervise transformer training. Since pseudo-labels are noisy, a refinement module containing branching sequences is used to refine the pseudo-labels. Each execution field in the refinement branch is marked and a refinement label is generated. At each stage, branches are optimized by aggregating labels from all previous branches to reduce label noise.

Description

Systems and methods for extracting fields from unlabeled data

cross reference

This immediate application claims priority from U.S. Non-Provisional Application Nos. 17/484,618 and 17/484,623, both filed on September 24, 2021, both of which are based on U.S. Provisional Application No. 63/189,579, filed on May 17, 2021 Non-provisional application, claiming priority under 35 U.S.C. 119.

All of the above applications are expressly incorporated by reference in their entirety.

Technical field

Embodiments relate generally to machine learning systems and computer vision, and more specifically to mechanisms for extracting fields from forms with unlabeled data.

Background technique

Form-like documents, such as bills, pay stubs, and patient referral forms, are commonly used in daily business workflows. Extracting fields from various forms is often a challenging task. For example, even for the same form type, the document layout and text representation can be different if the form is issued by a different vendor. For example, bills from different companies may have significantly different designs from different systems (e.g., ADP and Workday ) may have different textual representations of similar information, and/or similar situations. Traditionally, extracting information from such form documents requires a lot of labor. For example, workers are typically given a list of expected form fields, such as purchase orders, bill numbers, and totals, and their corresponding values are extracted based on those fields based on their understanding of the form.

Therefore, there is a need for an efficient system for extracting information from form documents.

Description of the drawings

Figure 1 is a simplified diagram illustrating an example of extracting fields from a bill, according to one embodiment described herein.

Figure 2 is a simplified diagram illustrating the overall self-supervised training framework of the field extraction system according to embodiments described herein.

3 is a block diagram illustrating an example framework for refining the field extraction framework described in FIG. 2 using pseudo-label sets (PLE), in accordance with implementations described herein.

4 is a simplified diagram of a computing device implementing a field extraction framework in accordance with some embodiments described herein.

Figure 5 is a simplified diagram of a method for field extraction from a form with unlabeled data by a field extraction model, according to some embodiments.

Figure 6 is a simplified diagram of a method for label refinement in field extraction from a form with unlabeled data through a field extraction model, according to some embodiments.

Figure 7 is an example key list and date type data table that provides a training data set of unlabeled form data, in accordance with some embodiments.

8A-8B are diagrams illustrating exemplary unlabeled forms in accordance with some embodiments.

Figures 9-16 provide exemplary results of data experiments for the field extraction model described in Figures 1-6, according to some embodiments.

In the drawings, elements with the same name have the same or similar functions.

Detailed ways

Machine learning systems have been widely used in computer vision, for example, pattern recognition, object localization, etc. Some recent machine learning methods formulate form field extraction as field value pairing or field labeling. For example, some existing systems employ representation learning methods that take field and value candidates as input and utilize metric learning techniques to enforce high pairwise scores for positive field-value pairs and low scores for negative field-value pairs. Another system uses a pre-trained transformer that takes text and its location as input. However, these existing methods usually require extensive field-level annotations for training. Obtaining field-level annotations for forms can be quite expensive and labor-intensive, and sometimes even impossible, because (1) forms often contain sensitive information and therefore have limited public data available for training purposes; and (2) due to the exposure of private Information risk, using external annotators is also not feasible.

Given the need for an efficient system for extracting information from form documents, embodiments describe a field extraction system that does not require field-level annotations for training. Specifically, the training process is guided by mining pseudo-labels from unlabeled forms using simple rules. Then, a transformer-based structure is used to model the interactions between text tokens in the input form and predict the field tags for each token accordingly. Pseudo labels are used to supervise transformer training. Since pseudo-labels are noisy, a refinement module containing branching sequences is used to refine the pseudo-labels. Each execution field in the refinement branch is marked and a refinement label is generated. At each stage, branches are optimized by aggregating labels from all previous branches to reduce label noise.

For example, field extraction systems are trained on self-supervised pseudo-labels from unlabeled data. Specifically, the field extraction system detects sets of words and their position in the form, and identifies field values based on geometric rules between words, for example, fields and field values can often be aligned horizontally and separated by colons. The identified field values can then be used as pseudo-labels to train a transformer network that encodes detected words and locations for classification.

In some embodiments, multiple pseudo-label ensemble (PLE) branches may be used to refine pseudo-labels for training. Specifically, the PLE branches are operated in parallel to generate predictive classifications from the encoded representations of detected words and locations. At each branch, a loss component is calculated by comparing the refined label at that branch with the predicted label generated by the "previous" PLE as a pseudo label. The loss components on the PLE branches are then summed to jointly update the PLE.

As used herein, the term "network" may include any hardware or software-based framework, including any artificial intelligence network or system, neural network or system and/or any training or learning model implemented on or with it.

As used herein, the term "module" may include a hardware or software-based framework that performs one or more functions. In some embodiments, this module may be implemented on one or more neural networks.

Figure 1 is a simplified diagram 100 illustrating an example of extracting fields from a bill, according to one embodiment described herein. Traditionally, in form processing, workers are usually given a list of expected form fields, for example, purchase order, bill number, and total, and the goal is to extract their corresponding values based on an understanding of the form. Keys, such as Bill #, PO Number, and Total, refer to the specific textual representation of the fields in the form, and are important indicators of value positioning. The key is often the most important characteristic for value location. Therefore, field extraction systems are designed to automatically extract field values from irrelevant information in forms, which is crucial to improve processing efficiency and reduce manpower.

As shown in diagram 100, the form contains various phrases such as "Bill #", "1234", "PO Number", "000001", etc. The field extraction system can identify "PO number" 102 as the anchor key and then determine whether any of the values "1234" 104, "00000001" 103, or "100.00" 105 match the anchor key. This match may be determined based on the geometric relationship between position key 102 and values 103 to 105. For example, a rule-based algorithm may be applied to determine the match, e.g., the value "0000001" 103 is more likely to be the value corresponding to the anchor key 102 because the value 103 has a position that is vertically aligned with the position of the anchor key 102 .

Unlike previous methods that gave access to large-scale labeling forms, rule-based methods can be used to generate noisy pseudo-labels (e.g., fields and values) from unlabeled data. The rule-based algorithm is built based on the following observations: (1) A field value (e.g., 103 in Figure 1) is usually displayed in a form with some key (e.g., 102 in Figure 1), and the key (e.g., Figure 1 102 in 1) is the specific text representation of the field; (2) the keys and their corresponding values have a strong geometric relationship (as shown in Figure 1, the keys are mostly next to their values vertically or horizontally) ;(3) Although the layout of the form is very diverse, some key-text is usually used in different form instances (for example, the key-text of the field purchase order can be "PO number", "PO#", etc.); and ( 4) Field values are always associated with a certain date type (for example, the data type of the value of "Bill Date" is date, and the data type of the value of "Total" is amount or number).

Therefore, a rule-based approach can be used to generate useful pseudo-labels for each field of interest from large-scale forms. As shown in Figure 1, key positioning 102 is performed first based on a string-match between text in the form and possible key strings for the field. Values 103 to 105 are then estimated based on the data type of the text and its geometric relationship to the anchor key 102 .

Figure 2 is a simplified diagram illustrating an overall self-supervised training framework 200 for a field extraction system according to embodiments described herein. The framework 200 includes an optical character recognition module 205, a transformer network 210, and a classifier 220. An unmarked form 202, such as a check, bill, pay stub, and/or the like, may include field information from a predefined list, {fd ₁ , fd ₂ , ..., fd _N }. Given a form as input, the general OCR detection and recognition module 205 is applied to the unlabeled form 202 to obtain a set of words, {w ₁ , w ₂ , ..., w _M }, with their positions represented as bounding boxes {b ₁ , b ₂ ,…, b _M }. Therefore, the goal of the field extraction method is to automatically extract the target value _vi matching the field fd _i from a large number of word candidates {w ₁ , w ₂ ,..., w _M } if the field information exists in the input form.

The word and bounding box location pairs { _wi ,Bi _} may then be input to the transformer encoder 210 to encode into feature representations. The pairs {w _i , B _i } may also be sent to pseudo-label inference module 215 configured to perform key location and value estimation, key location identifying the location of the key corresponding to each predefined field, and value estimation. Determine the corresponding field value for the targeting key.

For example, since keys and values can contain multiple words, upon receiving word and bounding box location pairs {W _i , B _i }, the pseudo-label inference module 215 can use the DBSCAN algorithm (Ester et al., 1996) based on their locations Group nearby identified words to obtain phrase candidates [ ^{phi 1} _{, phi} ² , ..., _phi ^T ] and their positions [B _i ¹ , B _i ² , ..., B _i ^T ] .

For each field of interest, fd _i , a list of commonly used keys, [k _i ¹ , k _i ² , ..., k _i ^L ], is determined based on domain knowledge. For example, a field name can be used as a unique key in a list. Module 215 may then measure the string distance between the phrase candidate ph _i ^j and each designed key ^{k ir} _as d(ph _i ^j , k _i ^r ). Module 215 may calculate a key score for each phrase candidate that indicates the likelihood of that candidate being the key to the field using the following equation:

Then, locate the key by finding the candidate with the largest key score, as follows:

Pseudo-label inference module 215 may then determine the value (or one or more values, if applicable) of the location key. Specifically, values are estimated based on two criteria. First, their data types should be consistent with their fields. Second, their position should be very consistent with the positioning keys. For each field, a list of qualified data types can be predetermined. For example, for the data field "Bill Number", the data type can include string or integer. A pretrained BERT-based model can be used to predict the data type of each phrase candidate, and only candidates with the correct data type _are ^retained .

In one embodiment, a value score is determined for each qualified candidate ph _i ^j as follows:

where key_score Indicates the key score of the anchor key, /> Indicates the geometric relationship score between the candidate and the anchor key. A key (eg, 102 in Figure 1) and its value (eg, 103 in Figure 1) are typically close to each other, and the value may be directly below the key or to the right of them. Therefore, determine geometric relationships (such as distances and angles) to measure key-value relationships:

in Indicates the distance between two phrases,/> indicates the angle from ph _i ^j to ph _i ^r , and φ(.|μ,σ) indicates the Gaussian function with μ as the mean and σ as the standard deviation. Here, μ _a is set to 0, and σ _b and σ _a are fixed to 0.5. To reward candidates whose angle relative to the bond is close to 0 or π/2, the maximum angle score toward these two options is as follows:

Therefore, as in Equation (5), if the value score of a candidate is the largest among all candidates and the score exceeds a threshold, for example, θ _v =0.1, then the candidate is determined as the predicted value of the field.

In one embodiment, the output of the pseudo-label inference module 215, eg, as an estimate of the field of the pseudo-label, may be used as a separate field extraction output. In another embodiment, the estimated value of a field can be used as a pseudo-label to guide training to further improve field extraction performance. Specifically, in order to predict the target label of a word, it is necessary to learn the meaning of the word and its interaction with the surrounding context. Transformer-based architectures (e.g., LayoutLM described in Xu et al., 2020) can be used to learn representations of words due to its strong ability to model contextual information. In addition to semantic representation, the position of words and the overall layout of the input form are also important and can be used to capture the distinguishing characteristics of words. Transformer encoder 210 can extract features from the input pair {W _i , Bi _} :

[f ₁ , f ₂ ,..., f _M ]=T([(w ₁ , b ₁ ), (w ₂ , b ₂ ),..., (w _M , b _M )]), (6)

where T(.) represents the transformer-based feature extractor, and _fi indicates the features of word i.

The classifier 220 for token classification may receive input of the encoded feature representation from a transformer encoder 210 that generates a prediction field that includes the context of each token from the original unlabeled form 202 . Specifically, the classifier 220 generates the field prediction score _sk by projecting the features into the field space ({background, fd ₁ , fd ₂ , . . . , fd _N )) via a fully connected (FC) layer. The predicted field scores from classifier 220 and the generated pseudo labels from pseudo label inference 215 may then be compared at loss module 230 to generate training targets. The training objectives may be further utilized to update the transformer 210 and classifier 220 via a backpropagation path (shown by the dashed lines).

In one embodiment, as further described in Figure 3, multiple progressive pseudo-label sets (PLEs) can be used to guide training.

Figure 3 is a block diagram illustrating an example framework 300 for utilizing PLE to refine the field extraction framework described in Figure 2, in accordance with implementations described herein. As depicted in Figure 2, the transformer 210 receives an input 302 of a word extracted from the unmarked form 202 and the location of a bounding box surrounding the word, (w ₁ , b ₁ ), (w ₂ , b ₂ ), .. ., (w _M , b _M ), based on this initial word-level field label (also known as Guided labels) are obtained from the estimated pseudo labels at the pseudo label inference module 215. Therefore, the cross-entropy loss L(s _k ,/> can be used ) to optimize the transformer network 210, the cross-entropy loss is calculated based on the field prediction scores from the classifier 220 and the generated guide labels.

However, using only noisy bootstrap labels as ground truth in training may degrade model performance. The transformer 210 is followed by a refinement module 304 that includes a plurality of PLEs, each PLE serving as a classification branch. Specifically, at each branch j, PLE independently performs field classification and refines pseudo labels based on their predictions Refine subsequent branches using the refinement labels obtained from previous branches.

For example, at branch k, the refined labels are generated according to the following steps: (1) Find the predicted field label of each word by argmax(s _kc ) And (2) for each field, a word is retained only if its prediction score is the highest among all words and is greater than the threshold (fixed to 0.1). For example, assume that PLE module 304 includes branches 304a through 304n. The first PLE branch 304a may receive pseudo labels generated from the pseudo label reasoning module 215/> The FC layer generates a field classification score s ₁ based on it, and then converts it into a pseudo label/> Then, bootstrap tag/> and output pseudo tags/> is fed to the second PLE branch 304b, based on which the FC layer generates a field classification score s ₂ and then converts it into a pseudo label/> Following a similar process, the k-th PLE branch receives the boot label/> and all generated pseudo tags/> The FC layer generates a field classification score _sk based on it, and then converts it into a pseudo label/>

Therefore, the final loss aggregates all losses and is calculated as follows:

where β is a hyperparameter that controls the contribution of the initial pseudo-labels.

In this way, progressive refinement of labels reduces label noise. However, using only refined labels at each level yields limited performance improvement because although labels become more precise after refinement, some low-confidence values are filtered out, which results in lower recall. To alleviate this problem, each branch is refined using the set labels of all previous levels. Set labels not only maintain a better balance between precision and recall, but are also more diverse and can be used as a regularizer for model optimization. During inference, the average score predicted from all branches can be used. A similar process can be applied to obtain final field values, such as generating refinement labels.

computer environment

Figure 4 is a simplified diagram of a computing device 400 implementing a field extraction framework in accordance with some embodiments described herein. As shown in FIG. 4 , computing device 400 includes processor 410 coupled to memory 420 . Operation of computing device 400 is controlled by processor 410. And although computing device 400 is shown with only one processor 410, it should be understood that processor 410 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital Signal processor, field programmable gate array (FPGA), application specific integrated circuit (ASIC), graphics processing unit (GPU) and/or the like. Computing device 400 may be implemented as a standalone subsystem, a board added to the computing device, and/or a virtual machine.

Memory 420 may be used to store software executed by computing device 400 and/or one or more data structures used during operation of computing device 400 . Memory 420 may include one or more types of machine-readable media. Some common forms of machine-readable media can include floppy disk, floppy disk, hard drive, magnetic tape, any other magnetic media, CD-ROM, any other optical media, punched cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM , EPROM, flash EPROM, any other memory chip or cartridge and/or any other medium suitable for reading by the processor or computer.

Processor 410 and/or memory 420 may be arranged in any suitable physical arrangement. In some embodiments, processor 410 and/or memory 420 may be implemented on the same board, the same package (eg, system-in-a-package), the same chip (eg, system-on-a-chip), etc. In some implementations, processor 410 and/or memory 420 may include distributed, virtualized, and/or containerized computing resources. Consistent with these embodiments, processor 410 and/or memory 420 may be located in one or more data centers and/or cloud computing facilities.

In some embodiments, memory 420 may include non-transitory, tangible, machine-readable media that includes executable code that, when executed by one or more processors (eg, processor 410) One or more processors may be caused to perform the methods described in further detail herein. For example, as shown, memory 420 includes instructions for field extraction module 430, which may be used to implement and/or simulate systems and models, and/or implement any of the methods further described herein. In some embodiments, field extraction module 430 may receive input 440 via data interface 415, for example, an unlabeled image instance such as a form. Data interface 415 may be a user interface that receives form image instances uploaded by a user, or may be any of a communication interface that receives or retrieves previously stored form image instances from a database. Field extraction module 430 may generate output 450 such as the extracted fields of input 440 .

In some implementations, the field extraction module 430 may further include a pseudo-label reasoning module 431 and a PLE module 432. The pseudo-label inference module 431 mines noisy pseudo-labels from the form using a rule-based approach, for example, as described in Figure 2. The PLE module 432 (similar to the refinement module 304 in Figure 3) can use the estimated values of the fields as pseudo labels during training to learn a data-driven model implemented as a token classification task with a set of extracted from the form Inputs of tokens and output including prediction fields for the context of each token. Further details of PLE module 432 will be discussed in conjunction with FIG. 3 .

Field extraction workflow

Figure 5 is a simplified diagram of a method 500 for field extraction from a form with unlabeled data by a field extraction model, according to some embodiments. One or more of the processes of method 500 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors, may cause a or one or more of the processors executing the process. In some embodiments, method 500 corresponds to the operation of field extraction module 430 (FIG. 4) to perform field extraction or a method of training a field extraction model. As shown, method 500 includes a plurality of enumerated steps, but aspects of method 500 may include additional steps before, after, and between the enumerated steps. In some aspects, one or more of the enumerated steps may be omitted or performed in a different order.

At step 502, an untagged form including a plurality of fields and a plurality of field values is received via a data interface (eg, 415 in Figure 4). For example, the unmarked form may take a form similar to that shown in Figures 8A-8B.

At step 504, word sets and position sets are detected within the unlabeled form of word sets. For example, words and locations can be detected by OCR module 205 in Figure 2.

At step 506, field values for the fields are identified from the set of words and the set of locations based at least in part on geometric relationships between the sets of words. For example, a field value may be identified by applying a first rule that one or more words in the form of a key are related to the field name of the field. For another embodiment, field values may be identified by applying a second rule, namely horizontally or vertically aligned pairs of words that are keys to fields and field values. For another embodiment, the field value may be identified by applying a third rule that a word from a set of words that matches the predefined key text is the key of the field.

In one implementation, key positioning corresponding to the field is determined. For example, by grouping nearby recognized words, a set of phrase candidates is determined from a set of words, and a set of corresponding phrase positions is determined from a set of positions. A key score is calculated for each phrase candidate, which indicates the likelihood that the corresponding phrase candidate is a key to the field. The key score is calculated based on the string distance between the corresponding phrase candidate and the predefined key, see e.g. equation (1). The key of the field is then determined based on the maximum key score in the phrase candidate set, for example, see equation (2).

Specifically, in order to calculate the key score, a neural model can be used to predict the corresponding data type of each phrase candidate. A subset of phrase candidates having a data type that matches the field's predefined data type is then determined. A value score is calculated for each phrase candidate in the subset that indicates the likelihood that the corresponding phrase candidate is the field value of the field. The value score is calculated based on the key score corresponding to the locating key of the field and the geometric relationship measure between the corresponding phrase candidate and the locating key, for example, equation (3). The geometric relationship metric is calculated based on the string distance and angle between the corresponding phrase candidate and the positioning key, for example, equation (4). The field value is then determined based on the maximum score in the subset of phrase candidates.

At step 508, an encoder (eg, transformer encoder 210 in FIG. 2) may encode the pair of first word and first position corresponding to the field value into a first representation.

At step 510, a classifier (eg, classifier 220 in Figure 2) may generate a field classification distribution from the first representation.

At step 512, a first loss target is calculated by comparing the field classification distribution with the field values as pseudo labels.

At step 514, the encoder is updated via backpropagation based on the first loss target.

Figure 6 is a simplified diagram of a method 600 for label refinement in field extraction from a form with unlabeled data through a field extraction model, according to some embodiments. One or more of the processes of method 600 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors, may cause a or one or more of the processors executing the process. In some embodiments, method 600 corresponds to the operation of field extraction module 430 (FIG. 4) to perform field extraction or a method of training a field extraction model. As shown, method 600 includes a plurality of enumerated steps, but aspects of method 600 may include additional steps before, after, and between the enumerated steps. In some aspects, one or more of the enumerated steps may be omitted or performed in a different order.

At step 602, an untagged form including a plurality of fields and a plurality of field values is received via a data interface (eg, 415 in Figure 4). For example, the unmarked form may take a form similar to that shown in Figures 8A-8B.

At step 604, a first word and a first position of the first word within the unmarked form are detected. For example, words and locations can be detected by OCR module 205 in Figure 2.

At step 606, an encoder (eg, transformer encoder 210 in Figure 2) encodes the pair of first word and first position into a first representation, eg, equation (6).

At step 608, a plurality of progressive label set (PLE) branches (eg, see 304a to 304n in Figure 3) generate a plurality of predicted labels based on the first representation respectively in parallel. Each of the plurality of PLE branches includes a respective classifier that generates a respective predicted label based on the first representation. Predicted labels at one PLE branch are generated by projecting the first representation into a set of field prediction scores via one or more fully connected layers, and generating predicted labels based on the maximum field prediction score in the word set. When the maximum field prediction score is greater than a predefined threshold, for a field in multiple fields, the word corresponding to the maximum field prediction score is selected from the word set.

At step 610, a PLE branch calculates a loss component by comparing the predicted label at one PLE branch with the predicted label from the previous PLE branch as a pseudo label.

At step 612, the loss target is calculated as the sum of loss components over multiple PLE branches, eg, equation (7).

At step 614, the plurality of PLE branches are updated via backpropagation based on the loss target. In one embodiment, the first PLE branch from the plurality of PLE branches uses the identification field value from the field of step 506 in Figure 5 as the first pseudo-label. The joint loss target is calculated by adding the loss target to the first loss target calculated at step 512 in Figure 5 . The encoder and multiple PLE branches are then jointly updated based on the joint loss objective.

Example performance

An example training data set could include real bills collected from different vendors. For example, the training set contains 7664 unlabeled billing forms from 2711 templates. The validation set contains 348 labeled bills from 222 templates. The test set contains 339 labeled bills from 222 templates. Each template has a maximum of 5 images in each set. Seven common fields are considered, including bill number, purchase order, bill date, due date, amount due, total amount, and total tax.

For the tobacco test set, 350 bills were collected from the tobacco collection of the industry repository 2 for public release. The validation and test sets of the internal in-bill data set have similar field statistical distributions, while the public tobacco test set does not. For example, bills from the tobacco set (shown in Figure 8A) may have lower resolution and a more cluttered background than other bills in the training dataset (shown in Figure 8B).

The end-to-end macro average F1 score over a field is used as a metric to evaluate the model. Specifically, exact string matches between our predicted values and ground truth values are used to calculate true positives, false positives, and false negatives. The precision-recall and F1 score of each field are obtained accordingly. The reported scores are averaged over 5 runs to reduce the effect of randomness.

Since there are no existing methods to perform field extraction using only unlabeled data, the following baselines are constructed to validate our approach: Guided Labels (B-labels): Initial pseudo-labels inferred using the proposed simple rules can be used to perform field extraction directly , without training data. Transformer is trained using B labels: Since the transformer is used as the backbone for extracting word features, the transformer model is trained using B labels as a baseline to evaluate data-driven models from (1) pipeline and (2) refinement Module performance gain. Both the content of the text and its position are important for field prediction. An example of a transformer backbone is LayoutLM, which takes both text and position as input. Furthermore, two popular transformer models are used, namely, BERT and RoBERTa, which only accept text as input.

The OCR engine is used to detect words and their positions and then sort the words in reading order. Example key lists and date types for each dataset are shown in Table 1 of Figure 7. The key list and data types are very extensive. α is set to 4.0 in equation (4). To further remove false positives, candidate values are removed if the location key is not within its adjacent region. Specifically, the adjacent area around the candidate value extends all the way to the left side of the image, with four candidate heights above it and one candidate height below it. The number of refinement branches k=3 for all experiments. When the number of levels > 1, a hidden FC layer with 768 units is added before classification. For all billing experiments, β in Equation (7) is set to 1.0, except for β = 5.0 for the BERT-based refinement in Table 4 of Figure 11 due to its better performance in the validation set. For the field extraction models and baselines described in this article, the model with the best F1 score in the validation set was selected. To prevent overfitting, a two-step training strategy is adopted, in which the first branch of the model is trained using pseudo-labels and then fixed together with the feature extractor during refinement. The batch size is set to 8 and the Adam optimizer with a learning rate of 5e ⁵ is used.

The proposed model is then validated using the in-bill dataset since it contains large-scale unlabeled training data and a sufficient amount of valid/test data, which is more suitable for our experimental setup. First, LayoutLM is used as the backbone to verify the proposed training method. The comparison results are shown in Table 2 of Figure 9 and Table 3 of Figure 10 . The guided labels (B-Labels) baseline achieves F1 scores of 43.8% and 44.1% in the valid set and test set respectively, which indicates that our B-Labels have reasonable accuracy but are still noisy. When B labels are used to train the LayoutLM transformer, significant performance improvements are obtained—approximately 15% increase in the effective set and approximately 17% increase in the test set. Adding the PLE refinement module significantly improved model accuracy—by about 6% for the valid set and about 7% for the test set—while slightly reducing recall, by about 2.5% for the valid set and 3% for the test set. This is because the refined labels become increasingly confident in later stages, leading to higher model accuracy. However, the refinement stage also removes some low-confidence false negatives that result in lower recall. Overall, the PLE refinement module further improves performance, resulting in a 3% improvement in F1 score.

Then, LayoutLM is used as the default feature backbone since both the text and its position are important to our task. Furthermore, to understand the impact of different transformer models as backbones, two other models, BERT and RoBERTa, were evaluated, which only use text as input. The comparison results are shown in Table 4 of Figure 11 and Table 5 of Figure 12 . It is observed that large improvements are achieved when BERT and RoBERTa are trained directly using B-Labels and PLE refinement modules, resulting in consistently improved baseline results for different transformer choices with different numbers of parameters (basic or large). However, LayoutLM still produces higher results compared to the other two backbones, indicating that text position is indeed very important to obtain good task performance.

The proposed model was then tested using the tobacco test set introduced in Table 6 of Figure 13. The simple rule-based approach achieved an F1 score of 25.1%, which is reasonable but much lower compared to the results on our in-house in-bill dataset. The reason is that the tobacco test set has visual noise, which leads to more text recognition errors. The LayoutLM baseline obtained great improvements when using B-Labels. Additionally, the PLE refinement module further improves the F1 score by approximately 2%. The results show that the proposed method adapts well to different scenarios. In Figures 8A to 8B, it is shown that the proposed method obtains good performance despite the fact that the sample bill for different templates is very diverse, with cluttered background and low resolution.

Using LayoutLM as the backbone, further ablation research was conducted on the bill data set. Effect of the number of stages: The proposed model is refined in k stages while fixing k = 3 in all experiments. It is evaluated using different series. Figure 15 shows that when the number of stages k increases, the model generally performs better on both the valid set and the test set. The performance of multi-stage is always higher than that of single-stage model (our converter baseline). When k=3, the model performance reaches the highest level. As shown in Figure 16, during model refinement, while recall decreases, precision increases. When k=3, the best balance between precision and recall is obtained. When k > 3, the recall decreases more than the precision increase, so worse F1 scores are observed.

Impact of Refinement Labels (R-Labels): To analyze the impact of this design, all refinement labels were removed from the final loss and only B-Labels were used to train the three branches independently and ensemble predictions during inference. As shown in Table 7 of Figure 14, removing the refinement labels leads to a decrease in F1 score of 2.2% and 2.6% in the valid set and test set, respectively.

Effect of B-Labels regularization. At each level, B-Labels are used as a type of regularization to prevent the model from overfitting to overly confident refined labels. B-Labels are used in the refinement stage by setting β = 0 in equation (7). As shown in Table 7 of Figure 14, without this regularization, model performance drops by about 2% in F1 score.

Impact of two-step training strategy: To avoid overfitting noisy labels, a two-step training strategy is adopted, where the backbone with the first branch is trained using B-Labels and then fixed during refinement. This effect is analyzed by training the model in a single step. Single-step training resulted in an F1 score drop of 1.8% and 1.4% in the valid set and test set, respectively.

Some embodiments of a computing device, such as computing device 400, may include non-transitory, tangible, machine-readable media that includes executable code that when executed by one or more processors (eg, processor 410) , the executable code may cause one or more processors to perform the processes of method 400. For example, some common forms of machine-readable media that may include the processes of method 400 may be floppy disks, floppy disks, hard disks, tapes, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tape, any other Hole pattern physical media, RAM, PROM, EPROM, Flash EPROM, any other memory chip or cartridge and/or any other media suitable for reading by a processor or computer.

This specification and drawings illustrating inventive aspects, embodiments, implementations or applications are not to be construed as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of the specification and claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the embodiments of the disclosure. Similar numbers in two or more figures represent the same or similar elements.

In this specification, specific details are set forth to describe certain embodiments consistent with the disclosure. Many specific details are set forth in order to provide a thorough understanding of the implementations. However, it will be apparent to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not restrictive. Those skilled in the art may implement other elements that, although not specifically described herein, are within the scope and spirit of the disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be combined in other embodiments unless otherwise specifically described, or if one or more features will render an embodiment doesn't work.

This application is further described with reference to the attachment in Appendix I, entitled "Field Extraction from Forms with Unmarked Data", page 9, which is deemed to be part of the disclosure and the entire contents of which are incorporated by reference.

Although illustrative embodiments have been shown and described, a wide range of modifications, changes and substitutions are contemplated in the foregoing disclosure, and in some cases some features of the embodiments may be employed without a corresponding use of other features. Those of ordinary skill in the art will recognize many variations, substitutions and modifications. Accordingly, the scope of the invention should be limited only by the appended claims, and it is appropriate that the claims be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims (40)

1. A method for field extraction from a form with unmarked data through a field extraction model, the method comprising: receiving, via a data interface, an unmarked form that includes multiple fields and multiple field values; detecting, by a processor, a set of words and a set of positions of said set of words within said unmarked form; identifying a field value for a field from the set of words and the set of locations based at least in part on a geometric relationship between the set of words; encoding, by an encoder, a pair of a first word and a first position corresponding to the field value as a first representation; Generate, by a classifier, a field classification distribution from the first representation; Compute a first loss target by comparing the field classification distribution with the field values as pseudo labels; and The encoder is updated via backpropagation based on the first loss target. 2. The method of claim 1, wherein identifying the field value of the field includes applying a first rule: one or more words in the form of a key are related to a field name of the field. 3. The method of claim 2, wherein identifying the field value of the field includes applying a second rule: horizontally or vertically aligned pairs of words are keys to the field and the field value. 4. The method of claim 3, wherein identifying the field value of the field includes applying a third rule: a word from the set of words that matches a predefined key text is the key of the field. 5. The method of claim 1, further comprising: determining a set of phrase candidates from the set of words and a set of corresponding phrase positions from the set of positions by grouping nearby identified words; Compute a key score for each phrase candidate, which indicates the likelihood that the corresponding phrase candidate is a key for the field; and The key for the field is determined based on the maximum key score in the phrase candidate set. 6. The method of claim 5, wherein the key score is calculated based on a string distance between the corresponding phrase candidate and a predefined key. 7. The method of claim 5, further comprising: Predict the corresponding data type of each phrase candidate via the neural model; determining a subset of phrase candidates having a data type that matches a predefined data type of the field; Compute for each phrase candidate in the subset a value score indicating the likelihood that the corresponding phrase candidate is the field value of the field; and The field value is determined based on the maximum score in the subset of phrase candidates. 8. The method of claim 7, wherein the value score is calculated based on a key score corresponding to an anchor key of the field and a geometric relationship measure between a corresponding phrase candidate and the anchor key. 9. The method of claim 8, wherein the geometric relationship measure is calculated based on string distances and angles between corresponding phrase candidates and the anchor key. 10. The method of claim 1, further comprising: Generate a plurality of predicted labels respectively based on the first representation through a plurality of progressive label set (PLE) branches in parallel; and At one PLE branch, the loss component is calculated by comparing the predicted label at said one PLE branch with the predicted label from the previous PLE branch as a pseudo label, wherein a first PLE branch from the plurality of PLE branches receives the identified field value of the field as a first pseudo label. 11. A system for field extraction from a form with unmarked data via a field extraction model, the system comprising: a data interface that receives an untagged form that includes multiple fields and multiple field values; memory that stores instructions for execution by multiple processors; and A processor that executes instructions executed by said processor to perform operations including: detecting a set of words and a set of positions of said set of words within said unmarked form; identifying a field value for a field from the set of words and the set of locations based at least in part on a geometric relationship between the set of words; encoding, by an encoder, a pair of a first word and a first position corresponding to the field value as a first representation; Generate, by a classifier, a field classification distribution from the first representation; Compute a first loss target by comparing the field classification distribution with the field values as pseudo labels; and The encoder is updated via backpropagation based on the first loss target. 12. The system of claim 11, wherein identifying the field value of the field includes applying a first rule: one or more words in the form of a key are related to a field name of the field. 13. The system of claim 12, wherein identifying the field value of the field includes applying a second rule: horizontally or vertically aligned pairs of words are keys to the field and the field value. 14. The system of claim 13, wherein identifying the field value of the field includes applying a third rule: a word from the set of words that matches a predefined key text is the key of the field. 15. The system of claim 11, wherein the operations further comprise: determining a set of phrase candidates from the set of words and a set of corresponding phrase positions from the set of positions by grouping nearby identified words; Compute a key score for each phrase candidate, which indicates the likelihood that the corresponding phrase candidate is a key for the field; and The key for the field is determined based on the maximum key score in the phrase candidate set. 16. The system of claim 15, wherein the key score is calculated based on a string distance between a corresponding phrase candidate and a predefined key. 17. The system of claim 15, wherein the operations further comprise: Predict the corresponding data type of each phrase candidate via the neural model; determining a subset of phrase candidates having a data type that matches a predefined data type of the field; Compute for each phrase candidate in the subset a value score indicating the likelihood that the corresponding phrase candidate is the field value of the field; and The field value is determined based on the maximum score in the subset of phrase candidates. 18. The system of claim 17, wherein the value score is calculated based on a key score corresponding to an anchor key of the field and a measure of the geometric relationship between the corresponding phrase candidate and the anchor key. 19. The system of claim 18, wherein the geometric relationship measure is calculated based on string distances and angles between corresponding phrase candidates and the anchor key. 20. The system of claim 1, wherein the operations further comprise: Generate a plurality of predicted labels respectively based on the first representation through a plurality of progressive label set (PLE) branches in parallel; and At one PLE branch, the loss component is calculated by comparing the predicted label at said one PLE branch with the predicted label from the previous PLE branch as a pseudo label, wherein a first PLE branch from a plurality of PLE branches receives the identified field value of the field as a first pseudo label. 21. A method for field extraction from a form with unmarked data via a field extraction model, the method comprising: receiving, via a data interface, an unmarked form that includes multiple fields and multiple field values; detecting, by the processor, a first word and a first position of the first word within the unmarked form; encoding, by an encoder, the pair of the first word and the first position into a first representation; Generate multiple predicted labels respectively based on the first representation through multiple progressive label set (PLE) branches in parallel; At one PLE branch, calculating a loss component by comparing the predicted label at the one PLE branch with the predicted label from the previous PLE branch as a pseudo label; Calculate the loss target as the sum of loss components over the plurality of PLE branches; and The plurality of PLE branches are updated via backpropagation based on the loss target. 22. The method of claim 21, wherein each of the plurality of PLE branches includes a respective classifier that generates a respective predicted label based on the first representation. 23. The method of claim 21, wherein the predicted label at the one PLE branch is generated by: projecting the first representation into a set of field prediction scores via one or more fully connected layers; and The predicted labels are generated based on the maximum field prediction score in the word set. 24. The method of claim 23, further comprising: When the maximum field prediction score is greater than a predefined threshold, for a field in the plurality of fields, a word corresponding to the maximum field prediction score is selected from the word set. 25. The method of claim 21, further comprising: detecting, by a processor, a set of words and a set of positions of said set of words within said unmarked form; identifying a field value for a field from the set of words and the set of locations based at least in part on a geometric relationship between the set of words; Generate, by a classifier, a field classification distribution from the first representation; and The first loss target is calculated by comparing the field classification distribution with the field values as pseudo labels. 26. The method of claim 25, wherein a first PLE branch from the plurality of PLE branches uses the identified field value of the field as the first pseudo-label. 27. The method of claim 25, further comprising: calculating a joint loss target by adding the loss target to the first loss target; and The encoder and the plurality of PLE branches are jointly updated via backpropagation based on the joint loss target. 28. The method of claim 25, further comprising: The encoder is updated via backpropagation based on the first loss target. 29. The method of claim 28, further comprising: The plurality of PLE branches are updated via backpropagation based on the loss target while fixing the parameters of the encoder after updating the encoder. 30. A system for field extraction from a form with unmarked data via a field extraction model, the system comprising: a data interface that receives an untagged form that includes multiple fields and multiple field values; memory that stores instructions for execution by multiple processors; and A processor that executes instructions executed by said processor to perform operations including: detecting a first word and a first position of the first word within the unmarked form; encoding, by an encoder, the pair of the first word and the first position into a first representation; Generate multiple predicted labels respectively based on the first representation through multiple progressive label set (PLE) branches in parallel; At one PLE branch, calculating a loss component by comparing the predicted label at the one PLE branch with the predicted label from the previous PLE branch as a pseudo label; Calculate the loss target as the sum of loss components over multiple PLE branches; and The plurality of PLE branches are updated via backpropagation based on the loss target. 31. The system of claim 30, wherein each of the plurality of PLE branches includes a respective classifier that generates a respective predicted label based on the first representation. 32. The system of claim 30, wherein the predicted label at the one PLE branch is generated by: projecting the first representation into a set of field prediction scores via one or more fully connected layers; and The predicted labels are generated based on the maximum field prediction score in the word set. 33. The system of claim 32, wherein the operations further comprise: When the maximum field prediction score is greater than a predefined threshold, for a field in the plurality of fields, a word corresponding to the maximum field prediction score is selected from the word set. 34. The system of claim 30, wherein the operations further comprise: detecting, by a processor, a set of words and a set of positions of said set of words within said unmarked form; identifying a field value for a field from the set of words and the set of locations based at least in part on a geometric relationship between the set of words; Generate, by a classifier, a field classification distribution from the first representation; and The first loss target is calculated by comparing the field classification distribution with the field values as pseudo labels. 35. The system of claim 34, wherein a first PLE branch from the plurality of PLE branches uses the identified field value of the field as the first pseudo label. 36. The system of claim 34, wherein the operations further comprise: calculating a joint loss target by adding the loss target to the first loss target; and The encoder and the plurality of PLE branches are jointly updated via backpropagation based on the joint loss target. 37. The system of claim 36, wherein the operations further comprise: The encoder is updated via backpropagation based on the first loss target. 38. The system of claim 37, wherein the operations further comprise: The plurality of PLE branches are updated via backpropagation based on the loss target while fixing the parameters of the encoder after updating the encoder. 39. A non-transitory storage processor-readable medium storing processor-executable instructions for performing field extraction from a form with unmarked data through a field extraction model, the instructions being executed by the processor to perform the following steps: Operation: receiving, via a data interface, an unmarked form that includes multiple fields and multiple field values; detecting, by the processor, a first word and a first position of the first word within the unmarked form; encoding, by an encoder, the pair of the first word and the first position into a first representation; Generate multiple predicted labels respectively based on the first representation through multiple progressive label set (PLE) branches in parallel; At one PLE branch, calculating a loss component by comparing the predicted label at the one PLE branch with the predicted label from the previous PLE branch as a pseudo label; Calculate the loss target as the sum of loss components over multiple PLE branches; and The plurality of PLE branches are updated via backpropagation based on the loss target. 40. The non-transitory storage processor-readable medium of claim 39, wherein each of the plurality of PLE branches includes a respective classifier that generates a respective predicted label based on the first representation, and The predicted label at the one PLE branch is generated by: projecting the first representation into a set of field prediction scores via one or more fully connected layers; and The predicted labels are generated based on the maximum field prediction score in the word set.