CN116822634A - Document visual language reasoning method based on layout perception prompt - Google Patents

Abstract

The invention discloses a document visual language reasoning method based on layout-aware prompts. The method uses a large language model to perform visually information-rich document reasoning, integrates text information and visual information of document images, and introduces layout information through prompt learning. , guide large language models to understand the relationship between text and visual content in questions, and use this information to improve context learning to generate answers, allowing single-modal large language models to also handle multi-modal document visual question and answer tasks, helping large languages The model achieves ideal results in few-shot learning, and the generalization of the method is tested on three different document visual question and answer data sets.

Description

A document visual language reasoning method based on layout-aware cues

Technical Field

The present invention belongs to the field of document understanding in cross-modal understanding and prompt learning, and specifically relates to a document visual language reasoning method based on layout-aware prompts, which is used to study how the existing large language model GPT-3 solves the document visual language multimodal reasoning task under the guidance of prompt learning, and how to use contextual examples to learn the reasoning process in the case of few samples, and analyze the fragility of the robustness of the existing document pre-training model when the training samples are in the form of few samples.

Background Art

Documents are indispensable to humans as they have been used to store knowledge and information throughout history. For this reason, a lot of research work has been done on improving machine understanding of documents. The research field of document analysis and recognition aims to automatically extract information presented on paper, originally targeting human understanding. Visual Question Answering (VQA) is a multimodal deep learning approach for answering text-based questions about images. There is a set of visual question answering tasks defined based on various application scenarios, including statistical charts, daily life photos, and digital birth documents. Among these visual question answering tasks, Document Visual Question Answering (VQA), which aims to extract information from documents and answer natural language questions, is more challenging. Given an input image and an associated natural language question, the document visual question answering task aims to provide a natural language answer. In recent years, document visual question answering has become an important problem spanning computer vision, natural language understanding, and artificial intelligence. Many pre-training techniques for document analysis and recognition have been proposed and proven to be effective for various documents.

Despite the promising results achieved by these models, research in this area is limited to retrieving documents by lexical content in word spotting, being blind to semantics and neglecting the task of extracting higher-level information from these collections. On the other hand, Visual Question Answering (VQA) has been one of the main related tasks in the past few years as a link between vision and language.

In order to accurately identify key text fields, it is inevitable to exploit the cross-modal nature of visually rich documents, where textual, visual, and layout information should be jointly modeled and learned end-to-end in a single framework. Through the pre-training-fine-tuning approach, the pre-trained model absorbs cross-modal knowledge from different document types, where local invariances between these layouts and styles are preserved. However, when the model needs to be transferred to another domain with a different document format, is it possible to require the model to achieve state-of-the-art accuracy with only a few labeled samples?

In order to answer the above question about the robust estimation of model capabilities in the case of few sample document images, a document image prompting method, namely Layout-Aware Prompting, is studied. In order to design Layout-Aware Prompting, the following design criteria are adhered to: (1) Few sample training data will lead to a significant decline in the performance of pre-trained models; (2) The mapping relationship between text information and layout information reveals that large language models cannot understand the relationship between the two well; (3) The dataset used should be solvable, easy to extend, and human-readable.

In summary, the present invention proposes a document visual language reasoning method based on layout-aware prompts, which aims to solve the document visual language multimodal reasoning task, and proposes a learning method based on contextual examples for learning the reasoning process in the case of few samples. This method can improve the performance of the model in multimodal reasoning tasks, has broad application prospects, and can be applied to natural language processing, computer vision, machine learning and other fields. The prompt learning method based on few-sample document images is developed from the following three aspects: 1) A prompt learning method called Layout-Aware Prompting is developed to guide the large language model GPT-3 to perform visual language reasoning tasks; 2) Layout-Aware Prompting converts the visual information of the document image into text information, and designs the mapping relationship between text information and visual information, so that GPT-3 can better understand the information conveyed by the data, covering 3 document visual question answering datasets; 3) The visual document image understanding pre-training model is evaluated,

Summary of the invention

The purpose of the invention is to study a processing method different from the existing document pre-training model, and with the help of prompt learning, guide the large language model to solve the visual language reasoning task through context learning, and the robustness of the two in the case of few sample data.

The present invention is a document visual language reasoning method based on layout-aware prompting, named Layout-Aware Prompting, which can help large language models to reason about documents with rich visual information. Layout-Aware Prompting is a prompting that integrates the text information and visual information of a document image, guiding the large language model to understand the relationship between the text and visual information in the question, and using the prompt information to improve contextual learning to generate answers. It helps large language models to surpass existing document pre-training models in few-shot learning, and tests the generalization of the method on three different document visual question-answering datasets.

The document visual question answering datasets are DocVQA (Document Visual Question Answering), InfographicVQA (Information Image Visual Question Answering), and VisualMRC (Visual Machine Reading Comprehension), all of which are related to answering questions about visual content. However, there are some important differences between them:

(1) DocVQA: DocVQA is a typical VQA-style task, where natural language questions are defined on a single-page document and the answer needs to be generated by interpreting the document image. No list of predefined responses is given, so the problem cannot be easily treated as an n-way classification task. It focuses on answering questions about documents, such as PDF files or text scanned images. These questions are usually about the content of the document, such as "What is the author's name?" or "What is the general idea of the second paragraph?" DocVQA systems usually use text recognition and layout analysis to extract relevant information from the document and answer the questions. The original DocVQA data consists of 10,194/1,286/1,287 images, containing 39,463/5,349/5,188 questions for training/validation/testing, respectively;

(2) InfographicVQA: InfographicVQA is similar to DocVQA, but focuses on answering questions about infographics, which are visual representations of information, data, or knowledge. These questions are usually related to the content of the infographic, such as "What percentage of people like apples but not oranges?" InfographicVQA systems usually use computer vision techniques to analyze the visual elements of the infographic and extract relevant information. The original InfographicVQA data consists of 4,406/500/579 images, containing 23,946/2,801/3,288 questions for training/validation/testing, respectively;

(3) VisualMRC: VisualMRC is a more general task that involves answering questions about any type of visual content, including photos, images, and videos. These questions can be about anything visible in the visual content, such as "What color is the car in the picture?" or "How many people are there in the crowd?" VisualMRC systems typically use a combination of computer vision and natural language processing techniques to analyze visual content and answer questions. The original VisualMRC data consists of 9,574/956/2,237 images, containing 21,015/2,839/6,708 questions for training/validation/testing, respectively;

In summary, DocVQA and InfographicVQA are more specific types of VisualMRC that focus on answering questions about documents and information graphics, respectively, while VisualMRC is a more general task that can be applied to any type of visual content.

The visual-linguistic reasoning task Document Visual Question Answering focuses on a specific type of visual question answering task where visual understanding of the information on the document image is necessary to provide an answer. This is not just about delivering document images through optical character recognition (OCR), but also about understanding all types of information conveyed by the document. Document visual questions refer to the visual elements and design issues involved in the document design and typesetting process. These issues may include the following aspects:

(1) Layout and typesetting: This includes how to organize the content in the document, choose fonts and font sizes, arrange paragraphs and headings, etc. Good layout and typesetting can help readers understand and absorb the content of the document more easily.

(2) Images and charts: Images and charts in a document should be used to support the text content and to clearly convey information. Designers need to consider how to choose the most appropriate types of images and charts and how to integrate them into the overall design of the document.

(3) Color and font: Color and font can affect the readability and visibility of a document. Designers need to choose colors and fonts that are appropriate for the subject of the document and the target audience to ensure that the document is easy to read and understand.

(4) Whitespace and Spacing: Appropriate whitespace and spacing can help the content of a document stand out better and make the document easier to read. Designers need to consider how to balance text and whitespace, and how to use spacing to help readers distinguish different text paragraphs and sections.

(5) Titles and Chapters: A good title and chapter structure can make it easier for readers to find the information they need and help them better understand the organization of the document. Designers need to consider how to choose the best title and chapter structure and ensure that these elements are coordinated with the overall design of the document.

In short, document visual issues are various visual elements and design issues that need to be considered in the document design and typesetting process. By carefully considering these issues, designers can create high-quality documents that are easy to read and understand.

The context learning scenario of the large language model, such as GPT-3, can be regarded as a conditional text generation problem. Specifically, the probability of generating the target text y depends on the text field related to the target text in the input extraction data, including the context C of k examples and the previous text x to be predicted. Therefore, the predicted target text y corresponding to the previous text x to be predicted can be expressed as:

Where LM represents the parameters of the language model, the tth token y _<t is the token predicted before the current token y _t (t＝1,2,...,T), and a string can be divided into T (C,x,y _t ) formats, C＝{x ₁ ,y ₁ ,x ₂ ,y ₂ ,...,x _k ,y _k } is the context string, where x _i ,y _i (i＝1,2,...,k) are the previous and next texts in the ith context string that are similar to x and y, respectively. In GPT-3, C is created by concatenating k training instances and their corresponding texts.

The prompt refers to the use of specific text or language prompts to guide the model to generate specific outputs. In the present invention, the use of specific text or language prompts is called "examples" as the input of the large language model, and the method of selecting examples is to use the all-mpnet-base-v2 model in the sentence-transformers algorithm, which maps sentences and paragraphs to a 768-dimensional dense vector space, and calculates the retrieval sample corresponding to the retrieval sample question with the highest semantic cosine similarity to the test sample question as the example sample in the prompt design:

A and B are the sentences to be calculated, corresponding to the test sample problem and the retrieval sample problem. cosine_sim(A,B) refers to the semantic cosine similarity between sentence A and sentence B. dot(A,B) represents the dot product of sentence vectors A and B. ||A|| and ||B|| respectively represent the Euclidean distance (size) between vectors A and B. In subsequent experiments, the effect difference between selecting examples based on semantic similarity and randomly selecting examples was compared.

The document pre-training models include text pre-training models: BERT and RoBEATA; text and layout model LiLT; text, layout and image model pre-training models: LayoutLM, LayoutLMv2, LayoutLMv3, ERNIELayout. Benchmark experiments evaluate the robustness of pre-trained visual document understanding models fine-tuned for downstream tasks from full samples to few samples.

The present invention proposes a document visual language reasoning method based on layout-aware cues. The present invention is used to study how the existing large language model GPT-3 solves the document visual language multimodal reasoning task (step 4) under the guidance of cue learning, and how to use contextual examples to learn the reasoning process in the case of few samples, and analyzes the fragility of the robustness of the existing document pre-training model when the training samples are in the case of few samples. It includes the following steps:

Step 1: Data preprocessing. Three data sets are selected for experiments, including DocVQA, InfographicVQA and VisualMRC. Through the optical character recognition (OCR) algorithm, for any of the three data sets, the document images in the data set are first preprocessed, such as denoising, binarization, rotation correction, tilt correction, etc., to improve the accuracy and efficiency of subsequent processing. Then, the feature information of the characters, such as contours, edges, projections, etc., are extracted from the document images for character recognition. Character recognition is to match the feature information with the trained optical character recognition (OCR) model to determine the characters in the document image. Finally, the recognition results are output as text formats that can be edited and processed by computers, such as TXT, JSON, etc. The above operations are performed on all three data sets, thereby extracting the text information in the data of the three data sets, and the layout information and the information about questions, question numbers, answers, etc. in the data sets are sorted into data texts in the corresponding JSON format files, and the data texts are divided into retrieval data sets and test data sets according to a preset ratio;

Step 2: Select sample samples. The sample samples are used to help the large language model understand the data format and question-answer format required for the task. Through the data text of the JSON format file obtained in step 1, the questions of all retrieval samples in the retrieval dataset are grouped into set A, and the questions of any test sample in the test dataset are extracted. Then, the all-mpnet-base-v2 model in the sentence-transformers algorithm is used to search in set A, and the semantic cosine similarity between the question of the test sample and each question in set A is calculated respectively, so as to retrieve the question of the retrieval sample with the highest semantic similarity to the question of the test sample. The retrieval sample corresponding to the question of the retrieval sample is used as the sample sample in the prompt design for the prompt design of step 3, wherein the all-mpnet-base-v2 model maps sentences and paragraphs to a 768-dimensional dense vector space;

Step 3: Design prompts. Prompts refer to the use of specific text or language prompts to guide the model to generate specific outputs. There are three types of prompts: pure text prompts, text and layout discrete prompts, and layout-aware prompts. (1) Pure text prompts, that is, only the text information of the document data without layout information; (2) Text and layout discrete prompts are to add text and layout to the prompts respectively and design a prompt header to inform the model of the format of text data and layout data, as well as a simple correspondence between the two; (3) Layout-aware prompts are to organize the data of the example samples and test samples obtained in step 2 into the prompts, and design the prompts as a data stream format of (prompt header, context samples, test samples), where the role of the prompting head is to inform GPT-3 of context samples (in-context demonstration) and test samples (testing The data form is {text:boxes},where each boxes is defined by four coordinates:[[x1,y1,x2,y2]].The x1 and y1 refer to the horizontal and vertical coordinates of the upper-left corner of the OCR boxes,and the x2 and y2 refer to the horizontal and vertical coordinates ofthe lower-right corner of the OCR boxes which indicate the position of the OCR boxes within the document,please answer the questions according to the data form. answer the question according to the above data form."). The text information and layout information extracted in step 1 are designed into a mapping relationship {text: corresponding OCR box}, that is, {text: boxes}, as the data format of context samples and test samples. This format is called "layout-aware prompts". The context samples are the sample data selected in step 2 based on the semantics closest to the test sample question to guide the large language model, such as GPT-3, to understand that the data style to be processed is the question-answering reasoning task data style. The context samples include context layout-aware prompts, context sample questions, and context sample answers; the test samples include test sample layout-aware prompts and test sample questions. Finally, the test results of the three prompt methods, namely, pure text prompts, text and layout discrete prompts, and layout-aware prompts, are compared through experiments;

Step 4: Pass the designed prompt to GPT-3 for the few-shot document visual question answering reasoning task, and use the Average Normalized Levenshtein distance criterion to evaluate the accuracy of the generated answer:

Where lev _a,b (i,j) represents the Levenshtein distance between the first i characters of string a and the first j characters of string b; 1 _(ai≠bj) is an indicator function, which is 0 when a _i = b _j and 1 otherwise, a _i represents the i-th character of string a and b _j represents the j-th character of string b. The first formula in the min operation lev _a,b (i-1,j)+1 represents deleting characters from string a to reach b; the second formula lev _a,b (i,j-1)+1 represents inserting characters from string a to reach b; the third formula lev _a,b (i-1,j-1)+1 _(ai≠bj) represents replacing characters from string a to reach b (depending on whether the current characters are the same). In linguistics, the Levenshtein distance is used as a measure to quantify text distance, that is, the difference between two texts. It is related to mutual intelligibility: the higher the text distance, the lower the mutual intelligibility, and the lower the text distance, the higher the mutual intelligibility. In subsequent experiments, we compared the random selection of examples with different semantically similar cases; min and max represent the minimum and maximum value operations, respectively;

Step 5: Study the robustness of different models in the case of few-shot. Different document pre-training models are used, including text pre-training models: BERT and RoBEATA; text and layout model LiLT; text, layout and image model pre-training models: LayoutLM, LayoutLMv2, LayoutLMv3, ERNIELayout. The parameters set for the three different document visual question answering datasets are the same: the learning rate is set to 2×e ^-5 , the training epoch is set to 40, and the resolution of all input images is 224x 224 pixels. In full-sample training, the batch in training is set to 4, and the batch in testing is set to 1. In few-shot training, the batch in training is set to 1, and the batch in testing is set to 1. The reasoning effects of these models in the case of few-shot are compared. In addition, the different numbers of example samples in the case of few-shot and the different numbers of questions in different example samples are explored.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

(1) Previous multimodal models used to solve document visual question answering tasks all adopted the pre-training-fine-tuning paradigm. However, this approach is very time-consuming and has very high requirements for machine configuration. In order to solve these problems, we use prompt learning to guide large language models to perform reasoning tasks. This method is simple and generally takes 3-6 hours to complete the evaluation. The pre-training-fine-tuning paradigm often requires multiple A100 GPUs to complete the evaluation within several hours. In addition, the reasoning accuracy of specific tasks is higher than that of the best pre-training-fine-tuning paradigm model.

(2) By introducing layout information through prompt learning, the large language model is guided to understand the relationship between the text and visual content in the question, and use this information to improve context learning to generate answers, so that the single-modal large language model can also handle multi-modal document visual question answering tasks with higher accuracy than ordinary answering methods;

(3) We evaluate and compare the multimodal model with Layout-Aware Prompting in a few-shot scenario on different document visual question answering datasets. We hope that the proposed prompt learning-guided large language model approach, empirical research, and in-depth analysis will be beneficial to future research to improve the accuracy of models in answering document visual reasoning tasks.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a sample diagram of a document image;

FIG2 is a flow chart of an implementation method of the present invention;

FIG3 is a schematic diagram of a design prompt learning method of the present invention;

FIG. 4 is a schematic diagram of an exemplary selection method of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in combination with the implementation mode and the accompanying drawings, so that those skilled in the relevant art can better understand the present invention. It should be noted that the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and are not intended to limit the scope of the invention claimed for protection. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Considering that the existing fine-tuned document pre-training models to solve the document visual question answering task are all solved in the pre-training-fine-tuning paradigm, but this approach is very time-consuming and has very high requirements for machine configuration, and this type of method is relatively poor when processing few-shot data. The present invention proposes a document visual language reasoning method based on layout-aware cues, which is used to guide the large language model to use powerful contextual learning capabilities to understand the information to be conveyed by the data, and is used to solve the document visual language reasoning task. Guide the large language model to understand the relationship between the text and the visual content in the question, and use this information to improve contextual learning to generate answers. Help large language models achieve ideal results in few-shot learning, and test the generalization of the method on three different document visual question answering datasets.

The document visual question answering datasets are DocVQA (Document Visual Question Answering), InfographicVQA (Infographic Visual Question Answering), and VisualMRC (Visual Machine Reading Comprehension). All three datasets are related to answering questions about visual content. However, there are some important differences between them:

(1) DocVQA: DocVQA is a typical VQA-style task, where natural language questions are defined on a single-page document and the answer needs to be generated by interpreting the document image. No list of predefined responses is given, so the problem cannot be easily treated as an n-way classification task. It focuses on answering questions about documents, such as PDF files or text scanned images. These questions are usually about the content of the document, such as "What is the author's name?" or "What is the general idea of the second paragraph?" DocVQA systems usually use text recognition and layout analysis to extract relevant information from the document and answer the questions. The original DocVQA data consists of 10,194/1,286/1,287 images, containing 39,463/5,349/5,188 questions for training/validation/testing, respectively;

(2) InfographicVQA: InfographicVQA is similar to DocVQA, but focuses on answering questions about infographics, which are visual representations of information, data, or knowledge. These questions are usually related to the content of the infographic, such as "What percentage of people like apples but not oranges?" InfographicVQA systems usually use computer vision techniques to analyze the visual elements of the infographic and extract relevant information. The original InfographicVQA data consists of 4,406/500/579 images, containing 23,946/2,801/3,288 questions for training/validation/testing, respectively;

(3) VisualMRC: VisualMRC is a more general task that involves answering questions about any type of visual content, including photos, images, and videos. These questions can be about anything visible in the visual content, such as "What color is the car in the picture?" or "How many people are there in the crowd?" VisualMRC systems typically use a combination of computer vision and natural language processing techniques to analyze visual content and answer questions. The original VisualMRC data consists of 9,574/956/2,237 images, containing 21,015/2,839/6,708 questions for training/validation/testing, respectively;

In summary, DocVQA and InfographicVQA are more specific types of VisualMRC that focus on answering questions about documents and information graphics, respectively, while VisualMRC is a more general task that can be applied to any type of visual content.

The visual-linguistic reasoning task Document Visual Question Answering (DocVQA) focuses on a specific type of visual question answering task, where visual understanding of the information on the document image is necessary to provide the answer. This is not just about passing the document image through OCR, but also about understanding all types of information conveyed by the document. Text content (handwritten or typed), non-text elements (marks, ticks, separators, charts), layout (page structure, forms, tables), and style (fonts, colors, highlights), etc., are all information that may be needed to answer the question at hand. Document visual questions refer to the visual elements and design issues involved in the document design and typesetting process. These issues may include the following aspects:

(1) Layout and typesetting: This includes how to organize the content in the document, choose fonts and font sizes, arrange paragraphs and headings, etc. Good layout and typesetting can help readers understand and absorb the content of the document more easily.

(2) Images and charts: Images and charts in a document should be used to support the text content and to clearly convey information. Designers need to consider how to choose the most appropriate types of images and charts and how to integrate them into the overall design of the document.

(3) Color and font: Color and font can affect the readability and visibility of a document. Designers need to choose colors and fonts that are appropriate for the subject of the document and the target audience to ensure that the document is easy to read and understand.

(4) Whitespace and Spacing: Appropriate whitespace and spacing can help the content of a document stand out better and make the document easier to read. Designers need to consider how to balance text and whitespace, and how to use spacing to help readers distinguish different text paragraphs and sections.

(5) Titles and Chapters: A good title and chapter structure can make it easier for readers to find the information they need and help them better understand the organization of the document. Designers need to consider how to choose the best title and chapter structure and ensure that these elements are coordinated with the overall design of the document.

In short, document visual issues are various visual elements and design issues that need to be considered in the document design and typesetting process. By carefully considering these issues, designers can create high-quality documents that are easy to read and understand.

The context learning scenario of the large language model, such as GPT3, can be regarded as a conditional text generation problem. Specifically, the probability of generating the target text y depends on the text field related to the target text in the input extraction data, including the context C of k examples and the previous text x to be predicted. Therefore, the predicted target text y corresponding to the previous text x to be predicted can be expressed as:

Where LM represents the parameters of the language model, the tth token is the token predicted before the current token to be predicted (t＝1,2,...,T), a string can be divided into T (C,x,y) formats, C＝{x ₁ ,y ₁ ,x ₂ ,y ₂ ,...,x _k ,y _k } is the context string, where x _i ,y _i (i＝1,2,...,k) are the previous and next texts in the ith context string with text formats similar to x and y, respectively. In GPT-3, C is created by concatenating k training instances and their corresponding texts.

The prompt refers to the use of specific text or language prompts to guide the model to generate specific outputs. In the present invention, the use of specific text or language prompts is called "examples" as the input of the large language model, and the method of selecting examples is to use the all-mpnet-base-v2 model in the sentence-transformers algorithm, which maps sentences and paragraphs to a 768-dimensional dense vector space, and calculates the retrieval sample corresponding to the retrieval sample question with the highest semantic cosine similarity with the test sample question as the example sample in the prompt design:

A and B are the sentences to be calculated, corresponding to the test sample problem and the retrieval sample problem. cosine_sim(A,B) refers to the semantic cosine similarity between sentence A and sentence B. dot(A,B) represents the dot product of sentence vectors A and B. ||A|| and ||B|| respectively represent the Euclidean distance (size) between vectors A and B. The subsequent ablation experiment compared the effect of selecting examples based on semantic similarity and randomly selecting examples.

The document pre-training models include text pre-training models: BERT and RoBEATA; text and layout model LiLT; text, layout and image model pre-training models: LayoutLM, LayoutLMv2, LayoutLMv3, ERNIELayout. Benchmark experiments evaluate the robustness of pre-trained visual document understanding models fine-tuned for downstream tasks from full samples to few samples.

Step 1: Data preprocessing. Three data sets are selected for experiments, including DocVQA, InfographicVQA and VisualMRC. Through the optical character recognition (OCR) algorithm, the document image of any one of the three data sets is first preprocessed, such as denoising, binarization, rotation correction, tilt correction, etc., to improve the accuracy and efficiency of subsequent processing. Then, the feature information of the characters, such as contours, edges, projections, etc., are extracted from the document image for character recognition. Character recognition is to match the feature information with the trained optical character recognition (OCR) model to determine the characters in the image. Finally, the recognition results are output as text formats that can be edited and processed by computers, such as TXT, JSON, etc. Through the above operations, the text information in the data of the three data sets is extracted, and the layout information and the information about questions, question numbers, answers, etc. in the data sets are sorted into the data text of the corresponding JSON format files, and the data text obtained from each data set is divided into a retrieval data set and a test data set;

Step 2: Select sample samples (examples), which are used to help the large language model understand the data format and question-answer format required for the task. Through the data text of the JSON format file obtained in step 1, the questions of all retrieval samples in the retrieval dataset are grouped into set A, and the questions of any test sample in the test dataset are extracted. Then, the all-mpnet-base-v2 model in the sentence-transformers algorithm is used to search in set A, and the semantic cosine similarity between the question of the test sample and each question in set A is calculated respectively, so as to retrieve the question of the retrieval sample with the highest semantic similarity to the question of the test sample. The retrieval sample corresponding to the question of the retrieval sample is used as the sample sample in the prompt design for the prompt design of step 3, wherein the all-mpnet-base-v2 model maps sentences and paragraphs to a 768-dimensional dense vector space. Different example selection methods also affect the effect of Layout-Aware Prompting. Table 1 shows that in DocVQA, using semantically similar examples to select examples is the best compared to randomly selecting examples. ANLS↑ represents the average normalized Levenshtein distance. The higher the text distance, the lower the mutual intelligibility. The lower the text distance, the higher the mutual intelligibility. Therefore, the higher the ANLS value, the better.

Table 1 Test results of different selection methods

Step 3: Design prompts. Prompts refer to the use of specific text or language prompts to guide the model to generate specific outputs. There are three types of prompts: plain text prompts, text and layout discrete prompts, and layout-aware prompts. (1) Plain text prompts, that is, only the text information of the document data is added without adding layout information; (2) Text and layout discrete prompts are to add text and layout to the prompts separately and design a prompt header to inform the model of the format of text data and layout data, as well as a simple correspondence between the two; (3) Layout-aware prompts are to organize the data of the example samples and test samples obtained in step 2 into the prompts, and design the prompts in the data stream format of (prompt header, previous examples, test examples), where the role of the prompting head is to inform GPT-3 of the previous examples (in-context demonstration) and test samples (testing The data form is {text:boxes},where each box is defined by four coordinates:[[x1,y1,x2,y2]].The x1 and y1 refer to the horizontal and vertical coordinates of the upper-left corner of the OCR boxes,and the x2 and y2 refer to the horizontal and vertical coordinates of the lower-right cornerof the OCR boxes which indicate the position of the OCR boxes within the document,please answer the question according to the above data form."). Then, the text information and layout information extracted in step 1 are designed into a mapping relationship {text: corresponding OCR box}, that is, {text: boxes}, as the data format of context samples and test samples. This format is called "layout-aware prompts". The context samples are the sample data selected in step 2 based on the semantics closest to the test sample questions to guide GPT-3 to understand the question-answering reasoning task data style to be processed. The context samples include context layout-aware prompts, context sample questions, and context sample answers; the test samples include test sample layout-aware prompts and test sample questions. Table 2 shows the different effects of whether or not to design the text information and layout information into a mapping relationship, proving that designing the text information and layout information into a mapping relationship is better for guiding GPT-3; Table 2 shows the effect of prompting with or without layout information, as well as the average accuracy of the model in different data sets and different sample situations when the text information and layout information are designed into a mapping relationship and when they are not designed into a mapping relationship. The results show that the effect of no layout information is relatively poor and the effect of designing the text information and layout information into a mapping relationship is better for guiding GPT-3. Specifically, compared with the case where no mapping relationship is designed, designing a mapping relationship can significantly improve the accuracy of the model in reasoning and answering questions. This shows that by designing a mapping relationship between text information and layout information, the layout information can be better used to guide the model to make predictions, thereby improving the accuracy of the model in reasoning and answering questions.

Table 2 shows the test results of different formats of the sample

Note: w/o boxes means no location information, w/boxes means location information, split means the text information and its location information are discrete, mapping means the text information and its location information are mapped, ANLS↑ means the average normalized Levenshtein distance, the higher the value, the better the effect

Step 4: Pass the designed prompt to GPT-3 for the few-shot document visual question answering reasoning task, and use the Average Normalized Levenshtein distance criterion to evaluate the accuracy of the generated answer:

Where lev _a,b (i,j) represents the Levenshtein distance between the first i characters of string a and the first j characters of string b; is an indicator function. When a _i = b _j , its value is 0, and it is equal to 1 otherwise. a _i represents the i-th character of string a, and b _j represents the j-th character of string b. The first formula in the min operation lev _a,b (i-1,j)+1 represents deleting characters from string a to reach b; the second formula j _a,b (i,j-1)+1 represents inserting characters from string a to reach b; the third formula Represents replacing characters from string a to reach b (depending on whether the current characters are the same). min and max represent the minimum and maximum operations, respectively. In linguistics, the Levenshtein distance is used as a measure to quantify text distance, that is, the difference between two texts. It is related to mutual comprehensibility: the higher the text distance, the lower the mutual comprehensibility, and the lower the text distance, the higher the mutual comprehensibility. Different numbers of samples and the number of questions in the sample will affect the final effect, but the more the better. Table 3 shows the average accuracy of the method of the present invention under different numbers of samples and the number of questions in the sample. The results show that in the DocVQA dataset, the best case is 1 text content and 4 questions in 1 sample (1-shot). This may be because the questions in the example sample are semantically similar to those in the test sample, indicating that step 2 plays a relatively important role in designing contextual learning examples. Because step 2 is designed to closely link the context information of the example sample with the question, so that the model can better reason and generalize. Therefore, these results show that in few-shot learning, properly designing the contextual information of example samples can significantly improve the model's ability to understand the examples and perform reasoning, thereby improving the accuracy of answering questions;

Table 3 Test results of different sample numbers and number of questions in samples on the document visual question answering dataset

Step 5: Study the robustness of different models in the case of few-shot. Different document pre-training models are used, including text pre-training models: BERT and RoBEATA; text and layout model LiLT; text, layout and image model pre-training models: LayoutLMv1, LayoutLMv2, LayoutLMv3, ERNIELayout. Compare the reasoning effects of these models in the case of few-shot. For the document visual question answering task, the parameters set for the three different document visual question answering datasets are the same: the learning rate is set to 2×e ^-5 , the training epoch is set to 40, and the resolution of all input images is 224x 224 pixels. In full-shot training, the batch size in training is set to 4, and the batch size in testing is set to 1. In few-shot training, the batch size in training is set to 1, and the batch size in testing is set to 1. Table 4 shows the reasoning effects of different models in the case of full-shot and few-shot. The results show that the effects of existing document pre-training models vary greatly from full-shot to few-shot, indicating that they have poor generalization in the case of few-shot. In contrast, Layout-aware Prompting (ours in Table 4) performs better in the case of few samples, but there is still a certain gap from the performance of existing document pre-training models in the case of full samples. This may be because the visual perception effect of the unimodal large language model is not as good as the effect of the visual module set in the existing document pre-training model. Therefore, these results show that the existing document pre-training model may have certain limitations when processing few-sample data, and Layout-aware Prompting can be an effective solution, but further improvements are still needed to improve its effect.

Table 4 Benchmark experiment results

Note: ours represents the method of the present invention, Full-sample represents the full sample situation, Few-shot represents the few-shot situation, ANLS↑ represents the average normalized Levenshtein distance, the higher the value, the better the effect

The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes; all the disclosed features, or all the steps in the methods or processes, except for mutually exclusive features and/or steps, can be combined in any way.

Claims (7)

1. A document visual language reasoning method based on layout perception prompt is characterized by comprising the following steps:

step 1: preprocessing data;

acquiring a data set, performing preprocessing operation on a document image in the data set through an optical character recognition algorithm to improve the accuracy and efficiency of subsequent processing, extracting character characteristic information from the document image for character recognition, wherein the character recognition is to match the characteristic information with a trained optical character recognition model to determine characters in the document image, and finally outputting a recognition result into a text format which can be edited and processed by a computer, so as to extract text information of data in the data set, layout information and information about questions, question numbers and answers in the data set, and sort the information into data texts of corresponding JSON format files, and dividing the data texts into a retrieval data set and a test data set according to a preset proportion;

step 2: selecting an example sample, wherein the example sample is used for helping a large language model to understand a data format and a question-answer form required by a task;

the method comprises the steps of forming a set A of problems of all search samples in a search data set, extracting problems of any one test sample in the test data set, searching in the set A through an all-mpnet-base-v2 model in a content-converters algorithm, and respectively calculating semantic cosine similarity of the problems of the test sample and each problem in the set A so as to search out the problem of the search sample with the highest semantic similarity to the problems of the test sample, wherein the search sample corresponding to the problem of the search sample is used as an example sample in a prompt design, and the all-mpnet-base-v2 model maps sentences and paragraphs to a 768-dimensional dense vector space;

step 3: designing a layout perception prompt;

the prompt means that a specific text or language prompt is used for guiding a large language model to generate specific output, the data of the example sample and the test sample obtained in the step 2 are arranged in the prompt, the prompt is designed into a data flow format of a prompt head, a context sample and a test sample, and the text information and the layout information extracted in the step 1 are designed into a mapping relation { text: corresponding OCR box }, and are used as the data formats of the context sample and the test sample;

step 4: and transmitting the designed prompt to a large language model to perform a small sample document visual question-answer reasoning task, and generating an answer.

2. The method for visual language reasoning of a document based on layout-aware cues as defined in claim 1, wherein the preprocessing operation comprises denoising, binarization, rotation correction, tilt correction.

3. The method for visual language reasoning of a document based on layout-aware cues as claimed in claim 2, wherein the characteristic information of the character comprises outline, edge, projection.

4. A method of visual language reasoning for documents based on layout aware cues as claimed in claim 3, characterised in that the text formats that the computer can edit and process include TXT, JSON.

5. The method for visual language reasoning of a document based on layout awareness cues as defined in claim 4, in which the context sample comprises a context layout awareness cue, a context sample question, and a context sample answer; the test sample comprises a test sample layout perception prompt and a test sample problem.

6. The method for visual language reasoning of a document based on layout-aware cues as defined in claim 5, in which the large language model selects GPT-3.

7. The method for visual language reasoning about documents based on layout aware cues as defined in claim 6, wherein the dataset is DocVQA, infographicVQA or visual mrc.