CN116862692A - Intelligent reimbursement method and system based on visual question and answer - Google Patents

Abstract

An intelligent reimbursement method and system based on visual question and answer, wherein the method comprises the following steps: acquiring identity information of a reimbursement person, and constructing a reimbursement person map based on the identity information; acquiring or acquiring a ticket image to be reimbursed, identifying the ticket image to be reimbursed based on a visual question-answering mechanism to obtain ticket information, and constructing a reimbursed ticket map based on the ticket information; acquiring a second problem set, and fusing the reimbursement person pattern and the bill pattern based on the second problem set to generate a reimbursement single pattern; and automatically generating a reimbursement bill according to the reimbursement bill map.

Description

Intelligent reimbursement method and system based on visual question and answer

Technical Field

The application relates to the technical field of enterprise reimbursement, in particular to an intelligent reimbursement method and system based on visual question and answer.

Background

The business trip reimbursement is a financial settlement behavior frequently occurring between staff and work units in modern enterprises and institutions, a large amount of receipts and data information can be generated in the implementation process, and great workload is added to reimbursement staff and financial auditing staff. In the process of reimbursement in business trip, reimbursement staff firstly arranges bills generated in the process of business trip, then fills reimbursement bills according to bill information, when the bills are relatively less, the reimbursement bills are relatively easy to fill, and the probability of mistakes in the process of filling reimbursement bills is gradually increased along with the increase of the number of the bills; when the bill and reimbursement bill are sent to the financial auditing personnel, if the reimbursement bill and bill are inconsistent, the financial auditing personnel need to audit the reimbursement bill and bill one by one to find out where the problem is, and the process greatly increases the workload of the financial auditing personnel.

Related patents such as intelligent account reigning machines are currently available, which mainly extract bill amount information by combining an industrial camera with an OCR technology, and then check the bill amount information with reimbursement bill information manually filled by a reimbursement staff to determine whether the reimbursement requirements are met. Although the above reimbursement process has advanced significantly over traditional purely manual reimbursement, it is still not intelligent enough and requires extensive intervention by reimbursement personnel and financial auditors.

Disclosure of Invention

In view of the above analysis, the present application aims to provide an intelligent reimbursement method based on question and answer, which is used for solving the problems that the existing reimbursement method is easy to make mistakes and needs a large amount of manual intervention.

In one aspect, the embodiment of the application provides an intelligent reimbursement method based on visual question and answer, which comprises the following steps:

acquiring identity information of a reimbursement person, and constructing a reimbursement person map based on the identity information;

acquiring a ticket image to be reimbursed, identifying the ticket image to be reimbursed based on a visual question-answering mechanism to obtain ticket information, and constructing a ticket reimbursement map based on the ticket information;

acquiring a second problem set, and fusing the reimbursement person pattern and the bill pattern based on the second problem set to generate a reimbursement single pattern; and automatically generating a reimbursement bill according to the reimbursement bill map.

Based on further improvement of the method, the obtaining the identity information of the reimbursement person, and constructing a reimbursement person map based on the identity information, comprises the following steps:

acquiring face data or fingerprint data of a sales person or acquiring identity data input by the sales person, and verifying the identity of the user based on the face data, the fingerprint data or the identity data;

extracting reimbursement person identity information from an enterprise information base if the user identity rule is met, wherein the identity information comprises job level information of the reimbursement person and reimbursement standards corresponding to the job level information; and constructing a reimbursement person map by taking the name of the reimbursement person, the position in the position information and each reimbursement type and the subsidy in the reimbursement standard as entity nodes and taking the reimbursement type and the corresponding amount or condition of the subsidy as attribute nodes.

Further, identifying the to-be-reimbursed bill image based on a visual question-answering mechanism to obtain bill information, and constructing a reimbursed bill map based on the bill information, wherein the method comprises the following steps:

acquiring a first problem set, performing word segmentation and filtering on each problem text in the first problem set, converting the problem text into vector representation based on the filtered word segmentation, and extracting text feature vectors of the problem text;

extracting image features, regional features and text features of each bill image, and fusing the image features, regional features and text features to obtain fusion features;

combining the text feature vector and the fusion feature of each question, and inputting the combined text feature vector and the fusion feature into a trained visual question-answering model to obtain an answer of each question in the first question set;

and constructing a current reimbursement bill map by taking the reimbursement bill as a central entity node and taking the answer as an entity node or an attribute node.

Further, extracting image features, region features and text features of each bill image, and fusing the image features, the region features and the text features to obtain fusion features, wherein the method comprises the following steps:

extracting image features by adopting a convolutional neural network;

detecting a text region in an image, and extracting position information of the text region as a region feature;

recognizing text information in a text region by adopting an OCR technology, converting the text information into vector representation, and extracting text characteristics of the text information;

and fusing the image features, the region features and the text features by adopting multi-modal feature fusion to obtain fusion features.

Further, fusing the reimbursement person spectrum and the bill spectrum based on the second problem set to generate a reimbursement single spectrum, including:

the method comprises the steps of establishing a reimbursement single entity node at the time, extracting reimbursement person entity nodes in a reimbursement person map, and establishing reimbursement relation between the reimbursement person entity nodes and the reimbursement single entity node at the time to form an initial reimbursement single map;

aiming at each problem in the second problem set, carrying out problem identification by adopting natural language processing to obtain reimbursement types and inquiry types corresponding to the problems, searching whether bill types corresponding to the reimbursement types exist in the bill map, and if so, judging whether the bill types exist: adding the bill type as entity node into a reimbursement single map, and establishing the inclusion relation between the reimbursement single entity node and the bill type node; searching whether a corresponding reimbursement type node exists in the reimbursement person map, if not, searching an answer corresponding to the reimbursement type in the bill map according to the reimbursement type, and establishing a corresponding relation between the bill type node and the answer attribute node by taking the answer as the attribute node; otherwise, using the sum or condition attribute node corresponding to the reimbursement type node in the reimbursement person map as a constraint condition, searching an answer corresponding to the inquiry type in the bill map according to the inquiry type, processing the answer by adopting the constraint condition to obtain a final answer, and establishing a corresponding relation between the bill type node and the answer attribute node by taking the final answer as the attribute node;

and obtaining the reimbursement single map after processing all the problems.

Further, after automatically generating the reimbursement sheet according to the reimbursement sheet map, the method further comprises:

taking reimbursement person nodes in the reimbursement single map as alignment nodes, and integrating the reimbursement single map into the reimbursement information knowledge map of the enterprise;

and acquiring a third problem input by the user, analyzing the problem to generate a query statement, querying a result corresponding to the query statement in the enterprise reimbursement information knowledge graph, and feeding back the result to the user.

Further, obtaining a problem related to the reimbursement business input by a user, analyzing the problem to generate a query statement, and querying a result corresponding to the query statement in the enterprise reimbursement information knowledge graph, wherein the method comprises the following steps:

the related problems of the reimbursement service are identified by named entities, and a word vector model is adopted to convert the text of the problems into sentence vectors;

performing problem classification on the sentence vectors by adopting a trained neural network model, and constructing a query sentence based on a named entity and the problem classification;

and inquiring a result corresponding to the inquiry statement in the enterprise reimbursement information knowledge graph.

Further, after the bill image to be reimbursed at this time is obtained, preprocessing is further included on the bill image.

Further, after identifying the to-be-reimbursed bill image based on the visual question-answering mechanism to obtain bill information, and before constructing the reimbursed bill map based on the bill information, performing bill auditing based on the identity information and the bill information of the reimburser, wherein the bill auditing comprises: ticket validity auditing, ticket repeatability auditing and ticket attribution authority auditing.

Compared with the prior art, the method can automatically identify the information of the reimbursement bill to be reimbursed to construct a reimbursement bill map by adopting an identification method based on a visual question-answer mechanism; and establishing a reimbursement person map according to the user identity information, automatically calculating to obtain data that the reimbursement person can reimburse actually at the time by fusing the reimbursement person map and the bill map according to the second problem set, and generating a reimbursement single map at the time, so that the reimbursement single map at the time is automatically obtained, manual operation of the reimbursement person is not needed, and the method is convenient and quick. Meanwhile, the information of the reimbursement person, the bill information and the reimbursement bill information are stored in the form of a knowledge graph, so that a user can conveniently check corresponding information, financial auditing is convenient, and the association relationship can be obtained rapidly through a knowledge base.

On the other hand, the embodiment of the application provides an intelligent reimbursement system based on visual question and answer, which comprises the following modules:

the reimbursement person map construction module is used for acquiring reimbursement person identity information and constructing reimbursement person maps based on the identity information;

the bill map construction module is used for acquiring the bill image to be reimbursed, identifying the bill image to be reimbursed based on a visual question-answer mechanism to obtain bill information, and constructing a reimbursed bill map based on the bill information;

the reimbursement sheet generation module is used for acquiring a second problem set, and fusing the reimbursement person spectrum and the bill spectrum based on the second problem set to generate a current reimbursement sheet spectrum; and automatically generating a reimbursement bill according to the reimbursement bill map.

In the application, the technical schemes can be mutually combined to realize more preferable combination schemes. Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and drawings.

Drawings

The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the application, like reference numerals being used to refer to like parts throughout the several views.

FIG. 1 is a flow chart of an intelligent reimbursement method based on visual questions and answers in an embodiment of the application;

FIG. 2 is a block diagram of an intelligent reimbursement system based on visual questions and answers in accordance with an embodiment of the present application;

FIG. 3 is a schematic illustration of a reimbursement human map in an embodiment of the present application;

fig. 4 is a schematic diagram of the present reimbursement ticket according to an embodiment of the present application.

Detailed Description

The following detailed description of preferred embodiments of the application is made in connection with the accompanying drawings, which form a part hereof, and together with the description of the embodiments of the application, are used to explain the principles of the application and are not intended to limit the scope of the application.

The application discloses an intelligent reimbursement method based on visual question and answer, which is shown in fig. 1 and comprises the following steps:

s1, acquiring identification information of a reimbursement person, and constructing a reimbursement person map based on the identification information;

s2, acquiring a ticket image to be reimbursed, identifying the ticket image to be reimbursed based on a visual question-answer mechanism to obtain ticket information, and constructing a reimbursed ticket map based on the ticket information;

s3, acquiring a second problem set, and fusing the reimbursement person pattern and the bill pattern based on the second problem set to generate a reimbursement single pattern; and automatically generating a reimbursement bill according to the reimbursement bill map.

In implementation, the second problem set can be flexibly configured by the enterprise according to the content to be filled in the reimbursement bill, so that the adaptability of the method is improved. By adopting an identification method based on a visual question-answering mechanism, the information of the reimbursement bill to be reimbursed can be automatically identified to construct a reimbursement bill map; and establishing a reimbursement person map according to the user identity information, automatically calculating to obtain data that the reimbursement person can reimburse actually at the time by fusing the reimbursement person map and the bill map according to the second problem set, and generating a reimbursement single map at the time, so that the reimbursement single map at the time is automatically obtained, manual operation of the reimbursement person is not needed, and the method is convenient and quick. Meanwhile, the information of the reimbursement person, the bill information and the reimbursement bill information are stored in the form of a knowledge graph, so that a user can conveniently check corresponding information, financial auditing is convenient, and the association relationship can be obtained rapidly through a knowledge base.

Specifically, in step S1, the obtaining the identity information of the reimbursement person, and constructing a reimbursement person map based on the identity information includes:

s11, acquiring face data or fingerprint data of a reimbursement person or acquiring identity data input by the reimbursement person, and verifying the identity of the user based on the face data, the fingerprint data or the identity data.

In the implementation, the identity data input by the reimburser can be data such as an identity card number or a work number which can uniquely indicate the identity information of the user, and whether the user is legal or not is verified by the face, the fingerprint or other identity data, and only the legal identity user is authorized to use, so that the safety is improved.

In practice, the enterprise employee may be a ticket reimbursement of himself, at which point a face, fingerprint, or other valid identity data may be used to verify his identity. In addition, the method can also be used for reimbursement of other persons, and the identity of the reimbursement person can be verified by inputting other effective identity data. For example, the other valid identity data is an identification card number.

S12, extracting reimbursement person identity information from an enterprise information base if a user identity rule is met, wherein the identity information comprises job level information of the reimbursement person and reimbursement standards corresponding to the job level information; and constructing a reimbursement person map by taking the name of the reimbursement person, the position in the position information and each reimbursement type and the subsidy in the reimbursement standard as entity nodes and taking the reimbursement type and the corresponding amount or condition of the subsidy as attribute nodes.

Because the reimbursement standards of reimbursement persons of different job levels may be different, job level information of the reimbursement persons and reimbursement standards corresponding to the job levels need to be obtained, and a data reference is provided for accurately filling in reimbursement sheet information later.

The knowledge graph technology is used as a component of the artificial intelligence technology, and the strong semantic processing and network organization capacity of the knowledge graph technology provide a basis for intelligent information application. The current knowledge graph is mainly applied to semantic search, intelligent question and answer, personalized recommendation, auxiliary decision making and the like. The knowledge graph expresses knowledge in the form of (entity-relation-entity/attribute value) triples, in the graph, the entity or attribute value is represented by a node, two nodes are connected by an edge, and the edge represents the relation between the two connected nodes, which can be an association relation or an attribute relation.

After the information of the reimbursement person is obtained, the reimbursement person, the position, the reimbursement type and the subsidy are taken as entity nodes, a post relation is formed between the reimbursement person and the position, a reimbursement relation is formed between the position and the reimbursement type, an attribute node is established for the attribute of each reimbursement type by taking the amount or the limiting condition corresponding to each reimbursement type, and an attribute relation is formed between the attribute node and the corresponding reimbursement type entity node. Illustratively, a reimbursement person map is shown in FIG. 3, wherein gray-bottom nodes represent attribute nodes. When the method is implemented, the generated reimbursement person knowledge graph can also be sent to a user interface, so that a user can check conveniently.

In step S2, the reimbursed bill image may be acquired by the imaging scanning device, or the bill image may be acquired by network transmission, for example, the bill image uploaded by the user may be acquired.

Preferably, in order to improve the recognition accuracy, after collecting or acquiring the bill image to be reimbursed, preprocessing is further included on the bill image.

Specifically, the preprocessing includes image illumination normalization, image correction and image noise reduction.

When the method is implemented, the collected image is firstly subjected to image graying, the RGB image is converted into a gray level image, and then illumination normalization is carried out, for example, the image can be processed by adopting methods such as logarithmic transformation, histogram equalization, gamma correction, self-adaptive histogram equalization or local histogram equalization, so that the influence of illumination change on image identification is eliminated, an illumination robust image is obtained, and the image contrast is enhanced. The obtained image is further subjected to image correction. The specific correction process is as follows: firstly, extracting edge vertexes of an image through a Canny operator, calculating a transformation matrix according to the edge vertexes, and then transforming the inclined image into a positive image through the transformation matrix; and further carrying out image segmentation on the corrected image, and segmenting the image from the background to obtain a preprocessed image. For example, an edge detection algorithm may be used to detect edges of the image and segment the image. During implementation, the filtering mode can be adopted to reduce the noise of the image, so that the image quality is improved, and bill information identification is facilitated.

Further, in step S2, identifying the image of the ticket to be reimbursed to obtain ticket information based on the visual question-answering mechanism, and constructing a reimbursed ticket map based on the ticket information, including:

s21, acquiring a first problem set, performing word segmentation and filtering on each problem text in the first problem set, converting the problem text into vector representation based on the filtered word segmentation, and extracting text feature vectors of the problem text;

in practice, the first question set is a pre-configured question set for acquiring ticket information of a ticket image, and illustratively, the first question set may include "what type of ticket is this? "what is the bill amount? "when is the date of invoicing? "etc. The specific questions can be set according to the ticket information which is needed to be obtained.

After the preset first question set is obtained, word segmentation and filtering processing are performed on each question text in the set, and exemplary text word segmentation can be performed by adopting a jiaba word segmentation tool, word segmentation filtering is performed according to a stop word list, and stop words are removed.

The filtered Word segmentation can adopt the existing Word vector model to carry out vector conversion, for example, static Word vector models such as glove, word2Vec and the like are adopted, dynamic Word vector models such as Bert or ELMo and the like are adopted, word segmentation is input into the Word vector models to obtain vector representation of each problem text, and a cyclic neural network can be adopted to extract text features.

S22, extracting image features, region features and text features of each bill image, and fusing the image features, the region features and the text features to obtain fusion features;

specifically, extracting image features, region features and text features of each bill image, and fusing the image features, the region features and the text features to obtain fusion features, wherein the method comprises the following steps:

and extracting image features by adopting a convolutional neural network. For the image characteristics of the bill image, a convolutional neural network is preferably adopted for characteristic extraction.

And detecting a text region in the image by adopting an OCR technology, and extracting position information of the text region as a region characteristic. The position information of the text region includes an abscissa of an upper left corner of the text region, a width of the text region, and a height of the text region.

And (3) recognizing the text information in the text region by adopting an OCR technology, converting the text information into vector representation, and extracting text characteristics of the text information. When the method is implemented, a jiaba word segmentation tool can be used for text word segmentation, word segmentation filtering is carried out according to a stop word list, and stop words are removed. Word vector models such as word2vec and glove can be used for coding, and a cyclic neural network model can be used for extracting text features.

And fusing the image features, the region features and the text features by adopting multi-modal feature fusion to obtain fusion features.

The specific multi-mode feature fusion is to integrate features of different modes into one feature, and when the multi-mode feature fusion is implemented, the image features, the region features and the text features can be spliced, and the feature vectors after the splicing are subjected to dimension reduction by adopting a principal component analysis method to obtain fusion features.

By extracting the image features, the region features and the text features of each text box in the bill image, the feature dimension is expanded, and the accuracy of answer prediction is improved.

S23, combining the text feature vector and the fusion feature of each question, and inputting the combined text feature vector and the fusion feature into a trained visual question-answering model to obtain an answer of each question in the first question set.

In implementation, the combination of the question text feature vector and the fusion feature can be combined by adopting a vector splicing method.

In practice, the trained visual question-answering model is a classifier model. The training process of the model comprises the steps of firstly, determining that some common answers form an answer candidate set according to images and questions in a training set, regarding each candidate answer as a class, carrying out answer prediction according to text feature vectors and fusion feature prediction, and finally taking the candidate answer with the highest probability as a prediction result. And training the model through the training set, and continuously optimizing model parameters to ensure that the model precision meets the requirement and obtain a trained classifier model.

S24, taking the reimbursement bill at this time as a central entity node, and constructing a reimbursement bill map at this time by taking the answer as the entity node or an attribute node.

When the reimbursement bill is used as a central entity node, the entity node can add label information needing to be recorded, such as reimbursement time and the like, so that a user can conveniently inquire related reimbursement information.

Illustratively, what is the type of ticket for the question? And the answer obtained based on the visual question and answer is a train ticket, and the answer is taken as an entity node, and the entity node of the reimbursement ticket and the train ticket node are in an inclusion relationship.

For the question "what the bill amount is", the answer obtained based on the visual question answer is "500 yuan", which is the attribute of the train ticket, and the answer is taken as an attribute node, and the cost relationship between the "train ticket" node and the "500 yuan" node is adopted.

Illustratively, the reimbursement ticket atlas at this time is shown in fig. 4, in which the gray-bottom node represents an attribute node. When the method is implemented, the generated reimbursement bill map can also be sent to a user interface, so that a user can check conveniently.

Preferably, after identifying the image of the to-be-reimbursed bill based on the visual question-answering mechanism to obtain bill information, and before constructing the reimbursed bill map based on the bill information, the method further comprises the step of performing bill auditing based on the identity information and the bill information of the reimbursement person, wherein the bill auditing comprises the following steps: ticket validity auditing, ticket repeatability auditing and ticket attribution authority auditing.

During implementation, the validity of the bill can be checked by connecting with a national tax bureau system. And searching whether the same bill exists in the reimbursed bill through the bill code so as to carry out repeatability verification. For tickets containing ticket attribution authority information, such as train tickets, it is checked whether the ticket attribution authority and the sales force are the same. For the notes which do not pass the examination, specific note information can be fed back to the user side, the user is prompted to change the notes, and the reimbursement flow is finished.

And if the verification is passed, generating a reimbursement single map according to the reimbursement human map and the bill map.

Specifically, in step S3, the reimbursement person spectrum and the ticket spectrum are fused based on the second problem set, so as to generate a reimbursement single spectrum, which includes:

s31, establishing a reimbursement single entity node at the time, extracting reimbursement person entity nodes in the reimbursement person map, and establishing reimbursement relation between the reimbursement person entity nodes and the reimbursement single entity node at the time to form an initial reimbursement single map.

The reimbursement single map needs to contain reimbursement person information, so reimbursement person nodes are directly extracted from the reimbursement person map, reimbursement relation between reimbursement person entity nodes and the reimbursement single entity nodes is established, and an initial reimbursement single map is formed.

S32, aiming at each problem in the second problem set, carrying out problem identification by adopting natural language processing to obtain reimbursement types and inquiry types corresponding to the problems, wherein the reimbursement types comprise airplane tickets, train tickets, taxi tickets, accommodation tickets and catering tickets, and the inquiry types comprise time, ticket number and amount.

Specifically, the problem in the second problem set is to form information required by the reimbursement bill, and the reimbursement bill can be flexibly set according to different reimbursement demands during implementation. For example, the question in the second question set may be "how much money the train ticket can reimburse? "," how much money can be reimbursed for lodging? "" how many train tickets are? "

For the questions in the second question set, since the scope of information to be filled in the reimbursement sheets is limited, and the knowledge base in the form of a map has been formed in steps S1 and S2, identification of the questions in the second question set mainly identifies the object of the question query and the type of the query. The BERT may thus be employed to identify a named entity object of a problem, such as a "train ticket" or "restaurant ticket", with the interrogated object corresponding to a ticket type node in the ticket graph. The type of inquiry includes time, number of notes, and amount, for example, "how much money is total? "" how many "etc., the type of query corresponds to an attribute node in the bill profile. In implementation, a jiaba word segmentation tool can be used for text word segmentation of the problem text in the second problem set, word segmentation filtering is performed according to a stop word list, stop words are removed, word vector models such as word2vec and glove can be used for coding, the problem is converted into vector representation, and text vectors are input into a pre-trained text classification model to identify the type of the query.

S33, searching whether a bill type corresponding to the reimbursement type exists in the bill map, if so, adding the bill type as an entity node into the reimbursement bill map, and establishing the inclusion relation between the reimbursement bill entity node and the bill type node; searching whether a corresponding reimbursement type node exists in the reimbursement person map, if not, searching an answer corresponding to the reimbursement type in the bill map according to the reimbursement type, and establishing a corresponding relation between the bill type node and the answer attribute node by taking the answer as the attribute node; otherwise, using the sum or condition attribute node corresponding to the reimbursement type node in the reimbursement person map as a constraint condition, searching an answer corresponding to the inquiry type in the bill map according to the inquiry type, adopting the constraint condition to process the answer to obtain a final answer, using the final answer as an attribute node, and establishing a corresponding relation between the bill type node and the answer attribute node;

and obtaining the reimbursement single map after processing all the problems.

For example, for the question "how much money is the train ticket? By text classification of named entities, the inquiry object is identified as a train ticket, and the inquiry type is the total amount.

And taking the train ticket as an entity node, and establishing the inclusion relationship between the train ticket node and the reimbursement ticket entity node.

The query object firstly searches whether the entity node exists in the reimbursement human atlas, if the entity node does not exist, the entity node means that reimbursement constraint is not generated for reimbursement of the type, at the moment, a query sentence is directly generated according to the query object and the query type, and a corresponding answer is searched in the reimbursement bill atlas. And taking the obtained answers as attribute nodes corresponding to the entity nodes, and establishing attribute relations between the train ticket nodes and the answer attribute nodes.

If the reimbursement type entity node is found in the reimbursement person map, the reimbursement constraint on the reimbursement type entity node means that reimbursement is restrained, and constraint conditions contained in the reimbursement type are found in the reimbursement person map, for example, the train ticket seat condition is equal.

Searching answers corresponding to the query types in the bill map according to the query types, for example, searching all train ticket nodes in the bill map to obtain the amounts and types of all train tickets, and filtering the query results according to the constraint condition 'second seat', so as to obtain the amounts of all train tickets with the types of 'second seat' as final answers. And taking the final answer as an attribute node, and establishing a corresponding relationship between the entity node of the train ticket and the attribute node of the answer.

For another example, the number of days of business trip is 2 days, the total amount of the catering fees is 300, and the actual amount of the catering is judged to be larger than the reimburseable amount according to the constraint condition of the amount of the catering fees, so that the final answer is 200 yuan, 200 yuan is taken as an attribute node, and the corresponding relation between the entity node of the catering and the attribute node of the 200 yuan is established.

And adding corresponding nodes in the reimbursement single map according to each problem in the second problem set, and obtaining the reimbursement single map after all the problems are processed. And automatically generating a reimbursement bill according to the reimbursement bill map. The generated reimbursement single map can also be sent to a user interface, so that the user can check conveniently. The reimbursement process is full-automatic, reimbursement manual filling is not needed, the user operation is convenient, filling errors can be avoided, and meanwhile, reimbursement data are clearly displayed, so that the user can check conveniently.

In order to further facilitate the use of users, after automatically generating the reimbursement bill according to the reimbursement bill map, the method further comprises the following steps:

s4, taking the reimbursement person nodes in the reimbursement single map as alignment nodes, and integrating the reimbursement single map into the reimbursement information knowledge map of the enterprise; acquiring a third problem input by a user, analyzing the problem to generate a query statement, querying a result corresponding to the query statement in the enterprise reimbursement information knowledge graph, and feeding back the result to the user

In addition, the reimbursement person nodes in the reimbursement single map are used as alignment nodes, and the reimbursement single map is integrated into the enterprise reimbursement information knowledge map, so that the enterprise reimbursement information knowledge map is updated. By constructing the enterprise reimbursement knowledge graph, on one hand, financial auditing can be rapidly carried out through the association relationship, the working efficiency is improved, and meanwhile, the user is convenient to inquire about related reimbursement business. The third question input by the user can be a natural language question, the question is analyzed through natural language processing to generate a corresponding query statement, and an answer to the question can be rapidly inferred through a knowledge graph, so that the user can inquire about the reimbursement related business conveniently.

In the implementation, the user in step S4 may be a sales person, an auditor, or other personnel.

Specifically, in step S4, a problem related to the reimbursement service input by the user is obtained, the problem is parsed to generate a query statement, and a result corresponding to the query statement is queried in the enterprise reimbursement information knowledge graph, including:

s41, carrying out named entity recognition on the reimbursement business related problems, and converting the problem text into sentence vectors by adopting a word vector model.

When the method is implemented, a jiaba word segmentation tool can be used for text word segmentation, word segmentation filtering is carried out according to a stop word list, stop words are removed, word vector models such as word2vec and glove can be used for encoding, and the problems are converted into vector representations. The BERT model may be employed for named entity recognition.

S42, performing problem classification on the sentence vectors by adopting a trained neural network model, and constructing a query sentence based on a named entity and the problem classification;

in practice, neural network model training may be performed as follows. First, a question template is designed, and a common question method for asking this question, such as asking "the number of reimbursements of XXX" that "XX reimburses several times" is listed for each template file. In implementation, template sentences corresponding to each template can be preset, wherein the named entities can be represented by placeholders.

And performing word segmentation on each problem, filtering and converting the word vectors into word vectors, so that sentence vectors are formed, and the sentence vectors and corresponding template labels are used as training data to form a training set. And performing model training by adopting a neural network model, such as a convolutional neural network, so as to obtain a trained sentence type recognition neural network model.

The sentence vector obtained in step S41 is input into a trained neural network model to wait for a corresponding template of the sentence, thereby identifying the type of user query. And replacing the placeholder corresponding to the named entity in the template with the named entity identified in the step S41, thereby generating a query statement.

S43, inquiring a result corresponding to the inquiry statement in the enterprise reimbursement information knowledge graph.

In implementation, the knowledge graph can be stored by adopting a neo4j database, and can be queried through a cypher statement.

In one embodiment of the present application, an intelligent reimbursement system based on visual question and answer is disclosed, as shown in fig. 2, comprising:

the reimbursement person map construction module is used for acquiring reimbursement person identity information and constructing reimbursement person maps based on the identity information;

the bill map construction module is used for acquiring the bill image to be reimbursed, identifying the bill image to be reimbursed based on a visual question-answer mechanism to obtain bill information, and constructing a reimbursed bill map based on the bill information;

the reimbursement sheet generation module is used for acquiring a second problem set, and fusing the reimbursement person spectrum and the bill spectrum based on the second problem set to generate a current reimbursement sheet spectrum; automatically generating a reimbursement bill according to the reimbursement bill map;

preferably, the system further comprises:

the user question-answering module takes reimbursement person nodes in the reimbursement single map as alignment nodes, and fuses the reimbursement single map into the reimbursement information knowledge map of the enterprise; and acquiring a third problem input by the user, analyzing the problem to generate a query statement, querying a result corresponding to the query statement in the enterprise reimbursement information knowledge graph, and feeding back the result to the user.

Preferably, the bill map construction module identifies the to-be-reimbursed bill image based on a visual question-answering mechanism to obtain bill information, and constructs the reimbursed bill map based on the bill information, including:

the problem feature extraction module is used for acquiring a first problem set, performing word segmentation and filtering processing on each problem text in the first problem set, and inputting the filtered word segmentation into the word vector model to obtain text feature vectors of each problem;

the image feature fusion module is used for extracting image features, region features and text features from each bill image, and fusing the image features, the region features and the text features to obtain fusion features;

the answer presumption module is used for inputting the text feature vector and the fusion feature of each question into a trained visual question-answer model after combining to obtain an answer of each question in the first question set;

and the map construction module is used for constructing the map of the reimbursement bill by taking the reimbursement bill as a central entity node and taking the answer as an entity node or an attribute node.

The method embodiment and the system embodiment are based on the same principle, and the related parts can be mutually referred to and can achieve the same technical effect. The specific implementation process refers to the foregoing embodiment, and will not be described herein.

Those skilled in the art will appreciate that all or part of the flow of the methods of the embodiments described above may be accomplished by way of a computer program to instruct associated hardware, where the program may be stored on a computer readable storage medium. Wherein the computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory, etc.

The present application is not limited to the above-mentioned embodiments, and any changes or substitutions that can be easily understood by those skilled in the art within the technical scope of the present application are intended to be included in the scope of the present application.

Claims (10)

1. An intelligent reimbursement method based on visual questions and answers is characterized by comprising the following steps:

acquiring identity information of a reimbursement person, and constructing a reimbursement person map based on the identity information;

acquiring a ticket image to be reimbursed, identifying the ticket image to be reimbursed based on a visual question-answering mechanism to obtain ticket information, and constructing a ticket reimbursement map based on the ticket information;

acquiring a second problem set, and fusing the reimbursement person pattern and the bill pattern based on the second problem set to generate a reimbursement single pattern; and automatically generating a reimbursement bill according to the reimbursement bill map.

2. The intelligent reimbursement method based on visual question and answer according to claim 1, wherein the obtaining the identity information of the reimbursement person and constructing a reimbursement person map based on the identity information comprises:

acquiring face data or fingerprint data of a sales person or acquiring identity data input by the sales person, and verifying the identity of the user based on the face data, the fingerprint data or the identity data;

extracting reimbursement person identity information from an enterprise information base if the user identity rule is met, wherein the identity information comprises job level information of the reimbursement person and reimbursement standards corresponding to the job level information; and constructing a reimbursement person map by taking the name of the reimbursement person, the position in the position information and each reimbursement type and the subsidy in the reimbursement standard as entity nodes and taking the reimbursement type and the corresponding amount or condition of the subsidy as attribute nodes.

3. The intelligent reimbursement method based on visual question and answer according to claim 1, wherein the method is characterized in that the method comprises the steps of identifying the image of the reimbursement bill to be reimbursed to obtain bill information based on a visual question and answer mechanism, constructing a reimbursement bill map based on the bill information, and comprises the following steps:

acquiring a first problem set, performing word segmentation and filtering on each problem text in the first problem set, converting the problem text into vector representation based on the filtered word segmentation, and extracting text feature vectors of the problem text;

extracting image features, regional features and text features of each bill image, and fusing the image features, regional features and text features to obtain fusion features;

combining the text feature vector and the fusion feature of each question, and inputting the combined text feature vector and the fusion feature into a trained visual question-answering model to obtain an answer of each question in the first question set;

and constructing a current reimbursement bill map by taking the reimbursement bill as a central entity node and taking the answer as an entity node or an attribute node.

4. The intelligent reimbursement method based on visual question and answer according to claim 3, wherein extracting image features, region features and text features of each bill image, and fusing the image features, the region features and the text features to obtain fusion features comprises:

extracting image features by adopting a convolutional neural network;

detecting a text region in an image, and extracting position information of the text region as a region feature;

recognizing text information in a text region by adopting an OCR technology, converting the text information into vector representation, and extracting text characteristics of the text information;

and fusing the image features, the region features and the text features by adopting multi-modal feature fusion to obtain fusion features.

5. The visual question-answering based intelligent reimbursement method according to claim 1, wherein fusing the reimbursement person pattern and the ticket pattern based on the second question set to generate a current reimbursement single pattern comprises:

the method comprises the steps of establishing a reimbursement single entity node at the time, extracting reimbursement person entity nodes in a reimbursement person map, and establishing reimbursement relation between the reimbursement person entity nodes and the reimbursement single entity node at the time to form an initial reimbursement single map;

aiming at each problem in the second problem set, carrying out problem identification by adopting natural language processing to obtain reimbursement types and inquiry types corresponding to the problems, searching whether a bill type corresponding to the reimbursement types exists in the bill map, if so, adding the bill type as an entity node into a reimbursement single map, and establishing the inclusion relation between the reimbursement single entity node and the bill type node; searching whether a corresponding reimbursement type node exists in the reimbursement person map, if not, searching an answer corresponding to the reimbursement type in the bill map according to the reimbursement type, and establishing a corresponding relation between the bill type node and the answer attribute node by taking the answer as the attribute node; otherwise, using the sum or condition attribute node corresponding to the reimbursement type node in the reimbursement person map as a constraint condition, searching an answer corresponding to the inquiry type in the bill map according to the inquiry type, processing the answer by adopting the constraint condition to obtain a final answer, and establishing a corresponding relation between the bill type node and the answer attribute node by taking the final answer as the attribute node;

and obtaining the reimbursement single map after processing all the problems.

6. The intelligent reimbursement method based on visual question and answer according to claim 1, wherein after automatically generating reimbursement sheets according to the reimbursement sheet map, the method further comprises:

taking reimbursement person nodes in the reimbursement single map as alignment nodes, and integrating the reimbursement single map into the reimbursement information knowledge map of the enterprise;

and acquiring a third problem input by the user, analyzing the problem to generate a query statement, querying a result corresponding to the query statement in the enterprise reimbursement information knowledge graph, and feeding back the result to the user.

7. The intelligent reimbursement method based on visual question and answer according to claim 6, wherein obtaining reimbursement business related problems input by a user, analyzing the problems to generate a query statement, and querying a result corresponding to the query statement in the enterprise reimbursement information knowledge graph, comprises:

the related problems of the reimbursement service are identified by named entities, and a word vector model is adopted to convert the text of the problems into sentence vectors;

performing problem classification on the sentence vectors by adopting a trained neural network model, and constructing a query sentence based on a named entity and the problem classification;

and inquiring a result corresponding to the inquiry statement in the enterprise reimbursement information knowledge graph.

8. The intelligent reimbursement method based on visual question and answer according to claim 2, wherein after acquiring the ticket image to be reimbursed this time, the method further comprises preprocessing the ticket image.

9. The intelligent reimbursement method based on visual question and answer according to claim 2, wherein after identifying a to-be-reimbursed bill image based on a visual question and answer mechanism to obtain bill information, before constructing a reimbursement bill map based on the bill information, the intelligent reimbursement method further comprises the step of performing bill verification based on identity information and bill information of the reimbursement person, and the bill verification comprises the following steps: ticket validity auditing, ticket repeatability auditing and ticket attribution authority auditing.

10. An intelligent reimbursement system based on visual questions and answers is characterized by comprising the following modules:

the reimbursement person map construction module is used for acquiring reimbursement person identity information and constructing reimbursement person maps based on the identity information;

the bill map construction module is used for acquiring the bill image to be reimbursed, identifying the bill image to be reimbursed based on a visual question-answer mechanism to obtain bill information, and constructing a reimbursed bill map based on the bill information;

the reimbursement sheet generation module is used for acquiring a second problem set, and fusing the reimbursement person spectrum and the bill spectrum based on the second problem set to generate a current reimbursement sheet spectrum; and automatically generating a reimbursement bill according to the reimbursement bill map.