JP2022013524A - Form information generation apparatus, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a form information generation apparatus for generating new form information on the basis of past form information, a method, and a program.

SOLUTION: A form information generation apparatus 100 includes: a storage unit which stores past feature quantities extracted from past form images and past form information based on character information included in the past form images, in association with each other; an extraction unit which extracts a target feature quantity from a targeted form image; a calculation unit which calculates similarities between the target feature quantity and the past feature quantities; a selection unit which selects past form information on the basis of the similarities; a read unit which reads character information of a predetermined item from the targeted form image; and a generation unit which generates new form information associated with the targeted form image, from the past form information selected in the selection unit and the character information read in the read unit, to be displayed.

SELECTED DRAWING: Figure 3

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to a form information generator, a method and a program.

In Patent Documents 1 and 2, "The management device 1 stores the image information indicating the image of the paper medium and the input information previously input corresponding to the image in the storage unit 11 in association with each other. From the image information stored in the storage unit 11, image information similar to the acquired image information is specified ((3) in FIG. 1). The management device 1 is input information associated with the specified image information. Is specified, and the input information and the character information extracted from the acquired image information are output to the input screen transmitted to the user terminal 2 ((4) in FIG. 1). " ..
[Prior Art Document]
[Patent Document]
[Patent Document 1] Japanese Patent Application Laid-Open No. 2017-174199
[Patent Document 2] Japanese Patent Application Laid-Open No. 2017-174200

In the first aspect of the present invention, the form information generation device is a past form information based on the past feature amount extracted from the past form image and the character information described in the past form image. An acquisition unit that acquires the target feature amount in association with each other, an extraction unit that extracts the target feature amount from the target form image, a calculation unit that calculates the similarity between the target feature amount and the past feature amount, and a past based on the similarity degree. A selection unit that selects the form information of, a reading unit that reads the character information of the predetermined item from the target form image, the past form information selected by the selection unit, and the characters read by the reading unit. It is provided with a generation unit that generates and displays new form information associated with the target form image from the information of.

In the second aspect of the present invention, the device for generating the billing information to be read by the accounting processing system, and the billing information and the billing image corresponding to the billing information are the first information and the second information, respectively. And, the acquisition unit that acquires the past invoice information in association with the past invoice image, and the past invoice image that is similar to the new invoice image is selected by referring to the acquisition unit. A first information acquisition unit that extracts the first information from the selected past billing information, and a second information acquisition unit that reads the second information from the new invoice image with an optical character recognition device (OCR). It includes a generation unit that generates new billing information from the first information extracted by the first information acquisition unit and the second information read by the second information acquisition unit.

In the third aspect of the present invention, there is a method of associating the past feature amount extracted from the past form image with the past form information based on the character information described in the past form image. A step to acquire, a step to extract the target feature amount from the target form image, a step to calculate the similarity between the target feature amount and the past feature amount, and a step to select the past form information based on the similarity degree. The target is from the step, the step of reading the character information of the predetermined item from the target form image, the past form information selected in the selection step, and the character information read in the reading step. It is provided with a step of generating and displaying new form information associated with the form image.

In the fourth aspect of the present invention, the program is a computer to realize the above method.

In the fifth aspect of the present invention, there is a method for generating billing information to be read by the accounting processing system, and the billing information and the billing image corresponding to the billing information are the first information and the second information, respectively. And, the step of associating the past invoice information with the past invoice image and acquiring the past invoice image, and the past invoice similar to the new invoice image by referring to the acquired past invoice image. A step of selecting a calligraphy image and extracting the first information from the selected past billing information, a step of reading the second information from the new billing image with an optical character recognition device (OCR), and the first information. , A step of generating new billing information from the second information.

In the sixth aspect of the present invention, the program is a computer to realize the above method.

The outline of the above invention does not list all the necessary features of the present invention. A subcombination of these feature groups can also be an invention.

A specific example of the invoice image 10 which is an example of the form image in this embodiment is shown. A specific example of the billing information 20 which is an example of the form information generated by the user from the invoice image 10 of FIG. 1 is shown. It is a block diagram of the form information generation apparatus 100 which concerns on this embodiment. The selection data 112 is schematically shown. The reading condition 113 is schematically shown. Correspondence relationship 114 is schematically shown. An example of the operation flow S10 of the form information generation apparatus 100 is shown. An example of a new invoice image 12 is shown. An example of the new billing information 22 generated by the generation unit 160 is shown. It is a block diagram of the learning apparatus 200 which generates the learned extraction unit 111 stored in the storage unit 110 of the form information generation apparatus 100. The method of deep distance learning is schematically shown. The correct answer rate when a similar image search is performed by the form information generation device 100 is shown in the case where the deep distance learning is used and the case where the learning by the category classification is used. Other methods of deep distance learning are schematically shown. An example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part is shown.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention to which the claims are made. Also, not all combinations of features described in the embodiments are essential to the means of solving the invention.

FIG. 1 shows a specific example of the invoice image 10 which is an example of the form image in the present embodiment. The invoice image 10 in FIG. 1 is a bill for electricity charges from the "ABC electric power company" to the user "○○ Co., Ltd." of the present embodiment. The invoice image 10 is sent on paper and converted into image data by a scanner or the like on the user side, or originally sent as general-purpose image data such as PDF.

FIG. 2 shows a specific example of the billing information 20 which is an example of the form information generated by the user from the invoice image 10 of FIG. When the user receives the invoice image 10, the user uses the information necessary or interested by the user himself / herself from the character information described in the invoice image 10 for, for example, an accounting processing system. The billing information 20 is preferably in a format such as csv or xls so that it can be easily input to the accounting processing system.

The billing information 20 is information used in the accounting processing system on the user side. By using the accounting system, users can record expenses according to their own rules and prepare financial statements. In the example of FIG. 2, the items of the billing information 20 include billing items, expense items, expense generation departments, amounts, billing dates, payment due dates, and billing sources. In addition to these items, the billing information 20 may be associated with information on the bank account of the business partner required for remittance processing, a remarks column, and the like. In addition, although each item is described here for each department, if it is not necessary to record each item as an expense for each department, for example, it is described as company-wide or general affairs department in the expense generation department.

Here, when the invoice image 10 and the billing information 20 are compared, there is character information that is not reflected in the billing information 20, such as the "discount" described in the invoice image 10. On the other hand, characters such as the "billing amount" of the invoice image 10 and the "amount" of the billing information 20 are not the same as the character information described in the invoice image 10, but are converted and reflected by their own rules. There is information on. Therefore, the billing information 20 desired by the user is not always easily generated from the invoice image 10 only by increasing the accuracy of the so-called OCR.

Further, when the electricity rate is a fixed amount contract on a yearly basis, the information on the electricity rate described in the invoice image 10 is the same as the previous month. Therefore, in such a case, it is preferable to accurately select which invoice of the previous month corresponds to the invoice.

Therefore, in the present embodiment, the billing information associated with the past feature amount similar to the feature amount of the new invoice image is selected. Further, by using the past billing information and the character information read from the new billing image, a device for easily generating the billing information desired by the user is provided.

FIG. 3 is a block diagram of the form information generation device 100 according to the present embodiment. An example of the form information generation device 100 is a general-purpose computer in which a program is installed. Further, the form information generation device 100 is configured to be able to communicate with the user's terminal device, and accepts correction of billing information from the user's terminal device.

The form information generation device 100 includes a storage unit 110, an extraction unit 120 that extracts a feature amount from an invoice image, and a calculation unit 130 that calculates the degree of similarity between the extracted feature amount and the feature amount stored in the storage unit 110. And a selection unit 140 that selects billing information stored in the storage unit 110 based on the degree of similarity. The form information generation device 100 further generates new billing information from the reading unit 150 that reads the character information of the predetermined item from the invoice image, and the selected billing information and the read character information. It is provided with a generation unit 160 to be displayed.

The storage unit 110 stores the learned extraction unit 111, the selection data 112, the reading condition 113, and the correspondence relationship 114. For example, these pieces of information are stored in advance by the user from another server or the like via the Internet or by a recording medium such as a USB memory. From the viewpoint of acquiring and storing the information from the outside, it can be said that the storage unit 110 also has a function as an acquisition unit.

The learned extraction unit 111 extracts the feature amount from the invoice image by a method learned in advance, and the details will be described later. Instead of storing the learned extraction unit 111 itself, each parameter or weighting value obtained by learning in advance may be stored in the extraction unit 120 so that it can be used.

FIG. 4 schematically shows the selection data 112. In the selection data 112, the invoice image 10, the feature amount 30, and the billing information 20 are associated with each other as one data set. Further, the invoice image 10 is associated with a label 40 for each billing company. However, the invoice image 10 is not limited to this, and the invoice image 10 may be associated with labels for different invoice formats from the same invoice source. Since the invoice image 10 and the label 40 are mainly used for learning described later, they do not have to be included in the selection data 112.

The invoice image 10 and the invoice information 20 may be those used by the user in the accounting system in the past. These are small amounts and many kinds of information, and for example, there are about 100 sheets of information for each company (that is, for each label 40) for about 10,000 companies (that is, for each label 40). In the example of FIG. 4, in order to simplify the explanation, there are two labels 40, "ABC power" and "DEF newspaper dealer", and three data sets for "ABC power", "DEF newspaper". It shows one data set for "dealer".

The feature amount V0 and the like represent the features of the invoice image 10 extracted by the learned extraction unit 111 with a d-dimensional vector (d is, for example, about 1000). The feature amount V0 or the like does not have to be stored in a form that can be visually recognized by the user. In other words, the storage unit 110 has at least the past feature amount, which is the feature amount extracted from the past invoice image, and the past billing based on the character information described in the past invoice image. It can be said that the information is associated and stored.

FIG. 5 schematically shows the reading condition 113. The reading condition 113 is associated with an item read by the reading unit 150 from the invoice image 10 and a condition for reading the item.

Here, the "condition" indicates where or how the character information corresponding to the item can be read by OCR in the invoice image 10 on a rule basis. In the example of FIG. 5, for example, the condition "1)" of the item "issue date" indicates that the issue date is obtained from the character string to the right of the character string "issue date" of the invoice image 10. Further, for example, the condition "2)" of the item "issue date" indicates that the issue date is obtained from the character string below the character string "issue date" of the invoice image 10. Further, for example, the condition "3)" of the item "issue date" is obtained from a character string in which the issue date includes any of the characters "year", "month", and "day" in the upper right area of the invoice image 10. Show that. As the reading condition, a character string including, for example, two characters of "departure", "line", and "day" may be used instead of the character string "issue date". This makes it possible to read a character string such as "issue date" as the issue date.

Further, in the example of FIG. 5, for example, the condition "1)" of the item "billing amount" indicates that the billing amount is obtained from the character string to the right of the character string "billing amount" of the invoice image 10. .. Further, for example, the condition "2)" of the item "billing amount" indicates that the billing amount is obtained from the character string below the character string "billing amount" of the invoice image 10. Further, for example, the condition "3)" of the item "billing amount" indicates that the billing amount is obtained from the character string to the right of the character string "total" of the invoice image 10. Further, for example, the condition "4)" of the item "billing amount" indicates that the billing amount is obtained from the character string below the character string "total" of the invoice image 10. As a reading condition, instead of the character string "billing amount", an item containing, for example, three characters of "contract", "request", "money", and "amount" is described as "billing amount". May be. This makes it possible to read a character string such as "billing amount" as the billing amount.

Further, in the example of FIG. 5, for example, the condition "1)" of the item "payment date" indicates that the payment date is obtained from the character string to the right of the character string "payment date" of the invoice image 10. .. Further, for example, the condition "2)" of the item "payment date" indicates that the payment date is obtained from the character string below the character string "payment date" of the invoice image 10. As a reading condition, instead of the character string "payment date", an item containing, for example, two characters of "payment", "payment", "period", "day", and "limit" is described as "payment date". It may be a character string. This makes it possible to read character strings such as "payment deadline" and "billing deadline" as the payment deadline.

Of the multiple conditions, reading is tried in ascending order of number, and if not found, the next condition is tried. The reading condition 113 is preferably set in advance by a person having knowledge of OCR, but may be changed by the user.

FIG. 6 schematically shows the correspondence 114. The correspondence relationship 114 is associated with a correspondence relationship between an item read by the reading unit 150 and an item of past billing information 20 of selection data, and a process performed by the generation unit 160 for the item. Correspondence relationship 114 may be arbitrarily set so as to obtain the billing information 20 desired by the user.

In the example of FIG. 6, the reading item "issue date" corresponds to the billing information item "billing date". Then, information for performing a process of overwriting the "billing date" of the billing information with the "issue date" read in the reading item is stored in the storage unit 110.

Further, in the example of FIG. 6, the reading item "billing amount" corresponds to the billing information item "amount". Then, information for performing a process of comparing the total of the "amounts" of the billing information with the "billing amount" read in the reading item is stored in the storage unit 110.

Further, in the example of FIG. 6, the reading item “payment date” corresponds to the billing information item “payment date”. Then, information for performing a process of overwriting with the total of the "payment date" of the billing information and the "payment date" read in the reading item is stored in the storage unit 110.

In the correspondence 114, it is not necessary to associate all the items read under the reading condition 113 with any item of the billing information 20. Further, in the correspondence relationship 114, it is not necessary to associate all the items included in the billing information 20 of the selection data 112 with any of the reading items.

Rather, when there is a lot of information that does not change each time, such as character information included in the invoice image 10, it may be preferable not to include the item related to the information that does not change in the correspondence 114. By not including it in the correspondence 114, it may be possible to prevent erroneous billing information from being generated based on the information of characters erroneously read by the reading unit 150.

FIG. 7 shows an example of the operation flow S10 of the form information generation device 100, and FIG. 8 shows an example of a new invoice image 12. The operation flow S10 is started when a new invoice image 12 is input. In the following description, the new billing information will be represented by reference numeral 22 and will be represented separately from the billing information 20 stored in the storage unit 110.

The extraction unit 120 reads the learned extraction unit 111 from the storage unit 110 and extracts the target feature amount, which is the feature amount of the invoice image 12 (S100). It can be said that the extraction unit 120 extracts the target feature amount by a method corresponding to the learned method of extracting the feature amount 30 from the past invoice image 10.

The calculation unit 130 calculates the degree of similarity between the target feature amount and the feature amount 30 of the selection data 11 stored in the storage unit 110 (S102). There are several methods for calculating the similarity, and it does not necessarily have to be the same as the method for calculating the similarity used for training the learned extraction unit 111, which will be described later.

The calculation unit 130 calculates the similarity with the target feature amount with respect to the feature amount 30 of all or a predetermined part of the selection data 112 (S104). As a result, when there is at least one feature amount 30 having a similarity equal to or higher than the threshold value (S106: Yes), the process proceeds to step S107. On the other hand, if there is no feature amount 30 having a similarity equal to or higher than the threshold value (S106: No), the process proceeds to step S116. The threshold value is preset and stored in the storage unit 110, but may be changed by the user.

In step S107, the selection unit 140 selects the billing information 20 of the selection data 112 based on the similarity (S107). In this case, the selection unit 140 selects (that is, the same number) the billing information 20 associated with the feature amount 30 of a predetermined number, for example, 10 in descending order of similarity.

Hereinafter, in order to simplify the description, an example in which the billing information 20 of FIG. 2 is selected will be described.

In parallel with steps S100 to S107, the reading unit 150 reads character information from the invoice image 12 for the item corresponding to the condition based on the reading condition 113 stored in the storage unit 110 (S120). In the case of the reading condition 113 in FIG. 5, the information on the issue date and the billing amount is read. If the invoice image 12 in FIG. 8 is correctly read, "June 10, 2nd year of Reiwa" is used as the character information of the issue date, and ": ¥ 110,000-" is used as the character information of the invoice amount. Is read.

In this case, the reading unit 150 may use a known atypical OCR engine. For example, a character area may be extracted by CTPN, EAST, SegLink, TextBoxes ++, PSENet, TextSnake, etc., and character information may be read from the character area using a CRNN (convolutional recurrent neural network) model. At this time, various networks such as VGG16, ResNet for the CNN part, LSTM, GRU, seq2seq for the RNN part, and attention mechanism can be used for the RNN part. Further, after extracting the character area, the characters may be divided into characters one by one, and then the character information may be read by a CNN model (VGG16, ResNet, etc.). Further, the process from extracting the character area to reading the character information may be performed end-to-end on one network.

In step S108, the generation unit 160 overwrites the billing date of the selected billing information 20 with the read issue date based on the correspondence 114 stored in the storage unit 110. In this case, the generation unit 160 may overwrite the read character information of the issue date as it is, or may convert and overwrite according to a predetermined conversion rule. The conversion rule may be set by the user as a process of the correspondence relationship 114, or may be stored in the storage unit 110 separately from the correspondence relation 114. In this embodiment, an example of converting the era name into the Christian era and then converting it into the form of "YYYY / MM / DD" will be described.

Next, the generation unit 160 compares the total amount of the selected billing information 20 with the read billing amount based on the correspondence 114 stored in the storage unit 110 (S110). In the case of the invoice image 12 of FIG. 8 and the example of the billing information 20 of FIG. 2, the generation unit 160 totals the columns of the amount of the billing information 20 to obtain "110,000". The generation unit 160 compares this with the numerical part "110,000" in the character information ": ¥ 110,000-" of the invoice amount read from the invoice image 12, and determines that they match.

When the determination in step S110 is Yes, the generation unit 160 generates the billing information overwritten in step S108 as new billing information and displays it on a display or the like (S114).

FIG. 9 shows an example of the new billing information 22 generated by the generation unit 160. In the billing information 22, the billing date and the payment due date among the character information of the billing information 20 of FIG. 2 are rewritten with the information of the issue date and the payment due date read by the reading unit 150. On the other hand, with respect to the non-rewritten character information such as the billing item, the character information of the selected past billing information 20 is used for the new billing information 22.

Further, the billing date is expressed in italics so as to be distinguishable from unrewritten information such as billing items. The distinguishable representation may be in other ways, such as bold, deficit, blinking, etc.

Further, the generation unit 160 may display the billing information 22 according to the accuracy of reading the character information read by the reading unit 150. For example, among the billing information 22, the information rewritten with the information read by the reading unit 150 may be displayed so as to be distinguishable between those having a higher accuracy than the threshold value and those having a lower accuracy than the threshold value. The distinguishable display may be italic, bold, deficit, blinking, etc., and it is preferable that the expression with low accuracy is easy for the user to see.

If the determination in step S110 is No, the generation unit 160 adds a warning to the newly generated billing information 22 (S112). An example of a warning is to make the amount column of billing information 22 italic, bold, deficit, blinking, and so on. It is preferable that the expression is different from the expression used for the above rewritten information. In addition to or instead of this, the text information "The billing amount is different" may be displayed in the margin of the billing information 22.

If the determination in step 106 is No, the selection unit 140 warns that there is no similar invoice image in the past (S116). For example, the text information "No invoice with high similarity was found" may be displayed on a display or the like.

As a result, the operation flow S10 ends. After that, the form information generation device 100 accepts corrections and additions of the user regarding the displayed billing information 22. Here, since a predetermined number (for example, 10) of billing information 20 is selected in order of increasing similarity in step S107, the new billing information 22 is displayed in the same number as the above number (for example, 10). Will be. The user selects the most preferable billing information from the plurality of billing information 22, and then corrects or adds the billing information. If the warning is given in step S116, the input of completely new billing information 22 may be accepted.

It should be noted that the steps S110 and S112 and / or step S116 are not necessarily necessary processes and may be omitted as appropriate.

When the user inputs that the billing information 22 is confirmed, the form information generation device 100 displays it on a display or the like as the confirmed information. Instead of or in addition to displaying it on a display or the like, it may be exported to an accounting processing system or output as data to the outside. Further, the form information generation device 100 associates the invoice image 12 and / or its feature amount with the billing information 22 and adds it to the selection data 112 of the storage unit 110 as a new data set. That is, it becomes a candidate to be selected as a similar image from the next time.

FIG. 10 is a block diagram of the learning device 200 that generates the learned extraction unit 111 stored in the storage unit 110 of the form information generation device 100. An example of the learning device 200 is a program installed on a general-purpose computer.

The learning device 200 is based on the storage unit 210, the extraction unit 220 that extracts the feature amount 30 from the invoice image 10, the similarity calculation unit 230 that calculates the similarity between the extracted feature amounts 30, and the similarity. It is provided with a loss function calculation unit 240 that calculates the value of the loss function and minimizes it. In the present embodiment, the learning device 200 uses the CNN as the extraction unit 220 to perform deep distance learning. The extraction unit 220 is not limited to the CNN, and may extract features using a neural network such as a fully coupled network, an RNN, or a self-attention. Further, the extraction unit 220 may extract local features such as SIFT and HOG instead of the neural network.

The storage unit 110 stores the learning data 211, the learning parameter 212, and the learned extraction unit 111. Before learning, the learned extraction unit 111 does not have to be recorded.

The learning data 211 includes at least a label 40 and a data set in which the invoice image 10 and the billing information 20 are associated with each other. The data set in which the label 40, the invoice image 10 and the billing information 20 are associated with each other is stored by the learner performing the learning, for example, from another server or the like via the Internet or by a recording medium such as a USB memory. It is stored in the unit 210.

The learning data 211 also includes a feature amount 30 associated with the billing information 20 as a result of learning. It is preferable that the data set of the training data 211 is the same as the data set of the selection data 112 used in the form information generation device 100, or at least a part thereof is preferably largely overlapped. For the sake of simplification of the description, in this embodiment, it is assumed that the data set of the learning data 211 and the data set of the selection data 112 are the same.

The learning parameter 212 stores the value of the parameter used for the learning according to the learning method. When CNN is used as the extraction unit 220 for deep distance learning, the parameters of the extraction unit 220 include the number of network layers, the number of nodes in each layer, the initial value of the weight, and the like. Further, the parameters of deep distance learning include the learning rate and the number of learnings as learning conditions, a loss function, and the like. The value of the parameter is set by the learner and may be changed as appropriate during learning.

FIG. 11 schematically shows a method of deep distance learning. First, prepare a pair of datasets.

Each of the paired invoice images G1 and G2 is input to the extraction unit 220, and the feature quantities C1 and C2, respectively, are calculated. The feature quantities C1 and C2 are input to the similarity calculation unit 230, and the similarity between them is calculated. The similarity may be any known method such as the reciprocal of the distance function such as the cosine similarity and the Euclidean distance of the feature space, but the one corresponding to the loss function is used.

The similarity and the labels L1 and L2 of the invoice image of the pair are input to the loss function calculation unit 240 to calculate the value of the loss function, and the extraction unit is subjected to error back propagation so that the value becomes smaller. Update the weight etc. of 220. If the label L1 and the label L2 are the same, the loss function has a smaller value as the similarity is larger (that is, the closer the distance between the vector of the feature amount is), and the label L1 and the label L2 are different. The smaller the similarity (that is, the farther the distance between the feature vectors), the smaller the value of the function. Contrastive Loss, ArcFace, CosFace, SphereFace and the like may be used as such a loss function.

The above update is repeated based on a preset number of learning times or early stopping ping. The above update may be performed at any of batch, online, and mini-batch update timings.

In the case of a large number of labels and a small number of data such as the invoice image 10, it is preferable to sample pairs having the same label and pairs having different labels so that they are used for learning at a predetermined ratio, for example, at equal ratios. .. As a result, it is possible to avoid the problem that the probability that pairs with the same label are used is small and the learning does not proceed, which occurs when sampling at random.

The learning device 200 stores the extraction unit 220 obtained as a result of learning as described above in the storage unit 210 as the learned extraction unit 111. Instead of storing the extraction unit 220 itself, parameters such as weights used in the extraction unit 220, which can reproduce the same operations as the extraction unit 220, may be stored in the learned extraction unit 111. By reading the learned extraction unit 111 itself or the parameters used therein into the extraction unit 120 of the form information generation device 100, the feature amount 30 is extracted by the same operation as the extraction unit 220 obtained as a result of learning above. Can be done.

As described above, the feature amount 30 of the past invoice image 10 and the extraction unit 120 of the form information generation device 100 correspond to those obtained by machine learning. This makes it possible to perform a highly accurate similar image search. Here, the "corresponding" may be the same, or may be reproduced by reading a parameter or the like, and an equivalent calculation result can be obtained within the range of other factors such as probability. Including being.

FIG. 12 shows the correct answer rate when a similar image search is performed by the form information generation device 100 in the case of using the deep distance learning and the case of using the learning by categorization. This is the result of group cross-validation by dividing approximately 1000 images of 1 to 100 labels with 100 labels into 3 groups of 30 to 40 labels as learning data. In FIG. 12, the horizontal axis indicates the number of search candidates, and the vertical axis indicates the top—k correct answer rate (the ratio of at least one invoice with the same label included in the search candidates).

As shown in FIG. 12, when deep distance learning is used as machine learning, it is possible to improve the accuracy of a large number of types and a small number of search targets, such as the invoice image 10, as compared with learning by categorization.

FIG. 13 schematically shows another method of deep distance learning. In the method of FIG. 13, instead of preparing a pair of data sets, the similarity between the feature amount for each data set and the representative feature amount (matrix) corresponding to each label is calculated. Then, the similarity and the label L1 of the invoice image are input to the loss function calculation unit 240 to calculate the value of the loss function, and the extraction unit 220 and each label are back-propagated by error so that the value becomes smaller. Update the value of the representative feature amount of.

Supplementally, the "representative feature amount" indicates the feature amount corresponding to a plurality of preset labels. Specifically, if the features for each data set are represented in d-dimensional and there are N labels, the representative features are represented by an N × d-dimensional matrix. Therefore, in the example shown in FIG. 13, the similarity calculation unit 230 calculates N similarities for each label with respect to the feature amount C1. Then, in the loss function calculation unit 240, the larger the similarity corresponding to the label L1, the smaller the value, and the smaller the similarity corresponding to other than the label L1, the smaller the value of the N similarity. Output as follows. The initial value of the representative feature amount is determined by a random number.

In other words, the extraction unit 220 described above includes a first label (L1) of one invoice image G1, a feature amount C1 obtained from the invoice image G1, and a plurality of preset second labels (L1 to LN). ), The similarity is large when the first label (L1) and the second label (L1 to LN) are the same label (L1) according to the input of the similarity of the representative feature amount including the feature amount corresponding to). When the value is smaller and the first label and the second label are different labels (L2 to LN), the smaller the similarity is, the smaller the value is.

As described above, according to the present embodiment, with respect to the feature amount of the target invoice image, the billing information associated with the past feature amount similar to the feature amount and the characters read from the target invoice image. Use information. Thereby, the billing information desired by the user can be easily generated from the new invoice image.

Specifically, if the form information generation device 100 according to the present embodiment is used, the billing information 22 includes at least "item", "expense item", "expense generation department", "amount", "billing date", "payment date", and "billing source". If it is associated and stored, "item", "expense item", "expense generation department", and "invoice source" can be extracted based on the past invoice image, and "invoice date" and "payment deadline" can be read by OCR. , The user can easily generate the billing information 22 required for the accounting system. In particular, when items other than the "amount" do not fluctuate so much, by using the form information generation device 100 according to the present embodiment, the labor required for the user to generate the billing information required for the accounting processing system is greatly increased. Can be reduced to.

The following can be considered as a modification of this embodiment. First, in addition to invoices, purchase orders, quotations, receipts, insurance policies, etc. can be considered as forms.

Further, instead of the reading condition 113 being rule-based, character information of a predetermined item may be read by machine learning. Further, as machine learning, instead of deep distance learning, although the accuracy is a little inferior, learning by category classification, self-supervised learning, transfer learning of a model learned from Imagenet, etc. may be used.

Further, the form information generation device 100 and the learning device 200 may be able to communicate with the server. In that case, a part or all of the data described as being stored in the storage unit 110 of the form information generation device 100 and / or stored in the storage unit 210 of the learning device 200 in the server. A part or all of the data described as may be stored. In this case, the transmission / reception unit in which the form information generation device 100 communicates with the server has the function of the acquisition unit for acquiring various information. Instead of this, the form information generation device 100 and the learning device 200 may be able to communicate with each other, and one of them may have the function of the server. Also in this case, the transmission / reception unit in which the form information generation device 100 communicates with the learning device 200 has the function of the acquisition unit for acquiring various information.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, wherein the block is (1) a stage of the process in which the operation is performed or (2) a device having a role of performing the operation. May represent a section of. Specific stages and sections are implemented by dedicated circuits, programmable circuits supplied with computer-readable instructions stored on computer-readable media, and / or processors supplied with computer-readable instructions stored on computer-readable media. It's okay. Dedicated circuits may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. Programmable circuits are memory elements such as logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLA), etc. May include reconfigurable hardware circuits, including, etc.

The computer readable medium may include any tangible device capable of storing instructions executed by the appropriate device, so that the computer readable medium having the instructions stored therein is specified in a flow chart or block diagram. It will be equipped with a product that contains instructions that can be executed to create means for performing the operation. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer readable media include floppy (registered trademark) disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), Electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD), Blu-ray (RTM) disk, memory stick, integrated A circuit card or the like may be included.

Computer-readable instructions are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming such as Smalltalk, JAVA®, C ++, etc. Includes either source code or object code written in any combination of one or more programming languages, including languages, and traditional procedural programming languages such as the "C" programming language or similar programming languages. good.

Computer-readable instructions are used locally or to a local area network (LAN), wide area network (WAN) such as the Internet, to a general purpose computer, a special purpose computer, or the processor or programmable circuit of another programmable data processing device. ) May execute computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

FIG. 14 shows an example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part. The program installed on the computer 1200 can cause the computer 1200 to function as an operation associated with the device according to an embodiment of the invention or as one or more sections of the device, or the operation or the one or more. The section can be run and / or the computer 1200 can be run the process according to an embodiment of the invention or a stage of the process. Such a program may be run by the CPU 1212 to cause the computer 1200 to perform certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, a graphic controller 1216, and a display device 1218, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a hard disk drive 1224, a DVD-ROM drive 1226, and an IC card drive, which are connected to the host controller 1210 via the input / output controller 1220. There is. The computer also includes legacy input / output units such as the ROM 1230 and keyboard 1242, which are connected to the input / output controller 1220 via the input / output chip 1240.

The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires the image data generated by the CPU 1212 in a frame buffer or the like provided in the RAM 1214 or itself so that the image data is displayed on the display device 1218.

The communication interface 1222 communicates with other electronic devices via the network. The hard disk drive 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD-ROM drive 1226 reads the program or data from the DVD-ROM 1201 and provides the program or data to the hard disk drive 1224 via the RAM 1214. The IC card drive reads programs and data from the IC card and / or writes programs and data to the IC card.

The ROM 1230 stores in it a boot program or the like executed by the computer 1200 at the time of activation and / or a program depending on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a parallel port, serial port, keyboard port, mouse port, and the like.

The program is provided by a computer readable medium such as a DVD-ROM 1201 or an IC card. The program is read from a computer readable medium, installed in a hard disk drive 1224, RAM 1214, or ROM 1230, which is also an example of a computer readable medium, and executed by the CPU 1212. The information processing described in these programs is read by the computer 1200 and provides a link between the program and the various types of hardware resources described above. The device or method may be configured to implement the manipulation or processing of information in accordance with the use of the computer 1200.

For example, when communication is executed between the computer 1200 and an external device, the CPU 1212 executes a communication program loaded in the RAM 1214, and performs communication processing with respect to the communication interface 1222 based on the processing described in the communication program. You may order. Under the control of the CPU 1212, the communication interface 1222 reads and reads transmission data stored in a transmission buffer processing area provided in a recording medium such as a RAM 1214, a hard disk drive 1224, a DVD-ROM 1201, or an IC card. The data is transmitted to the network, or the received data received from the network is written to the reception buffer processing area provided on the recording medium.

Further, the CPU 1212 makes the RAM 1214 read all or necessary parts of a file or a database stored in an external recording medium such as a hard disk drive 1224, a DVD-ROM drive 1226 (DVD-ROM1201), an IC card, or the like. Various types of processing may be performed on the data on the RAM 1214. The CPU 1212 then writes back the processed data to an external recording medium.

Various types of information such as various types of programs, data, tables, and databases may be stored in recording media and processed. The CPU 1212 describes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information retrieval described in various parts of the present disclosure with respect to the data read from the RAM 1214. Various types of processing may be performed, including / replacement, etc., and the results are written back to the RAM 1214. Further, the CPU 1212 may search for information in a file, database, or the like in the recording medium. For example, when a plurality of entries each having an attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. Search for an entry that matches the condition from the plurality of entries, read the attribute value of the second attribute stored in the entry, and associate it with the first attribute that satisfies the predetermined condition. The attribute value of the second attribute obtained may be acquired.

The program or software module described above may be stored on a computer 1200 or on a computer readable medium near the computer 1200. Also, a recording medium such as a hard disk or RAM provided in a dedicated communication network or a server system connected to the Internet can be used as a computer readable medium, thereby providing the program to the computer 1200 over the network. do.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that the form with such changes or improvements may be included in the technical scope of the present invention.

The order of execution of each process such as operation, procedure, step, and step in the apparatus, system, program, and method shown in the claims, specification, and drawings is particularly "before" and "prior to". It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. It's not a thing.

10 invoice image, 12 invoice image, 20 invoice information, 22 invoice information, 30 features, 40 labels, 100 form information generator, 110 storage unit, 111 learned extraction unit, 112 selection data, 113 reading conditions, 114 correspondence, 120 extraction unit, 130 calculation unit, 140 selection unit, 150 reading unit, 160 generation unit, 200 learning device, 210 storage unit, 211 learning data, 212 learning parameter, 220 extraction unit, 230 similarity calculation. Part, 240 Loss function calculation part

Claims (14)

An acquisition unit that acquires the past feature amount extracted from the past form image in association with the past form information based on the character information described in the past form image, and the acquisition unit.
An extraction unit that extracts the target feature amount from the target form image,
A calculation unit that calculates the degree of similarity between the target feature amount and the past feature amount,
A selection unit that selects past form information based on the similarity, and
A reading unit that reads character information of predetermined items from the target form image, and
A generation unit that generates and displays new form information associated with the target form image from the past form information selected by the selection unit and the character information read by the reading unit. A form information generator equipped with and. The form information generation device according to claim 1, wherein the extraction unit corresponds to the one obtained by deep distance learning. The extraction unit responds to the input of labels and similarity of a set of form images. When the labels are the same, the larger the similarity, the smaller the value, and when the labels are different, the smaller the similarity, the smaller the value. The form information generation device according to claim 1 or 2, which corresponds to the one obtained by learning using. The past feature amount and the extraction unit input the similarity between the first label of one form image, the feature amount obtained from the form image, and the feature amount corresponding to a plurality of preset second labels. When the first label and the second label are the same label, the larger the similarity is, the smaller the value is, and when the first label and the second label are different labels, the smaller the similarity is, the smaller the value is. The form information generation device according to claim 1 or 2, which corresponds to the one obtained by learning using a function. Any one of claims 1 to 4 in which the generation unit generates the new form information by rewriting the character information of the selected past form information with the read character information. The form information generator described in the section. The form information generation device according to claim 5, wherein the generation unit uses the character information of the selected past form information for the new form information with respect to the information of the characters that have not been rewritten. The form information generation device according to any one of claims 1 to 6, wherein the generation unit distinguishably displays a portion of the new form information that is the same as or different from the past form information. The form information generation device according to any one of claims 1 to 7, wherein the generation unit displays the new form information according to the accuracy of reading the character information read by the reading unit. The form information generation device according to any one of claims 1 to 8, wherein the selection unit warns when there is no past feature amount having a similarity equal to or higher than a predetermined threshold value. A device for generating billing information to be read by the accounting system.
The billing information and the invoice image corresponding to the billing information include the first information and the second information, respectively.
An acquisition unit that stores past invoice information in association with past invoice images,
A first information acquisition unit that selects a past invoice image similar to the new invoice image by referring to the acquisition unit and extracts the first information from the selected past invoice information.
A second information acquisition unit that reads the second information from the new invoice image with an optical character recognition device (OCR), and
A generation unit that generates new billing information from the first information extracted by the first information acquisition unit and the second information read by the second information acquisition unit.
A device equipped with. A step of associating a past feature amount extracted from a past form image with a past form information based on the character information described in the past form image, and a step of acquiring the past form information.
Steps to extract the target feature amount from the target form image,
A step of calculating the degree of similarity between the target feature amount and the past feature amount,
The step of selecting the past form information based on the similarity, and
The step of reading the character information of the predetermined item from the target form image, and
From the past form information selected in the selection step and the character information read in the reading step, new form information associated with the target form image is generated and displayed. How to prepare with steps. A program that realizes the method according to claim 11 on a computer. It is a method for generating billing information to be read by the accounting system.
The billing information and the invoice image corresponding to the billing information include the first information and the second information, respectively.
Steps to obtain past invoice information in association with past invoice images,
A step of selecting a past invoice image similar to the new invoice image by referring to the acquired past invoice image and extracting the first information from the selected past invoice information.
A step of reading the second information from the new invoice image with an optical character recognition device (OCR), and
A step of generating new billing information from the first information and the second information,
How to prepare. A program that realizes the method according to claim 13 on a computer.

Images