WO2022145343A1

WO2022145343A1 - Architecture for digitalizing documents using multi-model deep learning, and document image processing program

Info

Publication number: WO2022145343A1
Application number: PCT/JP2021/047935
Authority: WO
Inventors: ホサインシャハリアルシェイク
Original assignee: 有限責任監査法人トーマツ
Priority date: 2020-12-28
Filing date: 2021-12-23
Publication date: 2022-07-07
Also published as: JP2022169754A; JP7150809B2; AU2021412659A9; JP2022104411A; AU2021412659A1

Abstract

The purpose of the present invention is to convert a character string contained in a document image to text data by a method different from conventional optical character recognition. In the present invention, an electronic document generation device is provided with: a document image acquisition unit that acquires a document image obtained by imaging a document; a character string recognition unit that recognizes a character string contained in the document image acquired by the document image acquisition unit using a character string learning model that has learned correspondence between document images and character strings contained in the document images and outputs text data of the recognized character string; and an output unit that outputs the text data as text for an electronic medium.

Description

Document digitization architecture by multi-model deep learning, document image processing program

The present invention relates to an electronic document generator, an electronic document generation method, and an electronic document generation program, and more particularly to an electronic document generator that scans a paper document to generate an electronic document, an electronic document generation method, and an electronic document generation program. Is.

Although digital information technology has advanced and paperless offices have become widespread, the accumulation and transmission of information in paper documents is still widely used. Companies that have a huge amount of paper documents want a technology that can efficiently convert paper documents into digital documents.

In the conventional OCR text recognition technology, character recognition is performed in character units, so there is a problem that the recognition efficiency of character recognition is not good (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2010-244372

Therefore, the object of the electronic document generator, the electronic document generation method, and the electronic document generation program of the present disclosure is to convert a character string included in a document image into text data by a method different from the conventional optical character recognition. And.

That is, the electronic document generator according to the first aspect is a character string that learns the correspondence between the document image acquisition unit that acquires the document image obtained by imaging the document and the character string included in the document image. Using the learning model, the character string recognition unit that recognizes the character string included in the document image acquired by the document image acquisition unit and generates the text data related to the character string, and the text data as the text of the electronic medium. It has an output unit to output.

In the second aspect, in the electronic document generator according to the first aspect, a layout learning model in which a correspondence relationship between a plurality of elements included in a document image and identification information of each of the plurality of elements is learned is used. , Specify the range in each document image of the plurality of elements included in the document image acquired by the document image acquisition unit, recognize each type of the plurality of elements, and the document relating to each range of the plurality of elements. A layout recognition unit for acquiring position information in an image is further provided, and the character string recognition unit recognizes a character string included in a range specified by the layout recognition unit using a character string learning model and converts it into a character string. The text data may be generated, and the output unit may output each of the text data related to the plurality of elements as text in an electronic medium to each position information in the range related to the plurality of elements.

The third aspect is that in the electronic document generator according to the second aspect, the type of the element may be any of a character string, a table, an image, a seal, or a handwriting.

In the fourth aspect, in the electronic document generator according to the third aspect, in the element whose type recognized by the layout recognition unit corresponds to the table, each cell in the table included in the element is cut out and the cell is cut out. The character string recognition unit further includes a cutout unit for acquiring position information in each document image of the above, and the character string recognition unit uses a character string learning model for a character string included in each of the cells cut out in the cutout unit. It may be recognized and the text data related to the character string may be generated.

The fifth aspect is the document image including a plurality of elements in the electronic document generator according to the second to fourth aspects, and the element is given an annotation associated with the type corresponding to each of the elements. It also has a layout learning data generation unit that accumulates multiple document images with annotations and generates layout learning data, and the layout learning data is used for supervised learning of the layout learning model. May be good.

A sixth aspect is that in the electronic document generator according to the fifth aspect, position information in each document image of a range related to a plurality of elements included in the document image is added to the document image together with annotations. May be good.

A seventh aspect is a document of each type of a plurality of elements recognized by the layout recognition unit and a range of each of the plurality of elements based on the input in the electronic document generator according to the fifth or sixth aspect. At least one of the position information in the image may be modified, and a layout learning data correction unit for updating the layout learning data by adding the modified data may be further provided.

The eighth aspect further includes a layout learning unit that relearns the layout learning model using the layout learning data updated by the layout learning data correction unit in the electronic document generation device according to the seventh aspect. It may be that.

A ninth aspect is that the electronic document generator according to the second to eighth aspects further includes a character string learning data generation unit that generates character string learning data used for supervised learning of a character string learning model. May be good.

In the tenth aspect, in the electronic document generator according to the ninth aspect, the text data generated by the character string recognition unit is modified based on the input, and the modified text data is added to obtain the character string. A character string learning data correction unit for updating the learning data may be further provided.

The eleventh aspect is the character string learning in which the character string learning model is relearned by using the character string learning data updated by the character string learning data correction unit in the electronic document generator according to the tenth aspect. It may be provided further.

A twelfth aspect is the electronic document generator according to the second to eleventh aspects, wherein the character string recognition unit includes a plurality of character string learning models and is adapted to the language of the character string included in each of the plurality of elements. A character string learning model may be used.

A thirteenth aspect further comprises a preprocessing unit that performs preprocessing on a document image acquired by a document image acquisition unit in the electronic document generation device according to the second to twelfth aspects, and the preprocessing unit is a background removing unit. A tilt correction unit and a shape adjustment unit are provided, the background removal unit removes the background of the document image acquired by the document image acquisition unit, and the tilt correction unit corrects the inclination of the document image acquired by the document image acquisition unit. The shape adjusting unit may adjust the overall shape and size of the document image acquired by the document image acquisition unit.

A fourteenth aspect is the electronic document generator according to the second to thirteenth aspects, wherein the layout learning model includes a layout learning model for a contract, a layout learning model for an invoice, a layout learning model for a memorandum, and a delivery note. It may be either a layout learning model for a receipt or a layout learning model for a receipt.

In the electronic document generation method according to the fifteenth aspect, the computer used in the electronic document generation device includes a document image acquisition step of acquiring a document image in which a document is imaged, a document image, and a character string included in the document image. Using the character string learning model that learned the correspondence between, the character string recognition step that recognizes the character string included in the document image acquired in the document image acquisition step and generates the text data related to the character string. , An output step that outputs the text data as text on an electronic medium.

The electronic document generation program according to the sixteenth aspect includes a document image acquisition function for acquiring a document image in which a document is imaged, and a document image and a character string included in the document image on a computer used in the electronic document generation device. With the character string recognition function that recognizes the character string included in the document image acquired by the document image acquisition function and generates the text data related to the character string using the character string learning model that learned the correspondence between , The output function that outputs text data as text on an electronic medium is demonstrated.

The electronic document generator according to the present disclosure uses a document image acquisition unit that acquires a document image that is an image of a document, and a character string learning model that learns the correspondence between the document image and the character string included in the document image. A character string recognition unit that recognizes the character string included in the document image acquired by the document image acquisition unit and generates text data related to the character string, and an output unit that outputs the text data as text on an electronic medium. Since the character string included in the document image is recognized as characters using the machine-learned model, it is possible to improve the recognition efficiency of character recognition when converting the document image into text data.

It is a figure which shows the outline of the electronic document generation system including the electronic document generation apparatus which concerns on this embodiment. It is a block diagram which shows the physical structure of an electronic document generator. It is a figure which shows the outline of the process performed by an electronic document generator. It is a block diagram which shows the functional structure of an electronic document generator. It is a figure explaining the input data and output data of an electronic document generator. It is a figure explaining the background removal performed in the preprocessing. It is a figure explaining the tilt correction performed in the pre-processing. It is a figure explaining the shape adjustment performed in the pre-processing. It is a figure explaining the correction process of the omission elimination performed in the layout recognition process. It is a figure explaining the correction process of the overlap elimination performed in the layout recognition process. It is a figure explaining layout recognition performed in layout recognition processing. It is a figure explaining the recognition of the table performed in the layout recognition process. It is a figure explaining the cutout of a cell image. It is a figure explaining the character string in a cell image. It is a figure explaining the arrangement of the text data performed in the character string recognition process. It is a figure explaining the noise reduction performed in the character string recognition process. It is a figure which shows the example of the layout learning data which annotated. It is a figure which shows the example of the layout learning data which annotated. It is a figure which shows the example of the layout learning data which annotated. It is a figure which shows the example of the layout learning data which annotated. It is a figure which shows the example of the layout learning data which annotated. It is a figure which shows the example of the layout learning data which annotated. It is a figure which shows the example of the layout learning data which annotated. It is a figure which shows the example of the character string learning data which annotated. It is a flowchart of an electronic document generation program. It is a flowchart (1/3) of one Embodiment which concerns on an electronic document generation program. It is a flowchart (2/3) of one Embodiment which concerns on an electronic document generation program. It is a flowchart (3/3) of one Embodiment which concerns on an electronic document generation program.

An embodiment of the electronic document generator 10 according to the present disclosure will be described with reference to FIGS. 1 to 24. In this embodiment, an example is shown in which the electronic document generator 10 is used by connecting to the Internet and an information communication network 11 such as a LAN (Local Area Network). The outline of the electronic document generation system 100 including the electronic document generation device 10 will be described with reference to FIG. 1. FIG. 1 is a diagram showing an outline of an electronic document generation system 100 including an electronic document generation device 10.

The electronic document generation system 100 includes an electronic document generation device 10, a user terminal 12, a character string learning model 13, a layout learning model 14, a document image database 15, and the like. The electronic document generator 10, the user terminal 12, the character string learning model 13, the layout learning model 14, and the document image database 15 are connected to the information communication network 11, and each of them can communicate with each other.

The electronic document generation system 100 uses the electronic document generation device 10 to recognize character strings included in a document image and generate text data. The electronic document generation device 10 recognizes the layout of the document image by using the layout learning model, and recognizes the character string included in the document image by using the character string learning model.

The electronic document generator 10 is a kind of computer represented by, for example, a personal computer and is an information processing device. The electronic document generation device 10 also includes arithmetic processing devices and microcomputers included in various computers, and also includes devices and devices capable of realizing the functions according to the present disclosure by an application.

The character string learning model 13 is a learning model that recognizes an image of a character string included in a document image, and is used for character recognition of the electronic document generation device 10. The storage location of the character string learning model 13 is arbitrary as long as it can be used by the electronic document generation device 10 via the information communication network 11, and is stored in an information processing device such as a personal computer, a server device, or a database. For convenience of the description of the present embodiment, the character string learning model 13 represents an information processing device in which the character string learning model 13 is stored.

The character string learning model 13 may be configured by an existing learning model, or may be independently configured as a learning model suitable for use of the electronic document generation device 10. The character string learning model 13 is provided with learning models suitable for various languages such as Japanese, English, and Chinese, and in FIG. 1, the first character string learning model, the second character string learning model, and the third character string are provided. It shall be described as a learning model.

The character string learning model 13 is not limited to the one connected to the information communication network 11, but may be included in the electronic document generation device 10 and used under the direct control of the device 10. Further, the character string learning model 13 may be distributed and stored in a plurality of information processing devices connected to the information communication network 11.

The layout learning model 14 learns the correspondence between a plurality of elements included in the document image and the identification information of each of the plurality of elements based on the layout learning data described later, and recognizes the layout of the document image. It is a model and is used for layout recognition of the electronic document generator 10. Similar to the character string learning model 13, the layout learning model 14 has an arbitrary storage location as long as it can be used by the electronic document generator 10 via the information communication network 11, and the information connected to the information communication network 11 is available. It is stored in the processing device. For convenience of explanation of the present embodiment, the layout learning model 14 represents an information processing device in which the layout learning model 14 is stored.

The layout learning model 14 includes a layout learning model for contracts, a layout learning model for invoices, a layout learning model for memorandums, a layout learning model for delivery notes, a layout learning model for receipts, and the like.

The layout learning model for contracts is a learning model that recognizes the layout of document images of contracts, and learns using layout learning data for contracts. The layout learning model for contracts learns what kind of information is in what position in the contract, and in particular, it is described in bullet points, often without a table, and there is a handwritten signature column. Learn about the layout unique to books. The layout learning data for the contract is generated based on, for example, 200 types of contract forms, and at least 3 or 4 contract document images per form, to which the annotation described later is added. ..

The layout learning model for invoices is a learning model that recognizes the layout of document images of invoices, and learns using layout learning data for invoices. The layout learning model for invoices learns what information is in what position on the invoice, and in particular, the table often occupies a large area, and even if it is written in Japanese, it is alphanumerical. Learn about invoice-specific layouts, such as not a few words. The layout learning data for invoices is generated based on, for example, 200 types of invoice forms, and at least 3 or 4 invoice document images per form, to which the annotation described later is added. ..

The layout learning model for the memorandum is a learning model that recognizes the layout of the document image of the memorandum, and learns using the layout learning data of the memorandum. The layout learning model for the memorandum learns what kind of information is in which position of the memorandum, and in particular, it learns the layout peculiar to the memorandum such as the fact that there is often no table and there is a handwritten signature line. The layout learning data for the memorandum is generated based on, for example, 200 types of memorandum forms to which the annotation described later is added, and at least three or four memorandum document images per form.

The layout learning model for the delivery note is a learning model that recognizes the layout of the document image of the delivery note, and learns using the layout learning data for the delivery note. The layout learning model for the delivery note learns what kind of information is in which position on the delivery note, and in particular, the range occupied by the table is often large, and the product name and product number are often described. Learn about the layout peculiar to the delivery note. The layout learning data for the delivery note is generated based on, for example, 200 types of delivery note forms with the annotation described later, and at least 3 or 4 delivery note document images per form. ..

The layout learning model for receipts is a learning model that recognizes the layout of document images of receipts, and learns using layout learning data for receipts. The layout learning model for receipts learns what kind of information is in what position on the receipt, and in particular, it often contains a handwritten column of the amount or a table with the amount. Learn about the layout specific to receipts. The layout learning data for receipts is generated based on, for example, 200 types of receipt forms and at least 3 or 4 receipt document images per form, which are annotated as described below. ..

The layout learning model 14 is not limited to the use of the electronic document generation device 10 via the information communication network 11, but may be included in the electronic document generation device 10. Further, the layout learning model 14 may be distributed and stored in a plurality of information processing devices connected to the information communication network 11.

The document image database 15 is a database that stores images of documents. The electronic document generation device 10 acquires the document image stored in the document image database 15 and generates the character string learning data used for learning the character string learning model and the layout learning data used for learning the layout learning model. ..

The user terminal 12 is used for operating the electronic document generator 10. When there is an erroneously recognized character in the electronic document generated by the electronic document generator 10, or when the layout of the electronic document generated by the electronic document generator 10 is erroneously recognized. The electronic document is modified according to the modification input from the user of the user terminal 12, and the electronic document generator 10 accepts the modification and relearns at least one of the character string learning model 13 and the layout learning model 14. ..

Next, with reference to FIG. 2, the mechanical configuration of the electronic document generator 10 will be described. FIG. 2 is a block diagram showing a mechanical configuration of the electronic document generator 10. The electronic document generator 10 includes an input / output interface 20, a communication interface 21, a Read Only Memory (ROM) 22, a Random Access Memory (RAM) 23, a storage unit 24, a Central Processing Unit (CPU) 25, and a Graphics processing unit (GPU). It has 28 mag.

The input / output interface 20 sends / receives data or the like to / from an external device of the electronic document generation device 10. The external device is an input device 26 and an output device 27 that input / output data or the like to the electronic document generation device 10. The input device 26 is a keyboard, a mouse, a scanner, and the like, and the output device 27 is a monitor, a printer, a speaker, and the like.

The communication interface 21 has a function of inputting / outputting data of the electronic document generation device 10 when communicating with the outside via the information communication network 11.
The storage unit 24 can be used as a storage device, and various applications required for the electronic document generation device 10 to operate, various data used by the applications, and the like are recorded. The GPU 28 is suitable for a lot of repetitive operations performed in executing machine learning and the like, and is used together with the CPU 25.

The electronic document generation device 10 stores the electronic document generation program described later in the ROM 22 or the storage unit 24, and takes the electronic document generation program into the main memory composed of the RAM 23 and the like. The CPU 25 accesses the main memory in which the electronic document generation program is incorporated and executes the electronic document generation program.

Next, with reference to FIG. 3, the outline of the processing performed by the electronic document generator 10 will be described. FIG. 3 is a diagram showing an outline of processing performed by the electronic document generator 10.
The electronic document generator 10 performs the following processes I to III in this order.

Process I performs preprocessing 55 including "background removal", "tilt correction", and "shape adjustment" of the document image.
The pre-processing 55 refers to performing pre-processing for facilitating the execution (recognition) of character recognition using a learning model for an image including a character string, and is the recognition processing performed in processes II and III. The purpose is to improve recognition accuracy.

In the process II, the layout recognition process 56 is performed.
In the layout recognition process 56, first, "layout recognition" of the document image is performed. The layout recognition process 56 is a process of recognizing what kind of information is present at which position in the input image.

Information here refers to character strings, tables, images, seals, handwriting, etc. When the electronic document generator 10 recognizes the layout of the document image and the document image contains a table, the electronic document generator 10 performs "table recognition" and "cuts out the cell image" for the cells included in the table. ..

In the process III, the character string recognition process 57 is performed.
The character string recognition process 57 is a process of converting an image including a character string into text data by using a character string learning model 13 that has learned the correspondence between the image and the character string included in the image. The character string recognition process 57 may include processes such as “arrangement of text data” and “noise reduction”.

In the character string recognition process 57, the image of the character string is converted into text data, and "text data arrangement" and "noise removal" are performed.
"Arrangement of text data" means that when the image of the cut out character string contains a space, the space is recognized together with the character string, so that the text data is arranged together with the space.

"Noise removal" means that when noise is contained in the image of the cut out character string, the noise is passively removed from the text data because it is not recognized by the electronic document generator 10. The noise referred to here refers to pixels that do not form characters and are included in the image of the cut out character string.

Next, the functional configuration of the electronic document generator 10 will be described with reference to FIG. FIG. 4 is a block diagram showing a functional configuration of the electronic document generator 10. By executing an electronic document generation program described later, the electronic document generation device 10 has a document image acquisition unit 31, a preprocessing unit 32, a background removal unit 32a, an inclination correction unit 32b, a shape adjustment unit 32c, and a layout recognition on the CPU 25. Unit 33, cutting unit 34, character string recognition unit 35, output unit 36, layout learning data generation unit 40, layout learning data correction unit 41, layout learning unit 42, character string learning data generation unit 43, character string. It includes a learning data correction unit 44, a character string learning unit 45, and the like.

The document image acquisition unit 31 (see FIG. 4) acquires a document image in which a document is imaged.
The document image acquisition unit 31 may acquire a document image from the document image database 15. Alternatively, the document image acquisition unit 31 may obtain a document image from the scanner of the input device 26.

With reference to FIG. 5, the document image acquired by the document image acquisition unit 31 and the electronic document output from the electronic document generation device 10 will be described.
FIG. 5 is a diagram illustrating input data and output data of the electronic document generation device 10, and FIG. 5A shows a document image acquired by the document image acquisition unit 31 as input data. The document image contains noise such as a stapler mark 50, a handwriting 51, a seal 52, and an image 53.

These noises interfere with or become unnecessary for information processing devices such as humans and personal computers to understand the contents of the document. Other examples of noise include holes made for filing and creases left on the paper. The creases can be perceived as lines and need to be removed so that they are not reflected in the electronic document.

The electronic document generation device 10 converts the character string in the document image into text data and outputs the electronic document while maintaining the layout of the acquired document image (see FIG. 5 (b)). The electronic document generator 10 removes noise 54 by active processing of the hochiki mark 50, the handwriting 51, the seal 52, and the image 53 recognized as noise, and other pixels in the document image not recognized as character strings and noise. Is removed by passive processing that does not remain in the electronic document.

The table in the document image of FIG. 5B is output together with the text data as object data in the electronic document while maintaining the arrangement in the document image. The electronic document generator 10 can arbitrarily select the elements to be included in the electronic document to be output.

For example, the stapler mark 50, the handwritten 51, the seal 52, the image 53, and the like are removed in normal use, but the seal 52 and the image 53 can be included in an electronic document and output as image data.

The preprocessing unit 32 (see FIG. 4) performs preprocessing 55 on the document image acquired by the document image acquisition unit 31.
The preprocessing 55 is performed in order to improve the recognition accuracy of image recognition using the learning model by the layout recognition unit 33 and the character string recognition unit 35, which will be described later.

The pretreatment unit 32 includes a background removal unit 32a, an inclination correction unit 32b, and a shape adjustment unit 32c.
The background removing unit 32a (see FIG. 4) removes the background of the document image acquired by the document image acquisition unit 31.

The processing performed by the background removing unit 32a will be described with reference to FIG. FIG. 6 is a diagram illustrating background removal performed in the pretreatment 55. FIG. 6A shows a document image 58a before the background is removed, and FIG. 6B shows a document image 58b after the background is removed.

The background removing unit 32a removes the background of the document image by changing the background color of the document image to white. Specifically, the background removing unit 32a detects the background color of the acquired document image and determines whether or not the background color is white. When it is determined that the background color is not white, the background removing unit 32a extracts information other than the background of the document image, makes the background color white, and then superimposes the extracted information.

According to the background removing unit 32a, by deleting the background, noise that causes a malfunction of image recognition by the layout recognition unit 33 and the character string recognition unit 35 can be removed, and the recognition accuracy can be improved.

The tilt correction unit 32b (see FIG. 4) corrects the tilt of the document image acquired by the document image acquisition unit 31.
The processing performed by the tilt correction unit 32b will be described with reference to FIG. 7. FIG. 7 is a diagram illustrating the inclination correction performed in the preprocessing 55. FIG. 7A shows the document image 59a before the tilt correction, and FIG. 7B shows the document image 59b after the tilt correction.

The tilt correction unit 32b corrects the tilt of the character string when there is a tilted character string in the document image, and makes the character string parallel or perpendicular to the writing direction. When the document image is written vertically, the tilt correction unit 32b corrects the tilted character string so as to be parallel to the vertical writing direction, and when the document image is written horizontally, the tilted character string is written horizontally. Correct so that it is parallel to the direction.

Specifically, the tilt correction unit 32b extracts the character string of the document image and determines whether or not there is a tilted character string in the extracted character string. When it is determined that there is a tilted character string in the extracted character string, the tilt correction unit 32b detects the tilt angle of the tilted character string with respect to the writing direction, and tilts the tilted character string with respect to the tilted character string. Rotation processing is performed so that the angle becomes zero.

According to the tilt correction unit 32b, the recognition accuracy of image recognition by the character string recognition unit 35 can be improved by correcting the tilt of the character string. Further, it is possible to reduce the layout recognition error by the layout recognition unit 33.

The shape adjusting unit 32c (see FIG. 4) adjusts the overall shape and size of the document image acquired by the document image acquisition unit 31.
The processing performed by the shape adjusting unit 32c will be described with reference to FIG. FIG. 8 is a diagram illustrating shape adjustment performed in the pretreatment. FIG. 8A shows a document image 60a before the shape adjustment, and FIG. 8B shows a document image 60b after the shape adjustment.

When the overall shape of the document image acquired by the document image acquisition unit 31 is different from the actual document, the shape adjustment unit 32c adjusts the overall shape of the document image based on the overall shape of the actual document. conduct. Specifically, when the overall aspect ratio of the document image acquired by the document image acquisition unit 31 is different from the overall aspect ratio of the actual document, the overall aspect ratio of the document image is the overall aspect ratio of the actual document. The shape adjusting unit 32c adjusts so as to be equal to the ratio.

Further, if the size of the document image acquired by the document image acquisition unit 31 is too large or too small, the subsequent processing may not be performed normally. Therefore, the shape adjustment unit 32c performs the subsequent processing. Adjusts the size of the document image acquired by the document image acquisition unit 31 so that

According to the shape adjustment unit 32c, by adjusting the shape and size of the document image acquired by the document image acquisition unit 31, the layout recognition unit 33 that is performed thereafter improves the recognition accuracy of the layout according to the actual document. Further, the recognition accuracy of image recognition by the character string recognition unit 35 can be improved.

The layout recognition unit 33 (see FIG. 4) uses a layout learning model 14 that has learned the correspondence between the plurality of elements included in the document image 61 and the identification information of each of the plurality of elements, and the document image acquisition unit 33. The range of each of the plurality of elements included in the document image 61 acquired in 31 is specified in the document image 61, each type of the plurality of elements is recognized, and the document image 61 relating to each range of the plurality of elements is recognized. Get the position information in.

The type of the element may be any of a character string 48, a table 49, an image 53, a seal 52, or a handwriting 51. The type of the element is not limited to this, and stapler marks 50, punch hole marks, breakage (tear) marks, carbon stains for copying, and the like may be used.

The type of element may be suitable for the type of document (for example, contract, invoice, memorandum, invoice, receipt, etc.). For example, if carbon for copying is attached to the back side of the receipt and the carbon is transferred to the front surface and becomes a stain, use the stain with the carbon for copying as the element type and actively stain with the carbon for copying. May be removed.

The layout learning model 14 is either a layout learning model for contracts, a layout learning model for invoices, a layout learning model for memorandums, a layout learning model for delivery notes, or a layout learning model for receipts. It may be that.

The types of elements may be classified into necessary and unnecessary according to the type of document.
In this case, if the recognized element corresponds to an unnecessary element among the plurality of elements included in the document image 61, the layout recognition unit 33 does not acquire the position information of the element and requires the recognized element. If it corresponds to, the position information of the element may be acquired. Alternatively, the layout recognition unit 33 may recognize only the necessary elements among the plurality of elements included in the document image 61 and acquire the position information of the elements.

After recognizing each type of the element and acquiring the position information of the document image related to each range of the element, the layout recognition unit 33 may overlap the elements or the elements may be too far apart from each other. , Correct the range of each of the elements and the acquired position information based on the actual document.

With reference to FIG. 9, an example of correction processing for eliminating the omission performed by the layout recognition unit 33 when a omission occurs in the recognition range recognized by the layout recognition unit 33 will be described. The omission means that a part of the range to be recognized as an element by the layout recognition unit 33 is not recognized, and a part of the range of the element is insufficient. 9A and 9B are diagrams for explaining the correction process for eliminating the omission in the layout recognition process, FIG. 9A shows a state before the correction, and FIG. 9B shows a state after the correction.

When the layout recognition unit 33 recognizes the image 70 of the character string included in the document image acquired by the document image acquisition unit 31 as a character string, the layout recognition unit 33 determines whether or not there is a gap in the recognition range, and determines whether or not the recognition range is missing. If there is, perform correction processing to add the missing part.

FIG. 9A shows how the layout recognition unit 33 recognizes the image 70 of the character string as a character string in the recognition range 72a. The recognition range 72a has a defect in the left end portion of the image 70 of the character string. The layout recognition unit 33 determines whether or not there is a black line within a predetermined range around the recognition range 72a, and if there is a black line, a correction is added to add the range 72b including the black line to the recognition range 72a. (See FIG. 9 (b)).

The determination of presence / absence performed by the layout recognition unit 33 is not limited to the black line, and whether or not a line having the same color as the character or a line having a preset color is within a predetermined range around the recognition range 72a. May be determined. This is because the main purpose of the correction process for eliminating omissions performed in the layout recognition process is to improve the recognition accuracy of the character recognition process performed thereafter.

According to the correction process, even if the range of the element recognized by the layout recognition unit 33 is missing, it can be corrected to the normal recognition range by adding the missing range, and it is included in the element. About the character string The character string recognition unit 35 can normally perform character recognition.

With reference to FIG. 10, an example of correction performed by the layout recognition unit 33 when the recognition range recognized by the layout recognition unit 33 overlaps with other elements will be described. 10A and 10B are diagrams for explaining the correction process for eliminating the overlap performed in the layout recognition process, FIG. 10A shows a state before the correction, and FIG. 10B shows a state after the correction.

When the layout recognition unit 33 recognizes the image 73 of the character string included in the document image acquired by the document image acquisition unit 31 as a character string, the recognition range 75a overlaps with another element (for example, Table 74). It is determined whether or not the overlap occurs, and if an overlap occurs, a correction process for eliminating the overlap is performed.

FIG. 10A shows how the layout recognition unit 33 recognizes the image 73 of the character string as a character string in the recognition range 75a. The recognition range 75a overlaps the table 74 to the right of the image 73 of the character string beyond a blank (space). The layout recognition unit 33 determines whether or not there is a blank (space) of a predetermined size inside the recognition range 75a, and if there is the blank (space), the blank (space) and the blank. The recognition range 75a related to the portion on the right side of (space) is deleted to make the recognition range 75b (see FIG. 10B).

Since there is always a blank (space) of a predetermined size between an element and another element, the layout recognition unit 33 recognizes the blank (space) of a predetermined size inside the recognition range. We conclude that the range overlaps with other elements. According to the overlap elimination correction process performed in the layout recognition process, the layout recognition unit 33 can improve the layout recognition accuracy.

The processing performed by the layout recognition unit 33 will be described with reference to FIG. 11A and 11B are diagrams for explaining the layout recognition performed in the layout recognition process 56, FIG. 11A shows the state of the document image 61 before the layout is recognized, and FIG. 11B shows the state of the document image 61 after the layout is recognized. The state of the document image 62 of the above is shown.

The layout recognition unit 33 specifies the range of the elements (character string 48, table 49, seal 52, image 53) included in the document image 61 within the document image 61 by image recognition using the layout learning model 14.

In FIG. 11B, for convenience of explanation, the range of the specified character string 48 is surrounded by a solid line, and the ranges of the specified table 49, the seal 52, and the image 53 are surrounded by a broken line. The boundaries of the elements need not be visible to humans as long as they can be recognized by the electronic document generator 10.

The layout recognition unit 33 recognizes the type of the corresponding element by image recognition using the layout learning model 14 in the range in the specified document image 61, and relates to the document image 62 in the range together with the type of the element. Get location information. The position information may be represented by plane orthogonal coordinates with a predetermined point in the document image 62 as the origin.

The layout learning model 14 is preset according to the type of the document image 61, and the layout recognition unit 33 recognizes the layout of the document image 61 using the preset layout learning model 14.

That is, if the document image 61 acquired by the document image acquisition unit 31 is a contract, image recognition is performed using the layout learning model 14 for the contract, and if it is an invoice, the layout learning model for the invoice. Image recognition is performed using 14, and if it is a memorandum, image recognition is performed using the layout learning model 14 for the memorandum, and if it is a delivery note, image recognition is performed using the layout learning model 14 for the delivery note. If it is a receipt, image recognition is performed using the layout learning model 14 for the receipt.

Since the layout recognition unit 33 properly uses the layout learning model 14 according to the type of the document image 61 acquired by the document image acquisition unit 31, the recognition accuracy of the layout recognition of the document image 61 can be improved.

The cutout unit 34 (see FIG. 4) cuts out each of the cells in the table included in the element in the element whose type recognized by the layout recognition unit 33 corresponds to the table, and in each document image of the cell. Get the location information in.

With reference to FIG. 12, the recognition of Table 49 by the layout recognition unit 33 will be described. 12A and 12B are diagrams for explaining table recognition performed by the layout recognition process 56, FIG. 12A shows Table 63 before being recognized by the layout recognition unit 33, and FIG. 12B shows the layout recognition unit 33. Table 64 after being recognized in. In FIG. 12B, for convenience of explanation, the line recognized as the vertical line 65 is represented as a one-dot chain line, and the line recognized as the horizontal line 66 is represented as a broken line.

The layout recognition unit 33 recognizes the length and position of each of the vertical lines 65 and the horizontal lines 66 constituting the table 64. The layout recognition unit 33 recognizes all the cells included in the table 64 by recognizing the lengths and positions of all the vertical lines 65 and the horizontal lines 66 constituting the table 64. That is, the layout recognition unit 33 recognizes a quadrangle composed of two adjacent vertical lines 65 and two adjacent horizontal lines 66 as cells.

Further, the layout recognition unit 33 also recognizes the line types of the lines constituting Table 64. The recognized line type is reflected in the line object constituting the table included in the electronic document when the electronic document is reproduced based on the acquired document image. Therefore, for example, when the table line in the document image 62 is a broken line, the table line included in the electronic document reproduced based on the document image 62 is represented as a broken line object.

The cutout unit 34 cuts out all the cells included in the table 64 grasped by the layout recognition unit 33 into an image for each cell alone.
With reference to FIG. 13, cutting out of cell pixels by the cutting-out portion 34 will be described. FIG. 13 is a diagram illustrating cutting out of a cell image. The cell 67 cut out by the cutout portion 34 may include a plurality of character strings.

The cutting unit 34 acquires the image of each cell and the position information of the cell in the table 64 for all the cells included in the table 64. The position information may be represented by plane orthogonal coordinates with a predetermined point in Table 64 as the origin, or may be represented by (rows, columns) in Table 64.

The cutout unit 34 reproduces all the vertical lines and horizontal lines constituting the table recognized by the layout recognition unit 33, and generates the position information of all the cells.

A cell 67 containing a plurality of character strings will be described with reference to FIG. FIG. 14 is a diagram illustrating a character string in the cell image.

When the cut out cell 67 contains a character string of a plurality of lines, the cutout unit 34 further cuts out an image for each character string for all the character strings. The cell 67 shown in FIG. 14 contains two lines of character strings, and the cutout portion 34 cuts out an image 67a of the character string and an image 67b of the character string.

The character string recognition unit 35 (see FIG. 4) uses a character string learning model 13 that has learned the correspondence between the document image and the character string included in the document image, and the document image acquired by the document image acquisition unit 31. Characters are recognized for the character string included in, and text data related to the character string is generated.

The character string recognition unit 35 may recognize the character string included in the range recognized by the layout recognition unit 33 using the character string learning model 13 and generate text data related to the character string.

The character string recognition unit 35 may perform character recognition using the character string learning model 13 for the character strings included in each of the cells cut out by the cutout unit 34, and generate text data related to the character strings. ..

The character string recognition unit 35 includes a plurality of character string learning models 13, and may use a character string learning model 13 adapted to the language of the character string included in each of the plurality of elements.
When the character string recognition unit 35 recognizes a document image written in English, the recognition accuracy can be improved by using a character string learning model suitable for recognizing an English character string.

The character recognition performed by the character string recognition unit 35 will be described with reference to FIGS. 15 and 16.
15A and 15B are diagrams for explaining the arrangement of text data performed in the character string recognition process 57, FIG. 15A is an image 67a of a character string before character recognition is performed, and FIG. 15B is a character. It is a character string 68a after recognition, that is, text data 68a.

16A and 16B are diagrams for explaining noise removal performed by the character string recognition process 57, FIG. 16A is an image 71a of a character string before character recognition is performed, and FIG. 16B is a character recognition. It is the character string 71b after it is performed, that is, the text data 71b.

The image 67a of the character string shown in FIG. 15A has a handwritten check mark in addition to the character string of one line. The character string contains a space between words. The character string recognition unit 35 recognizes the entire image 67a of the character string by using the character string learning model 13 and generates text data.

The character string recognition unit 35 recognizes characters for two words "L / C NO:", "ILC18H000219", and a blank space between the two words in the image 67a of the character string, and 2 The text data corresponding to the tokens and the text data corresponding to the blank space between the two tokens are generated (68a: see FIG. 15B).
Therefore, since the character string recognition unit 35 also recognizes the space between words and phrases and converts them into text data, the two words and phrases can be arranged separately as in the image 67a.

When the character string recognition unit 35 recognizes the character of the image 67a of the character string, the handwriting check mark is not recognized and is not included in the text data. Therefore, the handwriting check mark is deleted from the output electronic document (the handwriting check mark is deleted). 68a: FIG. 15 (b). Therefore, noise such as handwriting check marks that are not the target of character recognition by the character string recognition unit 35 is passively removed from the electronic document.

In the image 71a of the character string shown in FIG. 16A, a part of the seal overlapped with the character string on one line remains as noise. The character string recognition unit 35 recognizes the entire image 71a of the character string using the character string learning model 13 and generates text data.

The character string recognition unit 35 recognizes characters for the entire image 71a of the character string, and generates the text data corresponding to the character string for the character string "autiated to act on behalf of the" (71b: FIG. 16B). reference).

Since the noise contained in the image 71a of the character string is not the target of character recognition by the character string recognition unit 35, it is passively removed from the electronic document (71b: see FIG. 16B).

Although the text data of the character string after performing character recognition on the document image and the sentence image has been described with reference to FIGS. 15 and 16, the character string learning model 13 has FIGS. 15 (a) and 15 (b). ) And the data associated with FIG. 16 (a) and FIG. 16 (b) as teacher data, by learning a large number of characters from the image using such deep learning. Recognition can be realized.

The character string recognition unit 35 acquires attribute data such as the size and typeface of the character included in the character string when recognizing the character string included in the

images

67a and 71a by using the character string learning model 13. You may. The attribute data of this character is reflected as the attribute data of the text data output by the output unit 36 described later.

The output unit 36 (see FIG. 4) outputs text data as text on an electronic medium.
The output unit 36 may output each of the text data related to the plurality of elements as text on an electronic medium to each position information in the range related to the plurality of elements.

The electronic medium is not limited to data electronically stored in a recording medium, but also includes data itself that can be handled by an information processing device such as a personal computer, not in a state of being stored in the recording medium.
The position information of the element may be represented by plane orthogonal coordinates with a predetermined point in the document image 62 as the origin.

According to the output unit 36, since the text data related to the plurality of elements is output based on the position information related to the element, noise is removed while maintaining the layout of the acquired document image 61, and the inside of the document image 61 is used. It is possible to convert the character string of the above into text data and output an electronic document.

The output unit 36 may reflect the character attribute data acquired by the character string recognition unit 35 in the text data and output it to an electronic document. According to the output unit 36, the electronic document generator 10 can reproduce attribute data such as character size and typeface included in the document image 61 as attribute data of text data included in the electronic document to be output. can.

The layout learning data generation unit 40 (see FIG. 4) is a document image including a plurality of elements, and annotations associated with the types corresponding to each of the elements are attached to the elements, and the annotations are added. Data for layout learning is generated by accumulating a plurality of document images.

The layout learning data is used for supervised learning of the layout learning model 14.
The document image stored in the layout learning data may be given position information in each document image of the range related to the plurality of elements included in the document image together with the annotation.

The layout learning data to which annotations are added will be described with reference to FIGS. 17 to 23. 17 to 23 are diagrams showing an example of layout learning data to which annotations are added.

The layout learning data generation unit 40 acquires a document image from the document image database 15, annotates the document image, and generates layout learning data. When generating the layout learning data, the user can manually generate the layout learning data without using the layout learning data generation unit 40. When the user manually generates the layout learning data, the document image acquired from the document image database 15 can be annotated by using the user terminal 12.

With reference to FIGS. 17 and 18, layout learning data used for learning the layout learning model 14 for invoices will be described. Annotation symbols are added to each element so that the electronic document generator 10 can identify and classify character strings, tables, images, stamps, outer frames, and noises as elements included in the document image.

Annotation symbol 76 of the character string is given to the element related to the character string, the character string is surrounded by a rectangular frame line, and the tag of "Text" is attached to the frame line as a mark. The portion surrounded by the rectangular frame line is learned by the layout learning model 14 as the range occupied by the elements related to the character string in the document image.

Annotation symbol 77 of the table is given to the element related to the table, a rectangular frame line is superimposed on the outer frame of the table, and the tag of "Border Table" is attached to the frame line as a mark. The portion surrounded by the rectangular frame line is learned by the layout learning model 14 as the range occupied by the elements related to the table in the document image.

An annotation symbol 78 of the image is attached to the element related to the image, a frame line indicating the annotation symbol is superimposed on the boundary line of the image, and the tag of "Image" is attached to the frame as a mark. Images shall include logos, marks, photographs, illustrations and the like. The portion surrounded by the frame line is learned by the layout learning model 14 as the range occupied by the elements related to the image in the document image.

Annotation symbol 79 of the seal is given to the element related to the seal, a frame line indicating the annotation symbol is superimposed on the boundary line of the seal, and the tag of "Hun" is attached to the frame line as a mark. The portion covered with the frame line is learned by the layout learning model 14 as the range occupied by the element related to the seal in the document image.

Annotation symbol 80 of the outer frame is given to the element related to the outer frame, the frame line is superimposed on the boundary line of the outer frame, and the tag of "Border" is attached to the frame line as a mark. The layout learning model 14 learns about the length and position of the four line segments constituting the frame line.

A noise annotation symbol 81 is added to the element related to noise, the noise is surrounded by a rectangular frame, and the tag of "Noise" is attached to the frame as a mark. The portion covered with the frame is learned by the layout learning model 14 as the range occupied by the element related to the noise in the document image.

The layout learning data used for learning table recognition will be described with reference to FIG. For the table included in the document image acquired from the document image database 15, the alternate long and short dash line, which is the vertical line annotation symbol 83, is superimposed on all the vertical lines constituting the table, and the horizontal line annotation symbol is applied to all the horizontal lines constituting the table. Overlay the dashed line 84.

By recognizing all the alternate long and short dash lines and broken lines, the layout learning model 14 can learn about the size of the table, the range occupied by the table, the position, and the information of all the cells included in the table. The cell information is the number of cells contained in the table and the position of each cell in the table, and the position in the table is represented by (row, column) of the table.

With reference to FIGS. 20 and 21, layout learning data used for learning the recognition of character strings in table cells will be described. FIG. 20 is layout learning data in which a character string is included in each cell. FIG. 21 is layout learning data for recognizing a table relating to a cell containing a one-line character string, a cell containing a two-line character string, and a cell containing a three-line character string.

As shown in FIGS. 20 and 21, a character string annotation symbol 76 is added to each of the character strings without being affected by the number of rows of the character string contained in one cell, and the character string is bounded by a rectangular border. Enclose with and attach the tag of "Text" to the frame line as a mark.

The layout learning model 14 learns the range of the annotation symbol 76 of the character string and the position of the character string in the table. The electronic document generator 10 can reproduce the table by outputting the text data related to the character string to the electronic document together with the object data related to all the vertical lines and the horizontal lines constituting the table.

With reference to FIG. 22, layout learning data used for learning the recognition of character strings in table cells will be described. An annotation symbol 76 for the character string is added to each of the character strings included in the cells of the table of FIG. 22, the character string is surrounded by a rectangular frame line, and the tag "Text" is attached to the frame line as a mark.

The layout learning model 14 learns about the range of the annotation symbol 76 of the character string and the position information of the character string in the document. The electronic document generator 10 can reproduce a table in an electronic document by placing text data related to a character string at a position in the document. The electronic document generator 10 can reproduce the table in the electronic document only by outputting the text data without reproducing the vertical lines and the horizontal lines constituting the table in the electronic document.

With reference to FIG. 23, the layout learning model used for learning the recognition of the seal will be described. According to the layout learning model according to FIG. 23, the layout learning model 14 can learn the range and position of the seal by the element related to the character string and the blank located at the bottom of the character string without using the element related to the seal. can.

The layout learning data correction unit 41 (see FIG. 4) has the position information in the document image of each type of the plurality of elements acquired by the layout recognition unit 33 and the range of each of the plurality of elements based on the input. At least one of the above is modified, and the layout learning data is updated by adding this modified data.

There may be a discrepancy between the document image 61 before the image is recognized by the layout recognition unit 33 and the document image 62 after the image is recognized by the layout recognition unit 33. For example, a part of the character string may not be recognized, an element to be recognized as an image may be recognized as a seal, or the position of the table may be misaligned.

In such a case, the document image 62 after the image is recognized by the layout recognition unit 33 is modified so as to match the document image 61 before the image is recognized by the layout recognition unit 33, and the corrected data is used. By adding to the layout learning data, the layout learning data is updated.

The layout learning unit 42 (see FIG. 4) relearns the layout learning model 14 using the layout learning data updated by the layout learning data correction unit 41.
By re-learning the layout learning model 14, the recognition accuracy of the layout of the document image can be improved.

The character string learning data generation unit 43 (see FIG. 4) generates character string learning data used for supervised learning of the character string learning model 13.
The character string learning data correction unit 44 (see FIG. 4) corrects the text data generated by the character string recognition unit 35 based on the input, and adds the corrected text data for character string learning. Update the data.

The character string learning unit 45 (see FIG. 4) relearns the character string learning model 13 using the character string learning data updated by the character string learning data correction unit 44.

The character string learning data generation unit 43 acquires a document image from the document image database 15, annotates the document image, and generates character string learning data. When generating the character string learning data, the user can manually generate the character string learning data without using the character string learning data generation unit 43. When the user manually generates the character string learning data, the document image acquired from the document image database 15 can be annotated by using the user terminal 12.

With reference to FIG. 24, the character string learning data used for learning the character string learning model 13 will be described. FIG. 21 is a diagram showing an example of character string learning data to which annotations are added. FIG. 24 is an output screen of the character string learning data generation unit 43, which is displayed on the user terminal 12 or the output device 27 of the electronic document generation device 10.

The character string learning data generation unit 43 assigns text data corresponding to each of the character strings to the character strings included in the document image acquired from the document image database 15 as the comment 85 of the text data.

Note that the annotation may be added as the text data annotation 85 instead of the text data. When the character string included in the document image contains a blank, the character string learning data generation unit 43 generates the character string learning data so that the text data corresponding to the character string also contains a blank.

Next, with reference to FIG. 25, an electronic document generation method executed by the electronic document generation device 10 according to the present embodiment will be described together with an electronic document generation program. FIG. 25 is a flowchart of an electronic document generation program. The electronic document generation method is executed by the CPU 25 of the electronic document generation device 10 based on the electronic document generation program.

The electronic document generation program realizes various functions such as a document image acquisition function, a preprocessing function, a layout recognition function, a cutting function, a character recognition function, and an output function for the CPU 25 of the electronic document generation device 10. These functions are executed in the order shown in FIG. 25, but the order may be changed as appropriate. Since each function overlaps with the description of the electronic document generation device 10 described above, a detailed description thereof will be omitted.

The document image acquisition function acquires a document image obtained by converting a document into an image (S31: document image acquisition step).
The format of the document image includes, for example, PDF, JPG, GIF, and the like, and other data formats that the electronic document generator 10 can process as an image may be included.

The pre-processing function performs pre-processing on the document image acquired by the document image acquisition function (S32: pre-processing step).
The pre-processing function has a background removal function, a tilt correction function, and a shape adjustment function. The background removal function removes the background of the document image acquired by the document image acquisition function, and the tilt correction function removes the background of the document image acquired by the document image acquisition function. The tilt of the image is corrected, and the shape adjustment function adjusts the overall shape and size of the document image acquired by the document image acquisition function.

The layout recognition function uses a layout learning model 14 that learns the correspondence between a plurality of elements included in the document image and the identification information of each of the plurality of elements to obtain a document image acquired by the document image acquisition function. The range in each document image of the plurality of elements included is specified, each type of the plurality of elements is recognized, and the position information in the document image relating to each range of the plurality of elements is acquired (S33:). Layout recognition step).

The types of elements may be classified into necessary and unnecessary according to the type of document.
In this case, if the recognized element corresponds to an unnecessary element among a plurality of elements included in the document image acquired by the document image acquisition function, the layout recognition function does not acquire the position information of the element and recognizes it. If the specified element corresponds to a necessary one, the position information of the element may be acquired. Alternatively, the layout recognition function may recognize only the necessary elements among the plurality of elements included in the document image 61 and acquire the position information of the elements.

The layout recognition function recognizes each type of element, acquires the position information of the document image related to each range of the element, and then if the elements overlap or the elements are too far apart, the layout recognition function recognizes each type of the element. Based on the actual document, the range of each of the elements and the acquired position information are corrected.

The layout recognition function recognizes the length and position of each of the vertical and horizontal lines that make up the table. The layout recognition function grasps all the cells included in the table by grasping the lengths and positions of all the vertical lines and horizontal lines constituting the table. That is, the layout recognition function recognizes a quadrangle composed of two adjacent vertical lines and two adjacent horizontal lines as cells.

Furthermore, the layout recognition function also recognizes the line types of the lines that make up the table. The recognized line type is reflected in the line object constituting the table included in the electronic document when the electronic document is reproduced based on the acquired document image. Thus, for example, if the table line in the document image is a dashed line, the table line contained in the electronic document reproduced based on the document image is represented as a dashed object.

The cutout function cuts out each of the cells in the table included in the element in the element whose type recognized by the layout recognition function corresponds to the table, and acquires the position information in each document image of the cell ( S34: Cutting step).
The cutout function reproduces all the vertical and horizontal lines constituting the table recognized by the layout recognition function, and generates the position information of all the cells.

The cell cut out by the cutout function may contain multiple character strings. When the cut-out cell contains a multi-line character string, the cutout function further cuts out an image for each character string for all the character strings.

The image of the character string recognized by the layout recognition function and the image of the character string cut out by the cutout function are sent to the character recognition function line by line.

The character recognition function uses a character string learning model that learns the correspondence between the document image and the character string included in the document image, and recognizes the character string included in the document image acquired by the document image acquisition function. , Generates text data related to the character string (S35: character recognition step).

The output function outputs the text data as text on an electronic medium (S36: output step).
The output function outputs text data based on the position information of the character string acquired by the layout recognition function and the position information in the document image of the cell acquired by the cutout portion, and reproduces the text as text on an electronic medium.

Next, with respect to the above-mentioned electronic document generation program, an embodiment for converting a document image of a receipt into an electronic document will be described with reference to FIGS. 26 to 28. 26 to 28 are flowcharts of an embodiment relating to an electronic document generation program. The flowcharts shown in FIGS. 26 to 28 show a flowchart of one electronic document generation program by combining them.

In step S102, the document image acquisition unit 31 acquires a document image or PDF from the document image database 15.
In step S103, it is determined whether or not the data acquired by the document image acquisition unit 31 is PDF. If it is not a PDF (No: S103), that is, if the data acquired by the document image acquisition unit 31 is a document image, the process proceeds to step S106.

When the data acquired by the document image acquisition unit 31 is a PDF (Yes: S103), the process proceeds to step S104, the PDF is converted into a document image, and then the document image is acquired (S105).

In step S106, the preprocessing unit 32 performs preprocessing on the acquired document image. The pretreatment unit 32 includes a background removal unit 32a, an inclination correction unit 32b, and a shape adjustment unit 32c.

The background removing unit 32a removes the background of the acquired document image. When the character string included in the acquired document image is tilted, the tilt correction unit 32b corrects the tilt and corrects the tilt of the character string. The shape adjusting unit 32c adjusts the overall shape and size of the acquired document image.

In step S107, the layout recognition unit 33 acquires a document image that has undergone preprocessing performed by the preprocessing unit 32.
The acquired document image after preprocessing is sent to the document image cutting process of step S115, step S120, and step S136 described later.

In step S108 and step S109, the layout recognition unit 33 performs layout recognition of the document image, specifies the range of a plurality of elements included in the document image for each element, and acquires the type and position information for each element. do.
The types of elements are character strings, tables, images, seals, and handwriting.

In step S110, the layout recognition unit 33 adjusts the position information of the minimum boundary box of the acquired element.
The minimum boundary box means the rectangle surrounding the element and having the smallest area, and means the range occupied by the element. The layout recognition unit 33 collates the document image with the acquired element, and if there is a discrepancy between the document image and the position information of the acquired element, the layout recognition unit 33 adjusts the position information of the minimum boundary box of the acquired element. ..

In step S111, the layout recognition unit 33 acquires the layout information after the adjustment process of the minimum boundary box performed in step S110. The layout information includes element types and position information.

In step S112, the layout recognition unit 33 refers to the layout information of the internally stored element sent by the process of step S130 described later, and determines whether or not other elements remain in the document image. do.

When the layout information of all the elements is included in the layout information of the internally stored elements sent by the process of step S130, the layout recognition unit 33 has other elements remaining in the document image. It is determined that there is no such thing (No: S112), the process proceeds to step S131, the loop termination processing of step S112 to step S130 is performed, and the process proceeds to step S132.

On the other hand, when the layout information of all the elements is not included in the layout information of the internally stored elements sent by the process of step S130, the layout recognition unit 33 has other elements in the document image. It is determined that it remains (Yes: S112), and the process proceeds to step S113.

In step S113, the layout recognition unit 33 determines whether or not the element remaining in the document image is a table.
If no table remains in the document image (No: S113), layout information other than the table is sent to step S130 described later.

If the table remains in the document image (Yes: S113), the process proceeds to step S114. Since the document image is related to the receipt, it often includes a table. Therefore, if it is determined that the document image does not include the table, the layout recognition unit 33 may interrupt the process and confirm whether or not the electronic document relates to a receipt.

In step S114, the layout recognition unit 33 acquires the size and position information of all the vertical lines and horizontal lines constituting the table in the document image. If the size and position information of all the vertical lines and the horizontal lines constituting the table are acquired, the size and position of the cells can be acquired for all the cells included in the table.

In step S115, the cutout unit 34 cuts out a table image from the preprocessed document image acquired by the process of step S107.
In step S116, the cutout unit 34 acquires an image of the table cut out in step S115.

In step S117 and step S118, the cutting unit 34 performs a process of extracting cells from the image of the table acquired in step S116 (step S117), and acquires cell information (step S118).

The cell information is the row, column, and coordinates corresponding to the cell position information in the table.
The cell information acquired in step S118 is sent to step S127, which will be described later.

In step S119, the cutting unit 34 refers to the layout information of the internally stored table sent by the process of step S127, and determines whether or not other cells remain in the table.

When the layout information of all the cells is included in the layout information of the internally stored cells sent by the process of step S127, the cutout unit 34 has no other cells left in the table. (No: S119), the process proceeds to step S128, the loop end processing of step S119 is performed, and the process proceeds to step S130.

On the other hand, when the layout information of all the cells is not included in the layout information of the internally stored cells sent by the process of step S127, the cutout portion 34 has other cells remaining in the table. (Yes: S119), and the process proceeds to step S120.

In step S120, the cutting unit 34 performs a process of cutting out a cell image from the preprocessed document image acquired by the process of step S107.
In step S121, the cutting unit 34 acquires an image of the cell cut out by the process of step S120.

In step S122, the character string recognition unit 35 performs a character string recognition process on the image of the cell acquired by the process of step S121.
In step S123, the character string recognition unit 35 acquires the position information of the character string for which the character string recognition process has been performed.

In step S124, the character string recognition unit 35 adjusts the position information of the minimum boundary box of the character string acquired by the process of step S123.
The character string recognition unit 35 collates the document image with the position information of the acquired character string, and if there is a discrepancy between the document image and the position information of the acquired character string, the minimum boundary box of the acquired character string. Adjust the position information of.

In step S125, the character string recognition unit 35 acquires the position information after the adjustment process of the position information of the minimum boundary box of the character string carried out in step S124.

In step S126 and step S127, the character string recognition unit 35 merges the cell information acquired by the process of step S118 and the position information of the adjusted character string acquired by the process of step S125 (step S127). , Internally stored in the internal storage device as table layout information (step S126). The internal storage device refers to either or both of the RAM 23 and the storage unit 24 shown in FIG.

The processing of steps S119 to S127 is performed for all cells included in the table. After the processing of steps S119 to S127 is performed on the last cell included in the table, the loop termination processing of step S128 is performed, and the character string recognition unit 35 shifts to step S130.

In step S129 and step S130, the output unit 36 merges the layout information of the table acquired by the process of step S126 and the layout information other than the table acquired by the process of step S113 (step S130), and all the elements. Is internally stored in the internal storage device as the layout information of (step S129).

The processing of steps S112 to S130 is performed for all the elements included in the document image. After the processing of steps S112 to S130 is performed for the last element included in the document image, the loop end processing of step S131 is performed, and the character string recognition unit 35 shifts to step S132.

In step S132, the character string recognition unit 35 determines whether or not other elements remain in the document image.
The character string recognition unit 35 refers to the layout information of the internally stored elements sent by the process of step S140 described later, and determines whether or not other elements remain in the document image.

When the layout information of all the elements is included in the layout information of the internally stored elements sent by the process of step S140, the character string recognition unit 35 has another degree in the document image. It is determined that there is no remaining (No: S132), the process proceeds to step S141, the loop termination process of step S132 to step S140 is performed, and the process proceeds to step S142.

On the other hand, when the layout information of all the elements is not included in the layout information of the internally stored elements sent by the process of step S140, the character string recognition unit 35 has another element in the document image. Is determined to remain (Yes: S132), and the process proceeds to step S133.

In step S133, the character string recognition unit 35 determines whether or not the element remaining in the document image is a character string.
When the character string recognition unit 35 determines that the element remaining in the document image is a character string (Yes: S133), the process proceeds to step S135.

When the character string recognition unit 35 determines that the element remaining in the document image is not a character string (No: S133), the loop continuation process of shifting to step S132 is performed (step S134).
In step S135, the character string recognition unit 35 acquires the position information of the character string.

In step S136 and step S137, the character string recognition unit 35 cuts out an image of the character string from the preprocessed document image acquired by the process of step S107 (step S136), and acquires the image of the character string.

In step S138 and step S139, the character string recognition unit 35 performs a character string recognition process on the image of the character string acquired by the process of step S137 (step S138), and the text data predicted by the character string recognition process. Is generated (step S139).

In step S140, the character string recognition unit 35 merges the position information of the character string acquired by the process of step S135 and the text data generated by the process of step S139 to generate the layout information of the element. The layout information of the generated element is sent to step S129. In step S129, the layout information of the sent elements is internally stored in the internal storage device. The internal storage device refers to either or both of the RAM 23 and the storage unit 24 shown in FIG.

The processing from step S132 to step S140 is performed until it is determined by the processing of step S132 that the layout information of all the elements is included in the layout information of the elements sent by the processing of step S140.

In step S141, in response to the determination by the process of step S132 that the layout information of all the elements is included in the layout information of the elements sent by the process of step S140, steps S132 to S140 The process of ending the loop up to is performed, and the process proceeds to step S142.

In step S142, the electronic document generator 10 performs post-processing. In the post-processing, the text data, images, and position information of all the elements are output to JSON (Javascript objectionation) and converted to TSV (Tab-Separated Values).
The processing of each of the above-mentioned functional units is a processing executed by the CPU 25 of the electronic document generation device 10.

In step S143, the output unit 36 has a simple text file as a final form, an HTML (HyperText Markup Language), a file format editable by commercially available character editing software, and a file format in which the information of all the elements that have undergone post-processing can be edited. Editable is output as an electronic document such as PDF.

According to the above-described embodiment, the electronic document generator 10 recognizes the layout of the document image using the layout learning model 14, and then performs character recognition of the document image using the character string learning model 13. That is, since the electronic document generation device 10 identifies the types of a plurality of elements included in the document image and performs character recognition suitable for the types of elements, the recognition accuracy of character recognition can be improved.

Further, according to the above-described embodiment, the electronic document generator 10 uses the character string learning model 13 to characterize a document image, as compared with character recognition in character units, which has been performed by conventional OCR text recognition technology. Since the recognition is performed for each character string, the recognition efficiency at the time of character recognition can be improved.

Further, according to the above-described embodiment, when the electronic document generator 10 performs character recognition, the character recognition is performed for each character string instead of the character recognition for each character, so that noise existing on the characters and the like are generated. Character recognition can be performed by suppressing the influence of the above, and the recognition accuracy of character recognition can be improved as compared with character recognition performed in units of one character.

Further, according to the above-described embodiment, even a character that is erroneously recognized by character recognition using the conventional OCR text recognition technique can be correctly recognized by character recognition using the character string learning model 13. For example, when a seal is superimposed on a character, the character may be erroneously recognized by the conventional OCR text recognition technique, but can be correctly recognized by character recognition using the character string learning model 13. ..

Further, according to the above-described embodiment, the electronic document generator 10 recognizes the elements whose types correspond to the table for each character string included in the image related to the cell alone, and therefore the characters included in the table. It is possible to improve the recognition accuracy of character recognition in a column.

Further, according to the above-described embodiment, the character string learning model 13 and the layout learning model 14 are learned from the annotated character string learning data and the layout learning data, so that the layout recognition unit 33 and the character string recognition unit 35 are learned. It is possible to improve the recognition accuracy of.

Further, according to the above-described embodiment, when there is a table in the document image, all the vertical lines and horizontal lines constituting the table are first recognized, and all the cells included in the table are recognized. After that, for all cells, the character string recognition is performed on the image for each cell without being affected by the position information inside the table, so that the recognition accuracy of the character recognition of the character string inside the cell can be improved. can.

The present disclosure is not limited to the electronic document generator 10 according to the above-described embodiment, and is carried out by various other modifications or applications as long as it does not deviate from the gist of the present disclosure described in the claims. It is possible.

10 Electronic document generator 11 Information communication network 12 User terminal 13 Character string learning model 14 Layout learning model 15 Document image database 20 Input / output interface 21 Communication interface 22 ROM
23 RAM
24 Storage unit 25 CPU
26 Input device 27 Output device 28 GPU
31 Document image acquisition unit 32 Preprocessing unit 32a Background removal unit 32b Tilt correction unit 32c Shape adjustment unit 33 Layout recognition unit 34 Cutout unit 35 Character string recognition unit 36 Output unit 40 Layout learning data generation unit 41 Layout learning data correction Part 42 Layout learning part 43 Character string learning data generation part 44 Character string learning data correction part 45 Character string learning part 47 Document image before tilt correction 48 Character string 49 Table 50 Hotchkiss mark 51 Handwritten 52 Seal 53 Image 54 Noise removal 55 Preprocessing 56 Layout recognition processing 57 Character

string recognition processing

58a, 59a,

60a Document image

58b, 59b,

60b Document image

61, 62

Document image

63, 64 Table 65 Vertical line 66 Horizontal line 67

Cell image

69, 70, 73 Character string Image 71a Character string image 71b Text data 72 Recognition range 73 Table 75 Recognition range 76 Character string comment symbol 77 Table comment symbol 78 Image comment symbol 79 Seal comment symbol 80 Outer frame comment symbol 81 Noise comment symbol 82 Handwritten comment symbol 83 Vertical line comment symbol 84 Horizontal line comment symbol 85 Text data comment 100 Electronic document generation system S31 Document image acquisition step S32 Preprocessing step S33 Layout recognition step S34 Cutout step S35 Character recognition step S36 Output step

Claims

A document image acquisition unit that acquires a document image that is an image of a document,
Using a character string learning model that learned the correspondence between the document image and the character string included in the document image,
A character string recognition unit that recognizes a character string included in the document image acquired by the document image acquisition unit and generates text data related to the character string, and a character string recognition unit.
An output unit that outputs the text data as text on an electronic medium,
An electronic document generator, characterized in that it comprises.
Using a layout learning model that learned the correspondence between a plurality of elements included in a document image and the identification information of each of the plurality of elements,
The range of each of the plurality of elements included in the document image acquired by the document image acquisition unit within the document image is specified, each type of the plurality of elements is recognized, and each of the plurality of elements is recognized. Further, a layout recognition unit for acquiring position information in the document image related to the range is provided.
The character string recognition unit recognizes a character string included in the range recognized by the layout recognition unit using the character string learning model, and generates text data related to the character string.
The output unit outputs each of the text data related to the plurality of elements to the position information in each of the ranges related to the plurality of elements as text on an electronic medium.
The electronic document generator according to claim 1.
The type of the element is either a string, a table, an image, a seal, or a handwriting.
The electronic document generator according to claim 2.
In the element whose type corresponds to the table recognized by the layout recognition unit, each of the cells in the table included in the element is cut out, and the position information of each of the cells in the document image is acquired. With more cutouts
The character string recognition unit recognizes a character string included in each of the cells cut out in the cutout unit using the character string learning model, and generates text data related to the character string.
The electronic document generator according to claim 3.
A document image containing a plurality of the elements, and the elements are annotated with the annotations associated with the types corresponding to each of the elements.
It further includes a layout learning data generation unit that accumulates a plurality of the document images to which the annotation is attached and generates layout learning data.
The layout learning data is used for supervised learning of the layout learning model.
The electronic document generator according to any one of claims 2 to 4, wherein the electronic document generator is characterized by the above.
The electronic document generator according to claim 5, wherein the document image is provided with position information in the document image for each of the ranges related to the plurality of elements included in the document image together with the annotation. ..
Based on the input, at least one of the types of the plurality of elements recognized by the layout recognition unit and the position information in the document image of each range of the plurality of elements is corrected and corrected. It further includes a layout learning data correction unit that updates the layout learning data by adding the data.
The electronic document generator according to claim 5 or 6.
A layout learning unit for re-learning the layout learning model using the layout learning data updated by the layout learning data correction unit is further provided.
The electronic document generator according to claim 7.
It further includes a character string learning data generation unit that generates character string learning data used for supervised learning of the character string learning model.
The electronic document generator according to any one of claims 2 to 8.
Based on the input, the text data generated by the character string recognition unit is modified, and the character string learning data correction unit that updates the character string learning data by adding the modified text data is further provided. ,
The electronic document generator according to claim 9.
A character string learning unit for re-learning the character string learning model using the character string learning data updated by the character string learning data correction unit is further provided.
The electronic document generator according to claim 10.
The character string recognition unit includes a plurality of the character string learning models, and uses the character string learning model adapted to the language of the character string included in each of the plurality of elements.
The electronic document generator according to any one of claims 2 to 11.
A pre-processing unit that performs pre-processing on the document image acquired by the document image acquisition unit is further provided.
The pretreatment unit includes a background removal unit, a tilt correction unit, and a shape adjustment unit.
The background removing unit removes the background of the document image acquired by the document image acquisition unit.
The tilt correction unit corrects the tilt of the document image acquired by the document image acquisition unit.
The electronic document generation device according to any one of claims 2 to 12, wherein the shape adjusting unit adjusts the overall shape and size of the document image acquired by the document image acquisition unit.
The layout learning model is either a layout learning model for contracts, a layout learning model for invoices, a layout learning model for memorandums, a layout learning model for invoices, or a layout learning model for receipts. The electronic document generator according to any one of claims 2 to 13, wherein the electronic document generator is characterized by the above.
The computer used for the electronic document generator
A document image acquisition step to acquire a document image that is an image of a document,
Using a character string learning model that learned the correspondence between the document image and the character string included in the document image,
A character string recognition step that recognizes a character string included in the document image acquired in the document image acquisition step and generates text data related to the character string, and a character string recognition step.
An output step for outputting the text data as text on an electronic medium, and
An electronic document generation method characterized by performing.
For computers used in electronic document generators
A document image acquisition function that acquires a document image that is an image of a document,
Using a character string learning model that learned the correspondence between the document image and the character string included in the document image,
A character string recognition function that recognizes a character string included in the document image acquired by the document image acquisition function and generates text data related to the character string, and a character string recognition function.
An output function that outputs the text data as text on an electronic medium,
An electronic document generation program characterized by demonstrating.