JP2023031036A - Information processing apparatus, information processing system, control method of information processing apparatus, and program - Google Patents

Abstract

To provide an information processing apparatus configured to accurately extract information from an image including a handwritten character, an information processing system, a control method of the information processing apparatus, and a program.SOLUTION: In a software configuration of an image processing system, a key value extraction server 102 includes: a printed character OCR unit 313 which specifies regions of a printed character and a handwritten character from image data of a document including the printed character and a handwritten character to perform character recognition; a ruled line extraction unit 312 which specifies positions of ruled lines included in the image data; a handwritten character region normalization unit 315 which normalizes the handwritten region on the basis of at least one of the printed character region and the positions of the ruled lines; and a key value extraction unit 316 which extracts a character string corresponding to a value of a predetermined item, using an extraction rule 330, on the basis of character strings recognized from handwritten characters corresponding to the printed character region and the normalized region. The extraction rule is trained with a result which is obtained by correcting a result extracted by the key value extraction unit by a user.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing device, an information processing system, a control method for an information processing device, and a program.

There are cloud-based accounting services that provide processing related to corporate accounting operations. In order to use an accounting service to manage and pay bills, it is necessary to extract necessary information from paper invoices and enter it into the accounting service. There are input support systems for assisting these input operations. The input support system scans the bill with a scanner and performs character recognition on the scanned bill image. After the user confirms and corrects the recognition result, the data of the corrected result is registered in the accounting service.
Input assistance systems exist that use learning models trained by machine learning to extract information from recognition results for registration with accounting services. By machine learning the characteristics of character recognition results and the labels assigned to the character recognition results (values) corresponding to the predetermined items (keys) required for accounting services, billing formats are not defined in advance. , information can be extracted. The features of the character recognition result used for this learning include rectangle information of the character area.
There is also an input support system that recognizes handwritten characters written on bills and supports registration in accounting services. In the case of handwritten characters, variations occur in the size and position of characters compared to printed characters. Therefore, the rectangular information of the character area, which is a feature of the character recognition result, also varies. A learning model trained using the features of character recognition results of handwritten characters has a problem in the accuracy of information extraction due to variations in the rectangle information of character regions.
Patent Literature 1 discloses a technique for normalizing character images.

JP-A-11-161740

By using the technique of normalizing the character image disclosed in Japanese Patent Application Laid-Open No. 2002-200315, it is possible to reduce variations in the rectangle information of the character areas of the handwritten characters. However, in the technique of Patent Document 1, since normalization is performed based on the word region specified as the target of recognition, the result of normalization differs for each bill even if the bills have the same format. Due to invoice-to-invoice variability, learning models trained using features from handwritten character recognition results still struggle with information extraction accuracy.
SUMMARY OF THE INVENTION It is an object of the present invention to provide an image processing system capable of accurately extracting information from an image including handwritten characters without defining the format of the document in advance.

In order to achieve the above object, the information processing apparatus of the present invention comprises recognition means for specifying character regions of the printed characters and handwritten characters from image data of a document containing the printed characters and handwritten characters, and recognizing the characters included in the image data. a normalizing means for normalizing the character area of the handwritten character based on at least one of the area of the printed character and the position of the ruled line; and the area of the printed character and the normalization. extracting means for extracting a character string corresponding to a value of a predetermined item using a rule, based on the character string recognized from the handwritten character corresponding to the identified area, wherein the rule is the extraction It is characterized in that it is a rule learned using a correction result obtained by correcting the extraction result by the means by the user.

According to the information processing apparatus of the present invention, it is possible to accurately extract information even from an image including handwritten characters without defining the format of the document in advance.

1 is a diagram showing the overall configuration of an image processing system; FIG. 1 is a diagram showing an example of the hardware configuration of an image processing system; FIG. FIG. 2 is a diagram showing an example of the software configuration of an image processing system; An example of image data generated in a step of the processing control flow. An example of ruled line information generated in a step of the process control flow. An example of a character recognition result generated in a step of the process control flow. An example of a character recognition result generated in a step of the process control flow. An example of the normalization result of the handwritten character area generated in the step of the process control flow. An example of image data generated in a step of the processing control flow. An example of an information extraction result generated in a step of the process control flow. An example of an information extraction result generated in a step of the process control flow. FIG. 10 is a diagram showing a screen displayed in a step of the processing control flow; The figure which shows a processing control flow. The figure which shows a processing control flow. FIG. 4 is a diagram for explaining transition of screens displayed in steps of the processing control flow;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. The embodiments do not limit the present invention, and not all configurations described in the embodiments are essential for solving the problems of the present invention.

(Embodiment 1)
[System configuration]
EMBODIMENT OF THE INVENTION Hereinafter, the form for implementing this invention is demonstrated using drawing.
First, an information processing system that is a premise of the present invention will be described.
FIG. 1 is a diagram showing the overall configuration of an information processing system according to an embodiment of the present invention. In FIG. 1, an MFP (Multifunction Peripheral) 101 is an image input device connected to a network and equipped with a paper document scanning function.
A key-value extraction server 102, which is an information processing device, provides a service for extracting information (value) corresponding to a predetermined item (key) required for accounting services from a scanned image of a paper document. The key-value extraction server 102, for example, extracts a character string "2019/06/12" corresponding to a value corresponding to a key such as "date" from the invoice image.
Accounting server 103, which is an information processing device, provides services for processing company accounting operations. The MFP 101 , key-value extraction server 102 and accounting server 103 are connected to each other via the Internet 100 .

[Hardware configuration]
FIG. 2(a) is a block diagram showing the basic hardware configuration of the key-value extraction server 102 and accounting server 103, which are information processing apparatuses.
In FIG. 2A, a CPU (Central Processing Unit) 201 is a unit that executes various programs and implements various functions. A RAM (Random Access Memory) 202 is a unit that stores various information. Also, the RAM 202 is a unit that is also used as a temporary work storage area for the CPU 201 . A ROM (Read Only Memory) 203 is a unit that stores various programs and the like. For example, the CPU 201 loads a program stored in the ROM 203 into the RAM 202 and executes the program.
In addition, the CPU 201 executes processing based on programs stored in an external storage device such as flash memory, HDD (Hard Disk Drive), or SSD (Solid State Disk). As a result, the software configuration constituting the key-value extraction server 102 as shown in FIG. 3, which will be described later, and the processing of each step of the sequence, which will be described later, are realized.
All or part of the functions of the key-value extraction server 102 and the processing related to the sequence described later may be realized using dedicated hardware such as ASIC.
An input/output interface 204 is a unit that provides an interface with an output device such as a display device and an input device such as a keyboard and mouse.
A NIC (Network Interface Card) 205 is a unit for connecting the key-value extraction server 102 and accounting server 103 to a network (not shown).
The units described above are configured to be able to transmit and receive data to and from each other via the bus 206 .
Note that the key-value extraction server 102 and accounting server 103 may be realized by one computer or may be realized by a plurality of computers. For example, the key-value extraction server 102 and accounting server 103 may be configured using cloud computing technology. That is, the key-value extraction server 102 and the accounting server 103 may be implemented as cloud services.

FIG. 2(b) is a block diagram showing the basic hardware configuration of the MFP 101, which is an image input device. In FIG. 2(b), the CPU 211, RAM 212, ROM 213, and NIC 214 are the same as the CPU 201, RAM 202, ROM 203, and NIC 205 in FIG. 2(a), respectively.
Scanner 215 is an input unit that converts paper documents into image data. A printer engine 216 is an output unit that prints image data. The operation panel 217 is an input unit that receives touch operations from the user, and is an output unit that displays information of the MFP 101 on the display.
The units described above are configured to be able to transmit and receive data via bus 218 .

[Software configuration]
Next, the configuration of software in the MFP 101 and the key/value extraction server 102 will be described. A description of the accounting server 103 is omitted.
FIG. 3(a) is a diagram showing a software configuration diagram of the MFP 101. FIG. The following software units are implemented by the CPU 101 developing programs stored in the ROM 203 in the RAM 202 and executing them.
A UI unit 301 receives input from the user via the operation panel 217 . A scan execution unit 302 converts data input from the scanner 215 into image data. The image data transmission unit 303 transmits image data to the key/value extraction server 102 via the NIC 214 . The key/value extraction result receiving unit 304 receives the key/value extraction result from the key/value extraction server 102 via the NIC 214 . A correction result transmission unit 305 transmits the correction result made by the user through the UI unit 301 to the key/value extraction server 102 through the NIC 214 .

FIG. 3(b) is a diagram showing a software configuration diagram of the key-value extraction server 102, which is an information processing device. The following software units are realized by the CPU 211 developing programs stored in the ROM 203 in the RAM 212 and executing them.
The image data receiving unit 311 receives the image data from the MFP 101 via the NIC 205 of the key/value extraction server 102 . A ruled line extraction unit 312 extracts ruled lines from the image data and specifies an area (cell area) surrounded by the ruled lines. A printed character OCR unit 313 identifies a printed character area from the image data and performs character recognition. A handwritten character OCR unit 314 identifies a character area of handwritten characters from the image data and performs character recognition. Handwritten character area normalization section 315 normalizes the handwritten character area. A key-value extraction unit 316 extracts a value corresponding to a predefined key from the character recognition result and the normalized character area. A key-value extraction result transmission unit 317 transmits the key-value extraction result to the MFP 101 via the NIC 205 . A correction result receiving unit 318 acquires the correction result made by the user from the MFP 101 via the NIC 205 . The learning unit 319 learns based on the received correction results. A key-value data transmission unit 320 transmits the key-value data to the accounting server 103 via the NIC 205 . Also, the extraction rule 330 is used to extract the value corresponding to the predefined key from the character recognition result and the normalized character area. Extraction rules 330 are updated by learning the modified results.
The extraction rule 330 in this embodiment is a learning model (learned model) trained by machine learning the features of the character recognition results and the labels assigned to the character recognition results corresponding to the keys. Features of character recognition results include:
・Features obtained from the character string resulting from character recognition ・Rectangle information of the character area ・Features obtained from the surrounding character strings resulting from character recognition

[Processing control flow]
A processing control flow in the information processing system of the present invention will be described with reference to FIG. 4 to 9A and 9B (hereinafter collectively referred to as "FIG. 9") show examples of image data, ruled line information, character recognition results, and key/value extraction results generated in the steps of the flow of FIG. . FIG. 10 shows a screen for confirming/correcting the key-value extraction result displayed on the operation panel 217 of the MFP 101 in the steps of the flow of FIG. FIG. 12 shows a detailed flow of steps for normalizing the character regions of the handwritten characters of FIG. FIG. 13 shows transition of screens displayed on the operation panel 217 of the MFP 101 in steps of the processing control flow of FIG.

First, in S1101, the user sets a document (paper document) on the scanner 215 of the MFP 101, and the processing control flow is started when the user issues an instruction to start scanning.
In S1102, the scan execution unit 302 of the MFP 101 scans the paper document and generates image data. The image data transmission unit 303 transmits to the key/value extraction server 102 . FIG. 4 shows an example of image data generated by the scan execution unit 302 of the MFP 101 and transmitted to the key/value extraction server 102. As shown in FIG.
In S1103, the image data receiving unit 311 of the key-value extraction server 102 receives the image data transmitted from the MFP 101, the ruled line extracting unit 312 extracts the ruled lines, and specifies the area (cell area) surrounded by the ruled lines. . FIG. 5 shows the results of extracting ruled lines from the image data of FIG. 4 and identifying cell regions in a table format. A table 501 shows (circumscribed) rectangular information of the extracted ruled lines, and the area of the ruled lines corresponding to the row 503 is 431 in FIG. Specifically, the x-coordinate of the upper left corner of the rectangle is 229, the y-coordinate is 428, the width of the rectangle is 162, and the height is 2. Since it is a ruled line, the height is small, but if the ruled line is slanted during reading, the height of the circumscribing rectangle may increase. A table 502 is rectangular information of the extracted cell area, and the area corresponding to the row 504 is 441 in FIG. The x-coordinate of the upper left corner of the rectangle is 107, the y-coordinate is 488, the width of the rectangle is 194, and the height is 20.
At S1104, the printed character OCR unit 313 performs character recognition of printed characters.
In S1105, the handwritten character OCR unit 314 performs character recognition of handwritten characters. FIGS. 6A and 6B (hereinafter collectively referred to as “FIG. 6”) show the results of printed character recognition and handwritten character recognition performed on the image of FIG. 4 in tabular form. Table 601 shows the character recognition results for printed characters. Table 611 shows the character recognition results for handwritten characters. Columns 602 and 612 are the rectangle information of the character regions of the character recognition results. Columns 603 and 613 show character strings resulting from character recognition. Dotted lines 401 to 408 in FIG. 4 indicate character areas of printed characters corresponding to lines 621 to 628, respectively. Dotted lines 411 to 420 in FIG. 4 indicate character areas of handwritten characters corresponding to lines 631 to 640, respectively.
In S1106, the handwritten character region normalization unit 315 normalizes the handwritten character region. A table 701 in FIG. 7 shows, in tabular form, the result of normalizing the handwritten character area for the handwritten character recognition result in the table 602 in FIG. A column 702 is rectangle information of the area obtained by normalizing the character area of the character recognition result. Dotted lines 801-810 in FIG. 8 indicate normalized character regions of handwritten characters corresponding to lines 711-720, respectively. Details of the process control for normalizing the character area of handwritten characters will be described later with reference to FIG.

In S1107, the key-value extraction unit 316 uses the extraction rule 330 to extract the character recognition result corresponding to the predefined key as a value. The keys are predetermined items required by the accounting server 103, and in this embodiment are five items: title, telephone number, number, date, and amount. A key-value extraction result transmission unit 317 transmits the character recognition result and the key-value extraction result to the MFP 101 . FIG. 9 shows, in tabular form, the result of key-value extraction (S1107) from the printed character area and the character recognition result in the table 601 of FIG. 6 and the normalized area of the handwritten character area and the character recognition result in FIG. showing. A column 902 of the table 901 shows the labels assigned to the character recognition results of printed characters. Column 912 of table 911 shows the labels given to the character recognition results of handwritten characters. The corresponding key is given as a label to the character recognition result extracted as the value corresponding to the key.
In S1108, the UI unit 301 of the MFP 101 displays the key-value extraction result on the operation panel 217. FIG. 1001 in FIG. 10 shows the screen displayed in S1105. 1002 to 1006 are value display fields for displaying values corresponding to keys. In the value display columns, the character recognition results with the labels shown in FIG. 9 are displayed. Also, by touching the value display column, it becomes an edit mode, and editing becomes possible. 1007 is a registration button.
In S1109, the user checks the extraction result of the value corresponding to the key on the screen 1001 (1301 in FIG. 13) displayed on the operation panel 217 of the MFP 101, and corrects any errors (1306 in FIG. 13). When the correction is detected, the UI unit 301 of the MFP 101 displays a preview screen of the image data generated in step S1102 on the operation panel in order to set the correct rectangular area because there is a possibility that the rectangular area with the label corresponding to the correction is incorrect. 217 (1303 in FIG. 13). The user touches and selects the character area of the corrected character string on the image data preview screen displayed on the operation panel 217 of the MFP 101 (1307 in FIG. 13). The UI unit 301 of the MFP 101 that has detected the user's touch on the preview screen displays a screen 1001 on the operation panel 217 . After completing a series of correction operations, the user presses the registration button 1007 (1308 in FIG. 13).
In S1110, the correction result transmission unit 305 updates the character string of the key-value extraction result with the corrected character string. Also, if the character area is changed, the label is added again. In this manner, the correction result transmitting unit 305 corrects the extraction result. Furthermore, the correction result transmission unit 305 transmits the extraction result (correction result) corrected as described above to the key-value extraction server 102 .
In S1111, the correction result receiving unit 318 of the key/value extraction server 102 receives and acquires the correction result transmitted from the MFP101. Then, learn based on the (normalized) character recognition result created in S1106, the label assigned in S1107, and the acquired correction result (i.e., normalized region, character recognition result (character string), label) The extraction rule 330 is updated as the unit 319 learns. Note that the key-value extraction server 102 may be configured to update the extraction rule 330 by requesting the above learning to an external device (which may be a cloud service) and receiving the learning result. In other words, the learning unit 319 may be outside the key/value extraction server 102 .
In S 1112 , the key-value data transmission unit 320 transmits the key and value (character string corresponding to the key) data to the accounting server 103 .

A detailed control flow of the handwritten character area normalization process (S1106) executed by the handwritten character area normalization unit 315 will be described with reference to FIG.
In S1201, the handwritten character area normalization unit 315 determines whether the character area of the handwritten character overlaps an area (cell area) surrounded by ruled lines. If it is determined that they overlap (Yes in S1201), the process proceeds to S1202.
In S1202, the handwritten character area normalization unit 315 determines whether or not there is a printed character area in the cell area overlapping the character area of the handwritten character. If it is determined that there is no printed character area (Yes in S1202), the process proceeds to S1203.
In S1203, the handwritten character area normalization unit 315 normalizes the character area of the handwritten character in the cell area. Specifically, the cell area is set as a normalized character area for handwritten characters. Row 717 in FIG. 7 (417 in FIG. 4) is an example (807 in FIG. 8) normalized by the cell area (441 in FIG. 4) overlapping the character area of handwritten characters. A cell area (row 504 in FIG. 5) overlapping the character area of line 717 is set to the normalized character area.
In S1202, if the handwritten character area normalization unit 315 determines that there is a printed character area (Yes in S1202), the process proceeds to S1204.
In S1204, the handwritten character area normalization unit 315 normalizes the size and position of the character area of the handwritten character based on the character area of the nearby printed character, or the closest printed character. Specifically, the height of the size of the character area is set to the height of the character area of the nearby printed character, and the width of the size of the character area is set to the height times the number of characters. This number of characters is the number of characters recognized by OCR processing in S1105. Sets the position of the character area to a position that touches the character area of the nearby type. Specifically, if there is a printed character on the left side of the handwritten character, the area to the right of the printed character is normalized to the height x number of characters area, and if there is a printed character above the handwritten character, the printed character and the left edge are normalized. Align and normalize to the region that follows below. Row 718 in FIG. 7 (418 in FIG. 4) is a normalized example (808 in FIG. 8) based on the character area of the type (406 in FIG. 4). The height of the character area of the type closest to the character area of line 718 (line 626 in FIG. 6) is set to the height of the normalized character area, and the width is set to height times the number of characters. The position of the normalized character area is set to a position in contact with the character area of the type.
If it is determined in S1201 that the character area of the handwritten character does not overlap the cell area, in S1205 the handwritten character area normalization unit 315 determines whether the character area of the handwritten character is within a predetermined distance range of the top, bottom, left, or right. Determines whether there is a character area for type in the If the handwritten character area normalization unit 315 determines that there is a printed character area, the process advances to S1204. 7, lines 711-715 (411-415) of FIG. ) are normalized based on the character area of the type (801-805 in FIG. 8). 7 are normalized based on the printed character areas of lines 627 (407) and 628 (408) of FIG. 809, 810 in FIG. 8).
If the handwritten character area normalization unit 315 determines in S1205 that there is no printed character area, then in S1206 there are ruled lines in the vicinity of the handwritten character area (overlapping the character area or bordering the upper, lower, left, and right sides of the character area). within one of the predefined distances). If it is determined that there is a ruled line, in S1207 the character area of the handwritten character is normalized based on the mode of the height of the character area of the printed character and the ruled line. Specifically, the height of the character area is set to the mode of the height of the character areas of all characters in the image, and the width of the character area is set to height×the number of characters. Align the beginning of the position of the character area with the ruled line and set it to a position that touches the ruled line. Row 716 in FIG. 7 (416 in FIG. 4) is a normalized example (806 in FIG. 8) based on the most frequent height and ruled line (431 in FIG. 4) of the character area of the type. The mode of the height of the character area of all the characters (the mode of height in column 602 of the table 601 in FIG. 6) is set to the height of the normalized character area, and the width is set to the height times the number of characters. is set. The position of the normalized character area is set such that the top matches the top of the ruled line and is in contact with the ruled line. Although the mode value is shown as an example of the representative value of the height of the character area of the type, an average value, a median value, or the like may be used.

In S1206, when the handwritten character area normalization unit 315 determines that there is no ruled line near the character area of the handwritten character, the handwritten character area normalization unit 315 terminates without normalizing the character area.
In this flow, on the premise that the horizontally written character area is normalized, the height of the type area and the height×the number of characters are set to the height and width of the normalized area, respectively. When normalizing a vertically written character area, set the width and width of the type area×the number of characters to the width and height of the normalized area, respectively.

As described above, the information processing system reduces variations in normalization results for forms (invoices, etc.) of the same format by normalizing based on the character areas and ruled lines of printed characters. Since the normalized handwritten character regions and the labels attached to them are learned as learning data, even images containing handwritten characters can be accurately extracted without defining a format in advance. be able to.

(Other examples)
In the image processing system described above, the MFP and the key/value extraction server are separate units, but a single image processing apparatus having all the functions may be used. That is, the MFP 101 may include a ruled line extraction unit 312, a printed character OCR unit 313, a handwritten character OCR unit 314, a handwritten character area normalization unit 315, a value extraction unit 316, a learning unit 319, a key/value data transmission unit 320, and the like. .
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.
Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

101 MFPs
102 Key-value extraction server
313 Type OCR Department
314 Handwritten Character OCR Section
319 Learning Department
330 extraction rules

Claims (9)

recognition means for identifying character regions of the printed characters and handwritten characters from image data of a document containing the printed characters and handwritten characters and performing character recognition;
an identifying means for identifying positions of ruled lines included in the image data;
normalization means for normalizing the character area of the handwritten character based on at least one of the area of the printed character and the position of the ruled line;
extracting means for extracting a character string corresponding to a value of a predetermined item using a rule, based on the character string recognized from the handwritten characters corresponding to the printed character area and the normalized area. death,
The information processing apparatus, wherein the rule is a rule learned using a correction result obtained by correcting a result of extraction by the extraction means by a user. Acquisition means for acquiring a correction result obtained by correcting the extraction result by the extraction means by a user;
learning means for updating the rule by learning using the correction result;
2. The information processing apparatus according to claim 1, wherein said extraction means performs said extraction using a rule updated by said learning means. When the character area of the handwritten character overlaps the area surrounded by the ruled lines, the normalization means normalizes the position and size of the character area of the handwritten character based on the area surrounded by the ruled lines. 3. The information processing device according to claim 1 or 2, wherein 3. The information processing according to claim 1, wherein said normalization means normalizes the position and size of the character area of the handwritten character based on the character area of the printed character in the vicinity of the character area of the handwritten character. Device. The normalization means normalizes the position of the character area of the handwritten character based on the position of a ruled line in the vicinity of the character area of the handwritten character, and normalizes the size of the character area of the handwritten character based on the size of the printed character. 3. The information processing device according to claim 1 or 2, wherein 6. The information processing apparatus according to claim 5, wherein the mode of the size of the type in the image is used as the size of the type. 7. An information processing system in which the information processing apparatus according to any one of claims 1 to 6 and an image input apparatus having scanning means and operation means are connected via a network. a recognition step of identifying character regions of the printed characters and handwritten characters from image data of a document containing printed characters and handwritten characters and performing character recognition;
an identifying step of identifying positions of ruled lines included in the image data;
a normalization step of normalizing the character area of the handwritten character based on at least one of the area of the printed character and the position of the ruled line;
an extraction step of extracting a character string corresponding to a value of a predetermined item using a rule, based on the character string recognized from the handwritten characters corresponding to the printed character area and the normalized area. death,
A control method for an information processing apparatus, wherein the rule is a rule learned using a correction result obtained by correcting an extraction result in the extraction step by a user. A program for causing a computer to execute the processing method of the information processing apparatus according to claim 8 .

Images