JP2022125220A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To record, even if there are many types of document formats, an arbitrary character to be recorded in an arbitrary document according to the type of document format.

SOLUTION: An image processing apparatus is configured to: acquire image data of a document form and a recorded character string that an operator has recorded among character strings included in the image data; extract a feature amount of the character string that matches the recorded character string among the character strings recognized by character recognition processing on the image data; machine-learn the feature amount of the character string that is extracted from the image data of each of the plurality of document forms of the same format to generate the feature amount to be read; and record the character string recognized from the image data of a new document form if the feature amount extracted from the character string recognized by the character recognition processing for the image data of the acquired new document form matches the feature amount to be read. Further, the feature amount is information indicating an attribute of the character string and a range of the character string in the image of the document form.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to an image processing device, an image processing method, and a program.

2. Description of the Related Art There is an image processing apparatus that performs OCR (Optical Character Recognition) processing on an image of a document obtained using a scanner or the like and reads the document. When documents have various formats, such an image processing apparatus stores format definition information of documents in advance and specifies a reading target area to improve the accuracy of OCR processing. A technique related to this is disclosed in Patent Document 1.

JP 2016-48444 A

However, when there are many types of document formats as described above, it takes a lot of work for the operator to store all the format definition information in the image processing apparatus.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an image processing apparatus, an image processing method, and a program that solve the above-described problems.

According to a first aspect of the present invention, an image processing apparatus includes an acquisition unit that acquires image data of a document form and a recorded character string recorded by an operator among character strings included in the image data; a feature amount extracting unit for extracting a feature amount of the character string that matches the recorded character string among character strings recognized by character recognition processing on the image data; Through machine learning of the feature amount of the extracted character string to generate the feature amount to be read, and character recognition processing for the image data of the new document acquired by the acquisition unit a recording unit that records the character string recognized from the image data of the new document form when the feature amount extracted from the recognized character string and the feature amount to be read match, The feature amount is information indicating the attribute of the character string and the range of the character string in the image of the document form.

According to a second aspect of the present invention, an image processing method acquires image data of a document and a recorded character string recorded by an operator among character strings included in the image data, and obtains the image data. out of the character strings recognized by character recognition processing for the character string that matches the recorded character string; machine-learning the amount to generate a feature amount to be read, and the feature amount extracted from the character string recognized by character recognition processing for the image data of the new document form matches the feature amount to be read. character strings recognized from the image data of the new document form are recorded, and the feature quantity is information indicating the attributes of the character strings and the range of the character strings in the image of the document form. be.

According to a third aspect of the present invention, a program obtains, in a computer, image data of a document and a recorded character string recorded by an operator among character strings included in the image data; a step of extracting a feature amount of the character string that matches the recorded character string among character strings recognized by character recognition processing on the image data; a step of performing machine learning on the feature quantity of the character string to generate a feature quantity to be read; and recording a character string recognized from the image data of the new document form when the read feature amount matches the feature amount to be read, wherein the feature amount is , the attribute of the character string and the range of the character string in the image of the document form.

According to the present invention, even if there are many kinds of document formats, arbitrary characters to be recorded in arbitrary documents corresponding to the kinds of formats can be recorded.

1 is a diagram showing an outline of an image processing system including an image processing device; FIG. 2 is a diagram showing the hardware configuration of an image processing apparatus; FIG. 1 is a functional block diagram of an image processing device; FIG. FIG. 4 is a diagram showing an example of a document form; 4 is a diagram showing an outline of a data table stored in a database; FIG. FIG. 4 is a first diagram showing a processing flow of the image processing apparatus according to the first embodiment; FIG. 7 is a second diagram showing the processing flow of the image processing apparatus according to the first embodiment; FIG. 5 is a functional block diagram of an image processing device according to a second embodiment; FIG. 11 is a first diagram showing a processing flow of the image processing device according to the second embodiment; FIG. 10 is a second diagram showing the processing flow of the image processing apparatus according to the second embodiment; FIG. 13 is a diagram showing a processing flow of an image processing device according to the sixth embodiment; FIG. 21 is a diagram showing an example of connection between an image processing system according to the seventh embodiment and an existing OCR system; FIG. 13 is a diagram showing a processing flow of an image processing device according to the sixth embodiment; FIG. 21 is a diagram showing an outline of an image processing system 200 according to an eighth embodiment; FIG. FIG. 22 is a diagram showing a processing flow of a terminal device according to the eighth embodiment; It is a figure which shows the processing flow of the server apparatus by 8th embodiment. It is a figure which shows the minimum structure of an image processing apparatus.

An image processing apparatus according to an embodiment of the present invention will be described below with reference to the drawings.
FIG. 1 is a diagram showing an overview of an image processing system including an image processing apparatus according to this embodiment. As shown in FIG. 1, the image processing system 100 is composed of an image processing device 1, an image reading device 2, a recording device 3, and a database 4. FIG.
The image processing device 1 is connected to the image reading device 2 by a communication cable. The image reading device 2 optically acquires image data such as a document form and outputs the image data to the image processing device 1 . The image processing apparatus 1 performs OCR processing on the image data of the document form and recognizes the characters. The image processing device 1 outputs the character recognition result to the recording device 3, and the recording device 3 records the character recognition result in a database. A database 4 is connected to the image processing device 1 and the recording device 3 . The database 4 stores correspondence relationships between image data of a plurality of document forms registered in the past from the recording device 3 and recorded character strings indicating character strings to be recorded among the character strings included in the image data. The character string indicated by the recorded character string is an important character string that should be recorded and saved in the database 4 among the character strings described in the document form. An operator who uses the image processing system 100 registers image data of a plurality of document forms registered in the past using the recording device 3 and recorded character strings among the character strings included in the image data in the database 4 in advance. Keep

In the recording device 3, the correspondence relationship between the image data of the document form and the information of the recording character string indicating the character string to be recorded among the information of the character string contained in the image data is sufficiently established for many document forms. shall be recorded. In such a state, the image processing apparatus 1 performs processing.

FIG. 2 is a diagram showing the hardware configuration of the image processing apparatus.
As shown in FIG. 2, the image processing apparatus 1 includes a CPU (Central Processing Unit) 11, an IF (Interface) 12, a communication module 13, a ROM (Read Only Memory) 14, a RAM (Random Access Memory) 15, a HDD (Hard Disk Drive) 16 or the like. The communication module 13 may perform wireless communication with the image reading device 2, the recording device 3, and the database 4, or may perform wired communication. good.

FIG. 3 is a functional block diagram of the image processing apparatus.
The CPU 11 of the image processing apparatus 1 has functions of a control unit 101, an acquisition unit 102, a feature amount extraction unit 103, a reading target feature amount generation unit 104, and a recording unit 105 by executing a stored program.

A control unit 101 controls other functional units.
The acquisition unit 102 acquires image data of a document form.
A feature amount extraction unit 103 extracts a feature amount representing a feature of a recorded character string included in image data of a document form for each image data of the document form, based on results of character recognition processing of image data of a plurality of document forms. .
The reading target feature quantity generation unit 104 generates a first feature quantity of the recorded character string using the feature quantity corresponding to the image data of the document form.
The recording unit 105 uses the first feature quantity to extract and record the recorded character string from the character string information read from the image data of the new document form.

Through such processing, the image processing apparatus 1 reduces labor for recording character string information to be recorded contained in image data of a new document form.

FIG. 4 is a diagram showing an example of a document form.
As shown in this figure, in the document form, the mark of the company that created the document, the date of creation, the person in charge of creation, and the content of the document are described in a format unique to the document form. For example, if the document form is an order form, the document content indicates one or more sets of information such as the name of the ordered product and the number of ordered products. Based on one document form, the operator uses the recording device 3 to record a specific record character string to be recorded among the character strings described in the document form in the database 4 . Specifically, the operator inputs a specific record character string to be recorded in the database 4 by the recording device 3 while looking at the document form. Also, the operator causes the image reading device 2 to read the image data of the document form. The document form is read by the image reading device 2 and output to the image processing device 1 based on the operator's operation. Then, based on the operation of the operator and the control of the image processing apparatus 1, the recording device 3 stores image data for one document form and specific recording characters to be recorded among the character strings described in the document form. The columns are associated and recorded in the database 4. In the example of FIG. 4, the date 51, supplier 52, product name 53, quantity 54, and amount 55 are specific record character strings to be recorded. Other information such as non-recorded character strings that are not recorded by the operator are also printed on the document form 5 . The information includes, for example, the name 501 of the orderer who issued the document, the emblem image 502 of the orderer, the title 503 of the document, the greeting 504, and the like.

FIG. 5 is a diagram showing an outline of a data table stored in the database.
As shown in FIG. 5, the database 4 associates the image data of the document form with a specific recorded character string to be recorded among the character strings described in the document form and stores them in a recording table.

<First Embodiment>
FIG. 6 is a first diagram showing the processing flow of the image processing apparatus according to the first embodiment.
Next, the processing flow of the image processing apparatus 1 will be explained step by step.
First, in the database 4, a combination of image data for a certain document form and a specific record character string described in the document form is recorded for a plurality of the same document forms. For example, it is assumed that specific recorded character string information for the document form 5 shown in FIG. 4 is recorded for the same multiple sheets. In such a state, the operator activates the image processing apparatus 1 and instructs the image processing apparatus 1 to start processing.

The acquisition unit 102 of the image processing apparatus 1 reads the information of the image data of the document form 5 and the recorded character string corresponding to the image data from the database 4 (step S601). Acquisition unit 102 outputs the image data and the recorded character string to feature amount extraction unit 103 . The feature amount extraction unit 103 performs OCR processing on the image data to detect all character strings in the image data and coordinates in the image data indicating the range of the character strings (step S602). A character string is a group of characters composed of a plurality of characters. The feature amount extracting unit 103 analyzes the range of one group based on the interval with other characters, etc., extracts one or more characters included in the range as a character string, and extracts the character string in the image data. Find the coordinates that indicate the extent of . Characters included as character strings may include symbols such as ideograms and phonetic characters, marks, icon images, and the like.

The feature quantity extraction unit 103 compares the character string extracted from the image data by OCR processing with the recorded character string read from the database 4 together with the image data. Among the character strings extracted from the image data by OCR processing, the feature amount extraction unit 103 extracts character strings in the image data that match character information of the recorded character strings, attributes of the characters included in the character strings, and their range. coordinates are specified (step S603). Character attributes are information represented by numbers, alphabets, hiragana, kanji, number of characters, character height, font, and the like. The coordinates of the character string range are information indicating the coordinates of the leading character and the coordinates of the ending character included in the character string. The feature amount extraction unit 103 generates feature amounts including the specified information for a plurality of identical document forms 5 (step S604).

A feature amount is generated for each recorded character string in the document form 5 by the feature amount extraction unit 103 . The reading target feature amount generation unit 104 acquires the feature amounts of the plurality of identical document forms 5, analyzes the character attributes included in each feature amount corresponding to the recorded character string, and the coordinates indicating the range of the character string. Each generates one feature quantity. Analysis of this feature amount is performed by processing such as machine learning. Machine learning is also simply called learning.
A feature amount generated by the reading target feature amount generation unit 104 through analysis such as machine learning is referred to as a first feature amount. That is, the reading target feature quantity generation unit 104 uses a plurality of sheets of the same document form 5 to generate the first feature quantity for each of the recorded character strings in the document form 5 (step S605). The first feature quantity is a feature quantity for recognizing a recorded character string, and includes character attributes and coordinates indicating the character string range. The reading target feature amount generation unit 104 records the first feature amount of each of one or more recorded character strings in the document form 5 in the database 4 in association with the identifier of the document form 5 (step S606).

For example, the reading target feature value generation unit 104 generates coordinates indicating character attributes and character string ranges of the date 51, orderer 52, product name 53, quantity 54, and amount 55, which are recorded character strings included in the document form 5, respectively. , etc., are linked to the identifier of the document form 5 and recorded in the database 4 .

Through the above processing, the image processing apparatus 1 can generate information (first feature value) to be used for reducing the labor of the operator to record the record character string by machine learning or the like, and store the information in the database 4. . As a result, the image processing apparatus 1 can automatically record the record character string in the database 4 based on the image data of the new document form. The processing will be described below.

FIG. 7 is a second diagram showing the processing flow of the image processing apparatus according to the first embodiment.
The operator performs an operation to cause the image reading device 2 to read a new document form. As a result, the image reading device 2 generates image data of the document form and outputs it to the image processing device 1 . The acquisition unit 102 of the image processing apparatus 1 acquires image data (step S701). Acquisition unit 102 outputs the image data to feature amount extraction unit 103 . The feature quantity extraction unit 103 performs OCR processing on the image data to detect a character string, features of characters included in the character string, and coordinates in the image data within the range of the character string (step S702). The feature amount extraction unit 103 generates a third feature amount including the detected information for each character string in the image data (step S703). That is, the third feature amount is information indicating the feature of the character string included in the document form of the newly read image data. After that, the group identification unit 107 reads out one or more first feature amounts of the document form of the image data to be processed from the database 4 (step S704). The group specifying unit 107 outputs the third feature amount and the first feature amount to the recording unit 105 .

The recording unit 105 acquires a third feature amount and one or more first feature amounts for one or more character strings in the image data. The recording unit 105 uses the coordinates indicating the range of the character string included in each first feature amount to determine whether or not there are all third feature amounts having coordinates corresponding to the coordinates indicated by each first feature amount. (Step S705). When there are all the third feature values having coordinates corresponding to the coordinates of the first feature values, characters are written in all items in the document form corresponding to the recorded character string. On the other hand, when there are no third feature values having coordinates corresponding to the coordinates of each first feature value, there is no description of characters in any item in the document form.

In the case of YES in step S705, the recording unit 105 determines whether or not the character attribute included in the first feature amount matches the character attribute included in the corresponding third feature amount specified based on the coordinates. (Step S706).

If the determination result in step S706 is YES and the character attributes match, the recording unit 105 adds a rectangular frame to the range of the recorded character string based on the coordinates indicated by one or more third feature values in the currently processed image data. Generate a confirmation screen that displays The recording unit 105 outputs the confirmation screen to the monitor (step S707). The operator can confirm the character string to be recorded by the image processing apparatus 1 by confirming the rectangular area displayed on the confirmation screen. This allows the operator to check whether the recorded character string is sufficient. An icon image of either OK or NG button is displayed on the confirmation screen. By selecting the OK button from the icon images of these buttons, the operator can indicate that the selection as the record character string is sufficient. On the other hand, by selecting the NG button from the icon images of the buttons, the operator can indicate that the selection as the record character string is insufficient.

The reason why the confirmation screen is output to the monitor will be explained with reference to FIG. In FIG. 4, six product names 53 are entered in the recorded character string. If the number of entries of 6 product names 53 is the maximum in the past document form, the product name 53 is automatically determined as a recorded character string within the range of 1 to 6 in the new document form. Therefore, for example, when seven product names 53 are described in a new form, since both steps S705 and S706 are YES for the first to sixth parts, the image processing apparatus 1 It exits without recording the string. In order to improve such an event, the image processing apparatus 1 displays a confirmation screen before recording the record character string in step S707, and asks the operator whether or not it is OK to finish recording. confirmation.

The recording unit 105 determines whether or not the selected character strings to be recorded are insufficient in response to the pressing of the icon image of the button by the operator (step S708). If there is no shortage, the recording unit 105 records the character string included in the third feature quantity in the recording table in association with the identification information of the document form (step S709).

For example, assume that a third feature amount a3, a third feature amount b3, a third feature amount c3, and a third feature amount d3 have been acquired from the image data of the document form. Then, the third feature amount a3 is the first feature amount a1 recorded in advance in the database, the third feature amount b3 is the first feature amount b1, the third feature amount c3 is the first feature amount c1, and the third feature amount It is assumed that the feature amount of the quantity d3 matches the first feature amount d1. In this case, the recording unit 105 records the character strings included in the third feature amount a3, the third feature amount b3, the third feature amount c3, and the third feature amount d3 in the record table of the document form.

In the case of NO in step S705, or in the case of NO in step S706, or in the case of NO in step S708, the recording unit 105 determines that a third feature having coordinates corresponding to the coordinates indicated by the first feature exists. If not, perform processing. Specifically, the recording unit 105 generates input image data of a form image in which an input field is provided in the range of coordinates of the first feature amount for which the third feature amount of the corresponding coordinates in the image data did not exist. output to the monitor (step S710). The input image data may be data described in a markup language such as HTML or XML. While viewing the input image data, the operator operates an input device such as a keyboard of the image processing apparatus 1 to input a record character string in the input field in the input image data displayed on the monitor. A save button is displayed on the input image data, and when the save button is pressed, the recording unit 105 newly inputs the third feature value already acquired for the document form into the input field of the input image data. A third feature amount including the character string is generated (step S711). The recording unit 105 associates the identifier of the form image data with the character string input in the input field and records them in the database 4 . The image processing apparatus 1 updates the first feature amount by executing the processing flow shown in FIG. 6 again, and can expand the range of character strings that can be automatically recorded. As a result, the next time the same document form is processed, the character string can be automatically recorded, saving the operator the trouble of inputting the character string. The recording unit 105 records the character strings included in all the third feature amounts in the recording table of the document form (step S712).

According to such processing, the image processing apparatus 1 automatically converts the recorded character string in the newly input image data of the document based on the image data of the document and the recorded character string recorded in advance by the operator. can be recorded in Therefore, the image processing apparatus 1 can reduce the labor of the operator for recording the record character string in the document form.
Also, even if the recorded character string is not described in the document form, if the description corresponding to the recorded character string that should be originally described is not described, the image processing apparatus 1 outputs the image data for input. . As a result, it is possible to find errors in items to be entered in the document form that have not been entered, and to easily record the record character string indicated by the items.

<Second embodiment>
FIG. 8 is a functional block diagram of the image processing device according to the second embodiment.
As shown in this figure, the image processing apparatus 1 according to the second embodiment has functions of a group classifying section 106 and a group identifying section 107 in addition to the functional sections shown in FIG. The hardware configuration of the image processing apparatus 1 is the same as the configuration shown in FIG.

FIG. 9 is a first diagram showing the processing flow of the image processing apparatus according to the second embodiment.
Next, the processing flow of the image processing apparatus 1 according to the second embodiment will be explained step by step. In the database 4, a large number of combinations of image data for a plurality of different document forms and specific recorded character strings described in each document form are recorded for each document form. For example, assume that a plurality of pieces of specific recorded character string information are recorded for each of the different document forms 5 shown in FIG. In such a state, the operator activates the image processing apparatus 1 and instructs the image processing apparatus 1 to start processing.

The acquisition unit 102 of the image processing apparatus 1 determines whether or not all the information of the image data of the document form 5 and the recorded character strings corresponding to the image data have been read from the database 4 (step S901). In the case of NO, the acquisition unit 102 reads the image data of the document form 5 and information of the recorded character string corresponding to the image data from the database 4 (step S902). Acquisition unit 102 outputs the image data and the recorded character string to feature amount extraction unit 103 . The feature amount extraction unit 103 performs OCR processing on the image data to detect all character strings in the image data and coordinates in the image data (step S903). A character string is a group of characters composed of a plurality of characters. The feature amount extracting unit 103 analyzes the range of one group based on the interval with other characters, etc., extracts one or more characters included in the range as a character string, and extracts the character string in the image data. Find the coordinates that indicate the extent of . Characters included as character strings may include symbols such as ideograms and phonetic characters, marks, icon images, and the like.

The feature quantity extraction unit 103 compares the character string extracted from the image data by OCR processing with the recorded character string read from the database 4 together with the image data. Among the character strings extracted from the image data by OCR processing, the feature amount extraction unit 103 extracts character strings in the image data that match character information of the recorded character strings, attributes of the characters included in the character strings, and their range. coordinates are identified (step S904). Character attributes are information represented by numbers, alphabets, hiragana, kanji, number of characters, character height, font, and the like. The coordinates of the character string range are information indicating the coordinates of the leading character and the coordinates of the ending character included in the character string. The feature quantity extraction unit 103 generates a feature quantity including the specified information for one document form 5 (step S905).

A feature amount is generated for each recorded character string in the document form 5 by the feature amount extraction unit 103 . That is, the feature quantity extraction unit 103 generates a feature quantity for each document form and for each recorded character string. The feature amount for each document form and for each recorded character string is called an individual first feature amount. The first individual feature amount may include either one or both of the character attribute and the coordinates indicating the range of the character string. The feature amount extraction unit 103 records the individual first feature amount of each of one or more recorded character strings in each document form 5 in the database 4 in association with the identifier of the document form 5 and the identifier of the recorded character string ( step S906). As the identifier of the recorded character string, for example, a coordinate value indicating the position of the recorded character string can be used.

For example, the feature amount extraction unit 103 extracts the character attributes of each of the date 51, orderer 52, product name 53, quantity 54, and amount 55, which are recorded character strings included in the document form 5, coordinates indicating the character string range, and the like. Each individual first feature quantity shown is recorded in the database 4 in association with the identifier of the document form 5 and the identifier of the recording character string.

The feature quantity extraction unit 103 also identifies non-recorded character strings in the image data that do not match the character information included in the recorded character strings, attributes of the characters included in the non-recorded character strings, and coordinates of the range. (Step S907). The feature quantity extraction unit 103 generates a feature quantity including the specified information for the document form 5 (step S908).

A feature amount is generated for each non-recorded character string in the document form 5 by the feature amount extraction unit 103 . That is, the feature quantity extraction unit 103 generates a feature quantity for each document form and for each non-recorded character string. The feature amount for each document form and for each non-recorded character string is called an individual second feature amount. The individual second feature amount may include either one or both of the character attribute and the coordinates indicating the range of the character string. The feature amount extraction unit 103 records the individual second feature amount of each of one or more non-recorded character strings in each document form 5 in the database 4 in association with the identifier of the document form 5 and the identifier of the non-recorded character string. (step S909). As the identifier of the non-recorded character string, for example, a coordinate value indicating the position of the recorded character string can be used.

For example, the feature amount extraction unit 103 extracts each individual second feature amount representing the name of the orderer 501, the emblem image of the orderer, the title 503 of the document form, the greeting sentence 504, etc., which are non-recorded character strings included in the document form 5. is recorded in the database 4 in association with the identifier of the document form 5 and the identifier of the recording character string.

Image data of a plurality of different document forms 5 and information of recorded character strings corresponding to the image data are recorded in the database 4 . The acquiring unit 102 of the image processing apparatus 1 repeats the processing of steps S901 to S911 until the image data and recorded character string information for all the document forms 5 are read. Then, it is assumed that it is determined that all have been read in step S901 (step S901: YES).

In this case, the grouping unit 106 groups the document forms 5 based on the individual second feature amount included in the image data of the document forms 5 (step S912). For example, the grouping unit 106 groups each document form 5 based on the degree of matching of non-recorded character strings indicated by the individual second feature amount, the degree of matching of emblem images, the degree of matching of coordinate ranges of non-recorded character strings, and the like. Divide. The grouping unit 106 determines the group identifier of the document form 5 in this grouping process. The grouping unit 106 determines whether all the document forms 5 have been grouped (step S913).

The grouping unit 106 repeats the process of step S912 if grouping of all the document forms 5 has not been completed. When grouping of all the document forms 5 is completed, the group classification unit 106 associates the identifier of the document form 5 with the group identifier given to the document form 5 and records them in the group table of the database 4. (Step S914).

Then, the reading target feature amount generation unit 104 reads each individual first feature amount and each individual second feature amount of one or a plurality of document forms 5 belonging to a certain group from the database 4, and reads each individual feature amount of the document forms 5 belonging to the group. Each group first feature amount and each group second feature amount corresponding to the first feature amount and each individual second feature amount are generated (step S915). Each group first feature amount may be a value such as an average of individual first feature amounts of the document forms 5 belonging to the group. Similarly, each group second feature amount may be a value such as the average of each individual second feature amount of the document forms 5 belonging to the group. Each group's first feature amount and each group's second feature amount are not the average of each individual first feature amount and the average of each individual second feature amount, but by using a method such as predetermined statistical processing or machine learning Each group first feature quantity, each A group second feature amount may be generated. The reading target feature amount generation unit 104 calculates each group first feature amount and each group second feature amount for each group, and records them in the database 4 in association with the identifier of the group (step S916).

Through the above-described processing, the image processing apparatus 1 can generate information necessary for each group of document forms and store it in the database 4 in order to reduce the labor of the operator to record the record character string. As a result, the image processing apparatus 1 can automatically record the record character string in the database 4 based on the image data of the new document form. The processing will be described below.

FIG. 10 is a second diagram showing the processing flow of the image processing apparatus according to the second embodiment.
The operator performs an operation to cause the image reading device 2 to read a new document form. As a result, the image reading device 2 generates image data of the document form and outputs it to the image processing device 1 . The acquisition unit 102 of the image processing apparatus 1 acquires image data (step S1001). Acquisition unit 102 outputs the image data to feature amount extraction unit 103 . The feature quantity extraction unit 103 performs OCR processing on the image data to detect a character string, features of characters included in the character string, and coordinates in the image data within the range of the character string (step S1002). The feature quantity extraction unit 103 generates a third feature quantity including the detected information for each character string in the image data (step S1003). The third feature amount is information indicating the feature of the character string included in the newly read document form of the image data.

Next, the group specifying unit 107 reads out the group second feature amount to be used for group specification of the new document form among the group second feature amounts from the database 4 . The group second feature amount may be, for example, a feature amount corresponding to the emblem image 502 of the orderer displayed in the image data of the document form. The group identification unit 107 determines whether or not information indicated by a certain group second feature amount can be identified from the image data of the document form acquired in step S1001. The group specifying unit 107 performs similar processing using the group second feature amount for all groups. When information matching the group second feature value read from the database 4 can be specified from the image data of the newly read document form, the group specifying unit 107 newly reads the group having the group second feature value. A group of image data of a document form is specified (step S1004). After that, the group specifying unit 107 reads out one or more group first feature amounts for the group from the database 4 (step S1005). The group specifying unit 107 outputs the third feature amount and one or more group first feature amounts to the recording unit 105 . The group first feature amount is a feature amount for specifying one or more recorded character strings in the document form belonging to the group.

The recording unit 105 acquires a third feature amount and one or more group first feature amounts for one or more character strings in the image data. The recording unit 105 uses the coordinates indicating the range of the character string included in each group first feature quantity to determine whether or not there are all third feature quantities having coordinates corresponding to the coordinates indicated by each group first feature quantity. Determine (step S1006). When there are all the third feature values having coordinates corresponding to the coordinates of the first feature value in each group, characters are written in all items in the document form corresponding to the recorded character string. On the other hand, when there are no third feature quantities having coordinates corresponding to the coordinates of the first feature quantity for each group, there is no description of characters in any item in the document form.

In the case of YES in step S1006, the recording unit 105 determines whether or not the character attribute included in the group first feature amount matches the character attribute included in the corresponding third feature amount specified based on the coordinates. (step S1007).

If the determination result in step S1007 is YES and the character attributes match, the recording unit 105 adds a rectangular frame to the range of the recording character string based on the coordinates indicated by one or more third feature values in the image data currently being processed. Generate a confirmation screen that displays The recording unit 105 outputs the confirmation screen to the monitor (step S1008). The operator can confirm the character string to be recorded by the image processing apparatus 1 by confirming the rectangular area displayed on the confirmation screen. This allows the operator to check whether the recorded character string is sufficient. An icon image of either OK or NG button is displayed on the confirmation screen. By selecting the OK button from the icon images of these buttons, the operator can indicate that the selection as the record character string is sufficient. On the other hand, by selecting the NG button from the icon images of the buttons, the operator can indicate that the selection as the record character string is insufficient.

The recording unit 105 determines whether or not the selected character strings to be recorded are insufficient in response to the pressing of the icon image of the button by the operator (step S1009). If there is no shortage, the recording unit 105 records the character string included in the third feature quantity in the recording table in association with the identification information of the document form (step S1010).

For example, assume that a third feature amount a3, a third feature amount b3, a third feature amount c3, and a third feature amount d3 have been acquired from the image data of the document form. Then, the third feature amount a3 is the group first feature amount g11 recorded in advance in the database, the third feature amount b3 is the group first feature amount g12, the third feature amount c3 is the group first feature amount g13, Assume that the third feature amount d3 matches the group first feature amount g14. In this case, the recording unit 105 records the character strings included in the third feature amount a3, the third feature amount b3, the third feature amount c3, and the third feature amount d3 in the record table of the document form.

If NO in step S1006, NO in step S1007, or NO in step S1009, the recording unit 105 determines that the third feature having coordinates corresponding to the coordinates indicated by the group first feature is Perform processing if it does not exist. Specifically, the recording unit 105 generates input image data of a form image in which input fields are provided in the range of coordinates of the group first feature amount for which the third feature amount of the corresponding coordinates in the image data did not exist. and output to the monitor (step S1011). The input image data may be data described in a markup language such as HTML or XML. While viewing the input image data, the operator operates an input device such as a keyboard of the image processing apparatus 1 to input a record character string in the input field in the input image data displayed on the monitor. A save button is displayed on the input image data, and when the save button is pressed, the recording unit 105 newly inputs the third feature value already acquired for the document form into the input field of the input image data. A third feature amount including the character string is generated (step S1012). The recording unit 105 associates the identifier of the form image data with the character string input in the input field and records them in the database 4 . By executing the processing flow shown in FIG. 9 again, the image processing apparatus 1 updates the group first feature amount and the group second feature amount, and can expand the range of character strings that can be automatically recorded. As a result, the next time the same document form is processed, the character string can be automatically recorded, saving the operator the trouble of inputting the character string. The recording unit 105 records the character strings included in all the third feature amounts in the recording table of the document form (step S1013).

According to such processing, the image processing apparatus 1 uses the image data and recorded character strings of a plurality of different document forms recorded in advance by the operator, regardless of the type of the newly input document form. A recording character string in image data of a document form can be automatically recorded. Therefore, the image processing apparatus 1 can reduce the labor of the operator for recording the record character string in the document form.
Also, even if the recorded character string is not described in the document form, if the description corresponding to the recorded character string that should be originally described is not described, the image processing apparatus 1 outputs the image data for input. . As a result, it is possible to find errors in items to be entered in the document form that have not been entered, and to easily record the record character string indicated by the items.

<Third Embodiment>
As another example of the processing of the image processing apparatus 1, the operator may register groups of document forms in the image processing apparatus 1 in advance. For example, when registering image data of a document form in the past, the operator entered a group identifier according to the type of the document form and registered it in the database 4 in association with the image data of the document form. As a result, different types of forms are not mixed in the same group due to a processing error of the image processing apparatus 1, etc., and it is possible to extract the group first feature amount and the group second feature amount with high accuracy. In this case, the operator inputs the group of the document form at the time of registration, but for the new form, the group is specified using the group second feature amount, as in step S1004.

<Fourth embodiment>
As another example of the processing of the image processing apparatus 1, in step S912, the image processing apparatus 1 not only groups document forms using the second individual feature amount, but also uses the first individual feature amount to Document forms may also be grouped using the individual first feature amount together with the individual second feature amount. The individual first feature quantity is the feature quantity of the recorded character string, but if the document forms are of the same type, the coordinates and character attributes of the recorded character string are considered to be the same. Forms can be grouped.
In this case, the acquisition unit 102 acquires a plurality of form image data and a recorded character string to be recorded among the character strings included in the form image data. Then, the grouping unit 106 groups the form image data based on the individual first feature amount. Then, the reading target feature amount generating unit 104 generates a group first feature amount of the recorded character string for each group using the individual first feature amount corresponding to the form image data included in the group.
Alternatively, the first grouping may be performed by the operator as shown in the third embodiment, and the new document form may be grouped using the individual first feature amount in the process of step S1004. This makes it possible to accurately read the recorded character string in OCR processing.

<Fifth Embodiment>
In the second embodiment, in step S1004, a new group of forms is specified based on the second feature amount. However, as another processing mode, the image processing apparatus 1 does not perform the process of specifying the groups, but sequentially specifies each group for all the groups set by the operator, and reads out the group first feature amount. , count the number of matches with the third feature quantity. In the case of the correct group, the first and third feature values of the group should match the most, so the image processing apparatus 1 determines the character string is recorded in step S1008. This makes it possible to record the recording string without specifying the group.
In this case, the image data of the document form stored in the database 4 for generating the group first feature amount may be grouped in advance by the operator. The acquisition unit 102 acquires a plurality of form image data and a recorded character string to be recorded among the character strings included in the form image data. Then, the feature amount extraction unit 103 extracts the individual first feature amount indicating the feature of the recorded character string based on the character recognition processing result of the form image data acquired by the acquisition unit 102 . The reading target feature amount generation unit 104 generates a group first feature amount of a recorded character string for each group using individual first feature amounts corresponding to form image data included in a predetermined group.

<Sixth embodiment>
In the processing of FIG. 7 of the first embodiment, the image processing apparatus 1 may perform machine learning of the feature amount both when character recognition succeeds and when character recognition fails. This point will be described in the sixth embodiment.
The sixth embodiment is the same as the first embodiment except for the processing in FIG. 7 of the first embodiment, and FIGS.

FIG. 11 is a diagram showing the processing flow of the image processing apparatus according to the sixth embodiment. Steps S1101 to S1108 in FIG. 11 are the same as steps S701 to S708 in FIG. Steps S1110-1112 in FIG. 11 are the same as steps S709-S711 in FIG. Step S1114 in FIG. 11 is the same as step S712 in FIG.
Therefore, in the process of FIG. 11, step S1109 is inserted between steps S708 and S709 of FIG. 7, and step S1113 is inserted between steps S711 and S712 of FIG. Similar to 7.
In both steps S1109 and S1113, the reading target feature amount generation unit 104 of the image processing apparatus 1 performs the processing of FIG. 6 to update the first feature amount. The reading target feature amount generation unit 104 does not need to redo the processing in FIG. 6 from the unlearned state, and may perform additional learning for one image data.

Here, machine learning is not performed in step S1109 of FIG. can be considered. However, in this case, if the image processing apparatus 1 succeeds in OCR processing, machine learning may not be performed, and as a result, statistical information on the format of the document form may not be reflected in machine learning.
For example, consider a case where 99 out of 100 document forms have a recorded character string in the lower left corner and only one sheet has a recorded character string in the upper right corner. In this case, if the image processing apparatus 1 succeeds in OCR processing for a document form with the recorded character string at the same position (step S1108: YES), if the recorded character string is at the lower left, the upper right In each case, learning is performed once. Although the ratio is actually 99:1, the image processing apparatus 1 learns at a ratio of 1:1, and there is a possibility of over-learning when the recorded character string is in the upper right corner. be.
On the other hand, as shown in FIG. 11, both when the image processing apparatus 1 succeeds in OCR processing and when it fails, machine learning (updating the first feature value) is performed to statistically analyze the format of the document form. Information can be reflected in machine learning.
Similarly, in the second to fifth embodiments, the image processing apparatus 1 performs the processing of FIG. 9 between steps S1009 and S1010 and between steps S1012 and S1013 of FIG. good too.

<Seventh Embodiment>
The image processing apparatus 1 may acquire data from a system that scans or photographs documents, such as an existing OCR system, and performs machine learning. This point will be explained in the seventh embodiment. A case where the image processing apparatus 1 acquires data from an existing OCR system will be described below as an example. However, the data provider to the image processing apparatus 1 is not limited to the existing OCR system, and can be various systems or devices. For example, the image processing apparatus 1 may acquire data from a system (for example, a sales information system or an accounting information system) that stores the date of sale, the place of sale, the amount of money, and the image data that serves as a trail in a database. good. In particular, it is not necessary that the system or device providing the information has OCR capabilities.

FIG. 12 is a diagram showing an example of connection between an image processing system according to the seventh embodiment and an existing OCR system. In the configuration shown in FIG. 12, an existing OCR system 9 is connected to the image processing apparatus 1 in addition to the configuration shown in FIG. With this configuration, the existing OCR system 9 uses the image data generation function (scan function) and user interface, and does not use the character recognition function. Character recognition is performed by the image processing apparatus 1 . Other points are the same as the first embodiment. With reference to FIGS. 1 to 7, acquisition of image data and user interface are read as those using the existing OCR system 9. FIG.

FIG. 13 is a diagram showing the processing flow of the image processing apparatus according to the seventh embodiment. FIG. 13 shows the relationship between the processing performed by the image processing apparatus 1 and the processing performed by the existing OCR system 9 . In the process of FIG. 13, the existing OCR system scans according to the operator's operation and generates image data (sequence S1301). The existing OCR system 9 transmits the obtained image data to the image processing apparatus 1 (sequence S1302).
Upon receiving the image data, the image processing apparatus 1 performs OCR processing (sequence S1303). For example, the image processing apparatus 1 performs the processing from step S702 onward in FIG. 7 to acquire the recorded character string. In particular, the recording unit 105 identifies a recorded character string from the character strings resulting from the OCR processing.
The image processing apparatus 1 transmits the character string (recorded character string) of the character recognition result to the existing OCR system 9 (sequence S1304).
The existing OCR system 9 presents the character recognition result to the operator, and receives confirmation operation after confirming and correcting the character recognition result (sequence S1305). The existing OCR system 9 transmits the confirmed character string (recorded character string of correction result) to the image processing apparatus 1 (sequence S1306).
Upon receiving the correction result, the image processing apparatus 1 performs machine learning (sequence S1307).
The image processing apparatus 1 acquires the image data in sequence S1302, and acquires the character string of the correction result confirmed by the operator in sequence S1306. Using this image data and character strings, machine learning can be performed by, for example, the processing procedure shown in FIG. The image processing apparatus 1 may perform machine learning collectively after a certain amount of data is accumulated, or may perform additional learning each time data is obtained.
Thus, according to the seventh embodiment, the image processing apparatus 1 can perform machine learning using the existing OCR system 9 commonly used by workers. In particular, the reading target feature quantity generation unit 104 can update the first feature quantity.
Similarly, in the second to sixth embodiments, the image processing apparatus may perform machine learning using an existing OCR system.

<Eighth Embodiment>
The learning function of the image processing apparatus may be implemented as a server to accelerate learning. This point will be explained in the eighth embodiment.
FIG. 14 is a diagram showing an outline of an image processing system 200 according to the eighth embodiment. In the configuration of FIG. 14, the image reading device 2 is the same as in the case of FIG.
The functions of the image processing apparatus 1 shown in FIG. 1 are divided into a learning function section 1b that performs machine learning and an OCR function section 1a that performs other functions, forming a server client configuration. In particular, the learning function unit 1b generates and updates the first feature amount. The learning function unit 1b also executes the functions of the recording device 3 and manages the learning result database 4b. The OCR function unit 1a executes functions such as OCR processing, extraction of the third feature amount, identification of recorded character strings, and user interface for the operator. The OCR function unit 1a performs OCR processing to detect a character string, detect characteristics of characters included in the character string, and detect coordinates within the range of the character string.
The OCR function section 1 a is stored in the terminal device 6 . The learning function unit 1b is stored in the server device 7. FIG.
The database 4 in FIG. 1 is divided into an OCR result database 4a and a learning result database 4b. The OCR result database 4a stores results of OCR processing such as a recording table. The learning result database 4b stores data obtained during machine learning, such as the first feature amount. The learning result database 4 b is stored in the server device 7 . On the other hand, the OCR result database is configured as one device.

FIG. 15 is a diagram showing the processing flow of the terminal device according to the eighth embodiment.
In the process of FIG. 15, the terminal device 6 reads the image data of the document form using the image processing device 2, and downloads the learning result corresponding to the image data from the server device 7 (step S1501). In particular, the terminal device 6 acquires the first feature amount as a learning result.
The terminal device 6 performs OCR processing on all character strings included in the image data, and displays on the screen those that match the learning result (step S1502). Specifically, the terminal device 6 uses the first feature amount, which is the learning result, to identify a recorded character string that matches the first feature amount.

The terminal device 6 determines whether all OCR target character strings in the image data are correctly read (step S1503). If it is determined that there is a character string that has not been read (step S1503: NO), the terminal device 6 presents the OCR result to the operator on screen display and receives a correction operation (step S1504).
The terminal device 6 transmits and records the read character string (or the corrected character string if corrected by the operator) to the OCR result database 4a (step S1505). Specifically, the terminal device 6 stores the recorded character string in the OCR result database 4a. If YES in step S1503, the process also proceeds to step S1505.
The terminal device 6 transmits learning data such as the read character string and position information to the server device 7 (step S1506).
15, the terminal device 6 records the recorded character string to be recorded as the OCR result in the OCR result database, and also provides the server device 7 with learning data.

FIG. 16 is a diagram showing the processing flow of the server device according to the eighth embodiment.
In the process of FIG. 16, the server device 7 receives and stores learning data from the terminal device 6 (step S1601).
The server device 7 determines whether N (N is a positive integer) unlearned data for learning have been accumulated (step S1602). If it is determined that the data is not accumulated (step S1602: NO), the process returns to step S1601.
On the other hand, if it is determined that N pieces of unlearned learning data have been accumulated (step S1602: YES), the server apparatus 7 reads the reception result data of the form received as learning data from the learning result database (step S1603). .
Next, the server device 7 adds new learning data to the read learning result data, and re-learns (step S1604).
The server device 7 registers the re-learning result in the learning result database 4b (step S1605).
Then, the server device 7 waits until learning data is received from the terminal device 6 (step S1606). After step S1606, the process returns to step S1601.
In the process of FIG. 16, the server device 7 updates the first feature quantity as the learning result. The updated first feature amount is registered in the learning result database 4 b and used by the terminal device 6 .
According to the eighth embodiment, the learning function of the image processing apparatus can be implemented as a server to aggregate learning data and accelerate learning.

Note that the timing at which the server apparatus 7 performs machine learning such as re-learning is not limited to the timing at which a certain number of learning data is accumulated as shown in step S1602 of FIG. For example, the server device 7 may perform learning at regular intervals. In particular, the server device 7 may start learning according to time conditions in addition to or instead of the number of data conditions.

As an example in which the server device 7 performs learning at regular intervals, there is a case where the server device 7 performs learning at night or on holidays when the terminal device 6 does not operate. Here, when the server device 7 performs learning while the terminal device 6 is in operation, the learning function unit 1b of the server device 7 accesses the learning result database. As a result, there is a possibility that the access from the terminal device 6 to the learning result database 4b will be delayed, or that access will not be possible at the required timing. Therefore, the server device 7 performs learning when the terminal device 6 is not in operation, thereby avoiding the influence on the access from the terminal device 6 to the learning result database 4b.

In addition to or instead of the server device 7, the terminal device 6 may perform machine learning.
Here, when the server device 7 performs machine learning and the terminal device 6 does not perform learning, the server device 7 performs machine learning, updates the learning result (first feature amount), and exchanges the learning result with the terminal device 6. The processing accuracy of the terminal device 6 does not improve until it is shared. For example, if the server device 7 performs machine learning only at night every day, the terminal device 6 can use the updated learning result only after the next morning. For example, even if the terminal device 6 generates learning data in the morning and transmits it to the server device 7, the result of the learning will not be reflected in the processing of the terminal device 6 until the next morning after about one day.

Therefore, each of the terminal devices 6 performs machine learning each time it processes one form. The terminal device 6 reflects the learning result in its own processing, and transmits the learning result or learning data to the server device 7 . The terminal device 6 generates a feature amount (third feature amount) by processing the form. Then, the terminal device 6 updates the first feature amount acquired from the learning result database using the generated feature amount. For example, in step S1506 of FIG. 15, the terminal device 6 transmits the learning data to the server device 7, performs machine learning by itself using the learning data, Update features. Alternatively, the terminal device 6 may transmit the learning result (obtained feature amount) to the server device 7 in addition to the learning data in step S1506 of FIG.
By learning by itself, the terminal device 6 can reflect its own learning result in its own processing. In this respect, it is expected that the processing accuracy of the terminal device 6 will be improved more quickly than when the terminal device 6 itself does not perform machine learning and waits for the update of the learning result by the server device 7 . Machine learning performed by each terminal device 6 is referred to as provisional learning.

When the terminal device 6 transmits learning data to the server device 7, the processing performed by the server device 7 is the same as described with reference to FIG.
On the other hand, when the terminal device 6 transmits the learning result to the server device 7 , the server device 7 accumulates the learning result from each terminal device 6 . Then, the server device 7 reflects the learning results obtained from the terminal device 6 at night, for example, in the learning results stored in the learning result database 4b. Specifically, the learning function unit 1b of the server device 7 updates the first feature quantity stored in the learning database 4b using the feature quantity acquired by the terminal device 6 through learning.
A process of updating the first feature amount stored in the learning result database is referred to as main learning.

FIG. 17 is a diagram showing the minimum configuration of the image processing apparatus.
As shown in this figure, the image processing apparatus 1 may include at least a feature amount extraction unit 103 and a reading target feature amount generation unit 104 .
The feature quantity extraction unit 103 extracts a feature quantity representing a feature of a character string included in the form image data for each form image data, based on the results of character recognition processing for a plurality of form image data registered in the past.
The reading target feature amount generation unit 104 uses the feature amount corresponding to the form image data to generate the first feature amount of the recorded character string in the form image data.

Each of the devices described above has an internal computer system. A program for causing each device to perform each process described above is stored in a computer-readable recording medium of each device. done. Here, the computer-readable recording medium refers to magnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, semiconductor memories, and the like. Alternatively, the computer program may be distributed to a computer via a communication line, and the computer receiving the distribution may execute the program.

Further, the program may be for implementing part of the functions of the processing units described above. Further, it may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

Reference Signs List 1 Image processing device 1a OCR function unit 1b Learning function unit 2 Image reading device 3 Recording device 4 Database 4a Learning result database 4b OCR result database 5 Document form 6 Terminal device 7 Server device 101 Control unit 102 Acquisition unit 103 Feature amount extraction unit 104 Read target feature amount generation Unit 105 Recording unit 106 Group classification unit 107 Group identification unit 108 Work target determination unit 109 Work data generation unit

Claims (7)

an acquisition unit for acquiring image data of a document form and a recorded character string recorded by an operator among character strings included in the image data;
a feature amount extracting unit for extracting a feature amount of the character string that matches the recorded character string from among the character strings recognized by character recognition processing on the image data;
a read target feature quantity generation unit that performs machine learning on the feature quantity of the character string extracted from the image data of each of a plurality of document forms having the same format to generate a read target feature quantity;
When the feature amount extracted from the character string recognized by the character recognition processing for the image data of the new document form acquired by the acquisition unit matches the feature amount of the reading target, the new document form is obtained. a recording unit that records the character string recognized from the image data of the
with
The image processing apparatus, wherein the feature amount is information indicating an attribute of the character string and a range of the character string in the image of the document form. The reading target feature amount generation unit machine-learns the feature amount of the character string extracted from the image data for each group in which the image data of each of the plurality of document forms is grouped in advance, and performs machine learning on each group. The image processing apparatus according to claim 1, wherein the feature amount of the reading object of is generated. Grouping the image data of each of the plurality of document forms based on the degree of matching indicating the degree of mutual matching of the feature quantity of the character string that matches the recorded character string extracted from the image data of each of the plurality of document forms. a group classification unit for dividing;
The image processing apparatus according to claim 2, comprising: The feature quantity extraction unit further extracts a feature quantity of a character string other than the character string matching the recorded character string,
each of the plurality of document forms based on the degree of matching indicating the degree of mutual matching of feature quantities of character strings other than the character string matching the recorded character string, extracted from the image data of each of the plurality of document forms; a grouping unit for grouping image data;
The image processing apparatus according to claim 2, comprising: The reading target feature quantity generation unit generates a feature quantity for each group based on a feature quantity of a character string other than the character string matching the recorded character string,
a group identification unit that identifies the group to which the image data of the new document form belongs based on the feature amount of each group and the feature amount extracted from the image data of the new document form; prepared,
If the feature amount extracted from the image data of the new document form matches the feature amount of the read target of the specified group, the recording unit record recognized strings,
The image processing apparatus according to claim 4. Acquiring image data of a document form and a recorded character string recorded by an operator among the character strings included in the image data,
extracting a feature amount of the character string that matches the recorded character string, from among the character strings recognized by character recognition processing on the image data;
machine-learning the feature amount of the character string extracted from the image data of each of a plurality of document forms of the same format to generate the feature amount to be read;
When the feature amount extracted from the character string recognized by the character recognition processing for the image data of the new document form matches the feature amount of the reading target, the character string is recognized from the image data of the new document form. record the string
The image processing method, wherein the feature amount is information indicating an attribute of the character string and a range of the character string in the image of the document form. to the computer,
a step of acquiring image data of a document form and a recorded character string recorded by an operator among the character strings included in the image data;
a step of extracting a feature amount of the character string that matches the recorded character string, from among the character strings recognized by the character recognition process for the image data;
a step of machine-learning the feature amount of the character string extracted from the image data of each of a plurality of document forms of the same format to generate a feature amount to be read;
When the feature amount extracted from the character string recognized by the character recognition processing for the image data of the new document form obtained in the obtaining step matches the feature amount of the reading target, the new document is obtained. a step of recording a character string recognized from the image data of the form;
and
The program, wherein the feature amount is information indicating an attribute of the character string and a range of the character string in the image of the document form.

Images