JP2021144673A - Image processing apparatus, image processing method and program - Google Patents

Abstract

To extract an index of an extraction target even in a case where a position of a text block of a scan image is different from that of a registered document.SOLUTION: An image formation apparatus 100 comprises an image processing unit 305 including: acquisition means which acquires image data of a document; detection means which detects a text block in an image indicated by the image data; specification means which specifies a document corresponding to the image as a registration document on the basis of a prescribed rule; estimation means which estimates a text block of an item being a processing target in the image on the basis of a partial layout including the text block of the item of the processing target and at least one text block other than the text block of the item of the processing target out of the text blocks prescribed in the registration document; and extraction means which extracts a character string in the estimated text block as a character string of the item of the processing target.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to a technique for extracting an index contained in an image.

There is a method of extracting a character string (hereinafter referred to as an index) of a desired item included in a scanned image obtained by scanning a paper document such as a form with an image reading device. OCR processing is required to extract the index from the contents of the document. However, when the OCR process is executed on the entire scanned image, the processing load increases and the waiting time of the user may increase.

Patent Document 1 describes a method in which information on an area including an index is registered in advance for each type of document, OCR processing is partially performed on the registered index area, and an index is extracted from a scanned image. It is disclosed.

Japanese Unexamined Patent Publication No. 2019-128715

However, even for the same type of document, the position of the character string area (hereinafter referred to as a text block) including the index may be shifted depending on the contents described. Therefore, even if the OCR process is partially performed on the registered index area, the index extraction may fail.

The technique of the present disclosure aims to extract an index to be extracted even when the position of the text block of the scanned image deviates from the registered position.

The image processing apparatus of the present disclosure defines an acquisition means for acquiring image data of a document, a detection means for detecting a text block in an image indicated by the image data, and a layout of a text block for each document group. Among the above, a specific means for specifying the document corresponding to the image as a registered document based on a predetermined rule, a text block corresponding to an item to be processed among the text blocks specified in the registered document, and a text block corresponding to the item to be processed. An estimation means for estimating a text block corresponding to the item to be processed in the image based on a partial layout having a layout including at least one text block other than the text block corresponding to the item to be processed. It is characterized by having an extraction means for extracting a character string in the estimated text block as a character string corresponding to the item to be processed.

According to the technique of the present disclosure, even if the position of the text block of the scanned image is different from the registered document, the index to be extracted can be extracted.

It is a figure which shows the configuration example of a system. It is a figure which shows the hardware configuration example of the image forming apparatus. It is a figure which shows the functional structure of an image forming apparatus. It is a flowchart of a file generation process of a scanned image. It is a flowchart of index extraction processing. It is a figure which shows the example of a block selection process. It is a figure which shows the example of the index extraction rule. It is a flowchart of index block estimation processing. It is a figure explaining the determination method of a pair block. It is a figure which shows the example of a partial pattern. It is a figure explaining the determination process of the Y candidate position. It is a figure which shows the example of the histogram of the shift amount in the Y direction. It is a figure explaining the calculation of the degree of coincidence of a partial pattern. It is a figure explaining the calculation of the degree of coincidence of a partial pattern. It is a figure explaining the method of determining a partial pattern range. It is a flowchart of index block estimation processing. It is a figure which shows the example of a partial pattern. It is a figure which shows the example of the XY candidate position group. It is a figure which shows the example of the similar position group. It is a figure explaining the correspondence between a similar position group and an XY candidate position group.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the technology of the present disclosure according to the claims, and all combinations of features described in the present embodiment are essential for solving the technology of the present disclosure. Not necessarily.

<Embodiment 1>
The image forming apparatus of the present embodiment scans a document document and generates a file name by combining character strings of predetermined items included in the image of the first page of the obtained scanned image. Then, the generated file name is recommended to the user as the file name of the scanned image. However, the processing load may increase in order to extract the character string of a predetermined item from the scanned image.

Therefore, the position information of the text block of a predetermined item is registered for each type of document. Then, it is conceivable to specify the document type of the scanned image and extract the character string of a predetermined item from the scanned image based on the position of the registered text block in the specified document. However, even in this case, even if the document type is the same, the position of the text block in the scanned image may be different from the registered position due to a change in the description content or the like.

For example, it is assumed that the document of FIG. 11A is registered and the information indicating the position of the text block 1003 is registered as the information of the area including the character string indicating the issuing company name. On the other hand, FIG. 11B is a scanned image obtained by scanning the same type of document as in FIG. 11A, but the number of item lines in the table structure is increasing, and the publisher to be extracted is The text block 1101 of the company name is shifted downward as compared with FIG. 11A. Therefore, even if it can be identified that the document scanned to obtain the scanned image of FIG. 11 (b) is of the same type as that of FIG. 11 (a), the character indicating the issuing company name of the image of FIG. 11 (b). Column extraction may fail. The description of FIG. 11 (c) will be described later.

Therefore, in the embodiment, in order to extract the text block of the item included in the scanned image, the layout of the text block indicating the item in the document of the same type as the scanned document manuscript and at least one other text block. Is used. In the present embodiment, a method of searching the scanned image for a region having a high degree of matching with the layout and estimating the text block of the item included in the scanned image based on the searched result will be described.

In the present embodiment, for example, a coordinate system is used in which the origin is in the upper left, the vertical direction is in the Y direction, and the horizontal direction in which the character strings are continuous extends in the X direction. As for the position of the text block, for example, the upper left coordinate value is held as each position.

[System configuration]
FIG. 1 is a diagram showing an overall configuration of a system to which this embodiment can be applied. The system 105 of this embodiment has an image forming apparatus 100 and a terminal 101. As shown in FIG. 1, the image forming apparatus 100 is connected to the LAN 102 and can communicate with a terminal 101 or the like such as a PC via the Internet 103 or the like. In this embodiment, the terminal 101 may not be provided, or only the image forming apparatus 100 may be configured.

The image forming apparatus 100 is a multifunction device (MFP) having a display / operation unit 123 (see FIG. 2), a scanner unit 122 (see FIG. 2), a printer unit 121 (see FIG. 2), and the like. The image forming apparatus 100 can be used as a scanning terminal for scanning a document document using the scanner unit 122. In addition, it has a display / operation unit 123 such as a touch panel and a hard button, and displays a file name and a recommendation result of a storage destination, and displays a user interface for receiving an instruction from a user.

[Hardware configuration of image forming apparatus]
FIG. 2 is a block diagram showing a hardware configuration of the image forming apparatus 100. The image forming apparatus 100 of the present embodiment includes a display / operation unit 123, a scanner unit 122, a printer unit 121, and a control unit 110.

The control unit 110 includes a CPU 111, a storage device 112 (ROM 118, RAM 119, HDD 120), a printer I / F unit 113, a network I / F unit 114, a scanner I / F unit 115, and a display / operation I / F unit 116. Further, in the control unit 110, these units are connected to each other so as to be able to communicate with each other via the system bus 117. The control unit 110 controls the operation of the entire image forming apparatus 100.

The CPU 111 functions as a means for executing each process such as reading control, image processing, and display control in the flowchart described later by reading and executing the control program stored in the storage device 112.

The storage device 112 stores and holds a control program, image data, metadata, setting data, processing result data, and the like. The storage device 112 includes a ROM 118 which is a non-volatile memory, a RAM 119 which is a volatile memory, an HDD 120 which is a large capacity storage area, and the like. The ROM 118 is a non-volatile memory that holds a control program and the like, and the CPU 111 reads and controls the control program. The RAM 119 is a volatile memory used as a temporary storage area such as a main memory and a work area of the CPU 111.

The network I / F unit 114 connects the control unit 110 (image forming apparatus 100) to the LAN 102 via the system bus 117. The network I / F unit 114 transmits image data to an external device on the LAN 102, and receives various information from the external device on the LAN 102.

The scanner I / F unit 115 connects the scanner unit 122 and the control unit 110 via the system bus 117. The scanner unit 122 reads the document document to generate scanned image data, and inputs the scanned image data to the control unit 110 via the scanner I / F unit 115. The scanner unit 122 is provided with a document feeder, and can feed a plurality of documents placed on the tray one by one and continuously read them.

The display / operation I / F unit 116 connects the display / operation unit 123 and the control unit 110 via the system bus 117. The display / operation unit 123 is provided with a liquid crystal display unit having a touch panel function, hard buttons, and the like.

The printer I / F unit 113 connects the printer unit 121 and the control unit 110 via the system bus 117. The printer unit 121 receives the image data generated by the CPU 111 via the printer I / F unit 113, and prints on the recording paper using the received image data. As described above, the image forming apparatus 100 according to the present embodiment can provide an image processing function by the above hardware configuration.

[Functional configuration of image forming apparatus]
FIG. 3 is a block diagram showing a functional configuration of the image forming apparatus 100. Note that FIG. 3 shows a function focusing on the functions related to the processing of scanning the document document, digitizing it (file-making), and saving it, among the various functions of the image forming apparatus 100.

The display control unit 301 displays a user interface screen (UI screen) for accepting various user operations on the touch panel of the display / operation unit 123. Various user operations include, for example, scan settings, scan start instructions, file name settings, file save instructions, and the like.

The scan control unit 302 instructs the scan execution unit 303 to execute the scan process together with the scan setting information in response to the user operation (for example, pressing the "scan start" button) performed on the UI screen. The scan execution unit 303 causes the scanner unit 122 to execute the scanning operation of the document document via the scanner I / F unit 115 in accordance with the execution instruction of the scan process from the scan control unit 302, and generates the scanned image data. The generated scan image data is stored in the HDD 120 by the scan image management unit 304.

The image processing unit 305 performs image analysis processing such as text block detection processing, OCR processing (character recognition processing), and determination processing of similar documents, as well as image processing processing such as rotation and tilt correction, on the scanned image data. The image processing unit 305 also causes the image forming apparatus 100 to function as an image processing apparatus. The character string area detected from the scanned image is also called a "text block". The details of image processing will be described later.

The functions of each part of FIG. 3 are realized by the CPU of the image forming apparatus 100 expanding the program code stored in the ROM into the RAM and executing the program code. Alternatively, some or all the functions of each part of FIG. 3 may be realized by hardware such as an ASIC or an electronic circuit.

[Flowchart of scanned image file generation process]
The image forming apparatus 100 reads the document document, performs image processing on the scanned image of the first page of the document document, generates a file name using the character string included in the scanned image, and the user passes through the display / operation unit 123. The whole process of recommending to is described.

The series of processes shown in the flowchart of FIG. 4 is performed by the CPU of the image forming apparatus 100 expanding the program code stored in the ROM into the RAM and executing it. Further, some or all the functions of the steps in FIG. 4 may be realized by hardware such as an ASIC or an electronic circuit. The symbol "S" in the description of each process means that the step is a step in the flowchart, and the same applies to the subsequent flowcharts.

In S400, when the scan control unit 302 receives a user's scan instruction via the display / operation unit 123, the scan execution unit 303 reads a plurality of document documents one by one from the tray of the document feeder of the scanner unit 122 (scanning). ) Is executed. Then, the scan control unit 302 acquires image data of an image (called a scanned image) obtained as a result of scanning.

In S401, the image processing unit 305 analyzes the image data acquired in S400 and performs a process (index extraction process) of extracting an index included in the scanned image. The "index" is a character string of a predetermined item such as a document title, a management number, and a company name. In this embodiment, the index is used as a file name or metadata when storing the scanned image. The details of the index extraction process in this step will be described later with reference to FIG.

Index usage is not limited to filename generation or metadata extraction. It may be used to set other property information such as a folder path. That is, the file name and metadata are a kind of information set as properties (attributes) related to the scanned image data.

In S402, the display control unit 301 generates a file name using the index extracted in S401, displays the generated file name and metadata on the display / operation unit 123, and presents (recommends) the user. In addition, the display control unit 301 accepts confirmation or correction of the presented file name by the user. When the display control unit 301 receives confirmation or correction from the user via the display / operation unit 123, the presented file name or, if corrected, the corrected file name is determined as the file name of the scanned image. When the user modifies it via the display / operation unit 123, the index extraction rule is updated. The index extraction rule will be described later.

In S403, the image processing unit 305 creates a file from the image data acquired in S400, and sets the file name determined in S402. In the present embodiment, as an example, it will be described as a file format in which the scanned image is stored in PDF (Portable Document Format). In the case of PDF, it is possible to divide the image data into pages and save them. When a plurality of document manuscripts are scanned in S400, the image data corresponding to each document manuscript is put into one file as separate pages. It will be saved.

In S404, the scan image management unit 304 transmits the file created in S403 to a predetermined destination via LAN 102.

[About index extraction processing (S401)]
FIG. 5 is a flowchart showing details of the index extraction process of S401. The details of the index extraction process will be described with reference to FIG. In the index extraction process, the orientation of one page of image data is corrected, the type of the document is specified, and the index is extracted according to the type of the document.

In S500, the image processing unit 305 detects the tilt angle of the scanned image from the image data, and corrects the tilt of the scanned image by rotating the image in the opposite direction by the detected tilt. The tilt to be corrected for tilt occurs because, for example, when scanning a document document, the document document cannot be read straight due to wear of a roller in the document feeder of the scanner unit 122. Or, it occurs because the scanned document document was not printed straight at the time of printing.

As a method of detecting the tilt angle, first, objects included in the image data are detected, and a group of adjacent objects in the horizontal direction or the vertical direction are connected. Then, how much the angle connecting the center positions of the connected objects is tilted from the horizontal direction or the vertical direction is derived to obtain the tilt. The method of detecting the inclination is not limited to this method. In addition, for example, the center coordinates of the objects included in the image data are acquired, the center coordinate group is rotated in units of 0.1 degrees, and the angle at which the center coordinate groups are lined up in the horizontal or vertical direction is the highest. It may be a method of obtaining the inclination of the scanned image. By correcting the inclination of the scanned image, it is possible to improve the processing accuracy of each of the subsequent rotation correction, block selection processing, and OCR processing.

In S501, the image processing unit 305 rotates and corrects the scanned image obtained as a result of the processing of S500 in units of 90 degrees so that the characters in the image are oriented upright. The rotation correction method is, for example, using the scanned image after tilt correction as a reference image, a reference image, an image obtained by rotating the reference image by 90 degrees, an image obtained by rotating the reference image by 180 degrees, and an image obtained by rotating the reference image by 270 degrees. And, prepare four images. Then, there is a method in which a simple OCR process capable of high-speed processing is executed for each image, and the image having the largest number of characters recognized with a certain degree of certainty or higher is used as the image after rotation correction. .. However, the rotation correction method is not limited to this method. The subsequent scanned images shall refer to the scanned images corrected by S500 and S501 unless otherwise specified.

In S502, the image processing unit 305 executes a block selection process on the scanned image. The block selection process is a process of classifying an image into a foreground area and a background area, dividing the foreground area into a text block and other blocks, and detecting the text block.

Specifically, contour line tracking is performed on a scanned image binarized to black and white, and a pixel block surrounded by a black pixel contour is extracted. Then, for a black pixel block having an area larger than a predetermined size, the outline of the white pixel inside is also traced to extract the white pixel block, and then the white pixel having an area larger than a certain size is extracted. A mass of black pixels is recursively extracted from the inside of the mass. The mass of black pixels thus obtained is determined as the foreground region. The determined foreground area is classified according to size and shape, and is classified into areas having different attributes. For example, a foreground region having an aspect ratio close to 1 and a constant size is defined as a pixel block corresponding to a character, and an region in which adjacent characters can be grouped in a well-aligned manner is determined to be a character string region (TEXT). A flat pixel block is determined as a line region (LINE). The table area (TABLE) is determined to be the range occupied by the black pixel clusters having a certain size or larger and containing the rectangular white pixel clusters in a well-aligned manner. A region in which irregular pixel clusters are scattered is determined as a photographic region (PHOTO). Then, a pixel block having a shape other than that is determined as a drawing area (PICTURE). The foreground area (TEXT) determined to have the character attribute is detected as a text block from the area divided for each attribute of the object.

FIG. 6 is a diagram showing an example of the result of the block selection process. FIG. 6A shows a scanned image after rotation correction. FIG. 6B shows the result of the block selection processing for the scanned image of FIG. 6A, and the rectangle shown by the dotted line represents the foreground region. In FIG. 6B, the attributes of all the foreground areas are determined, but the attributes are displayed only for a part of the foreground areas. The information of each text block (information indicating the attribute and the position and size of each block) detected in this step is used in the subsequent processing such as OCR processing and similarity calculation.

In the block selection process of this step, only the text block is detected. The reason is that the position of the character string expresses the structure of the scanned image well and is closely related to the index information. Therefore, it does not exclude the use of the information of the block determined to have other attributes such as the photo area and the table area in the subsequent processing.

In S503, the image processing unit 305 acquires an index extraction rule from the HDD 120 and expands it into the RAM 119.

FIG. 7 is a diagram showing a part of an index extraction rule (hereinafter, simply referred to as an extraction rule). FIG. 7 shows a record of an extraction rule in which "0001" is assigned and registered as a form ID included in the extraction rule. In the extraction rule, each data of "document ID", "thumbnail", "document identification information", and "index information" is associated with each registered document in record units. The extraction rule keeps these combinations (records) as many as the number of registered documents. The document ID is a unique ID that represents the type of document.

The document identification information is information on the position and size of a text block obtained as a result of executing a block selection process on a scanned image of a registered document. The document identification information is information for specifying the type of document and is used in document matching described later.

The index information is information for extracting an index included in the scanned image. The index is used to determine the filename or metadata to give to the file. Specifically, the index information includes information on the coordinates and size of the text block including the character string (index) of each item in the registered document. The image 701 of the "index information" of FIG. 7 is illustrated by arranging the positions and sizes of the text blocks including the respective items at the coordinates on the image. In addition, the index information includes information indicating the index used to generate the file name and its order, and information to be assigned as metadata.

In the "file name rule" of index information, the index of the item of title (title), issuer company name (sender), and form number (number) can be connected with an underscore which is a separator to generate a file name. It is shown. In addition, "metadata" indicates that the index of the item of total amount (total_price) is used as metadata. That is, by extracting the index of a predetermined item, it is possible to generate a file name recommended to the user and extract metadata.

In this embodiment, an example of using the extracted index as a file name or metadata is shown, but a rule for determining the folder information of the destination of the file, which is other property information, is retained. May be good. Also in that case, the property information generated by using the index is recommended to the user in S402, and the property information is set in the scanned image file in S403.

Further, as a rule for extracting the registered document, as shown in the “thumbnail” of FIG. 7, the thumbnail of the scanned image corresponding to the registered document may be held together.

In S504, the image processing unit 305 executes document matching on the scanned image. In document matching, it is determined whether or not a document of the same type as the document (input document) scanned to obtain a scanned image is in the document group registered in the extraction rule. Then, when it is determined that a document of the same type as the input document is registered, it is a process of specifying the type.

In the present embodiment, first, the scanned image and each document registered in the extraction rule are compared on a one-to-one basis, and the degree of similarity indicating how similar the shapes and arrangements of the included text blocks are. Is calculated. As a method of calculating the similarity, for example, the entire text block of the scanned image and the entire text block of the registered document are aligned. Then, the square of the sum of the areas where each text block of the scanned image and each text block of the registered document overlap (value A) is obtained. Further, the product (value B) of the total area of the text blocks of the scanned image and the total area of the text blocks of the registered document is obtained. Then, there is a method in which the value obtained by dividing the value A by the value B is used as the similarity. This similarity is calculated between the scanned image and all the documents registered in the extraction rule.

Then, the document registered in the extraction rule having a degree of similarity equal to or higher than a predetermined value and having the highest degree of similarity is identified as a document of the same type as the scanned input document. If there is no document having a similarity equal to or higher than a predetermined value in the extraction rule, it is determined that the document of the same type as the input document is not registered in the extraction rule.

In S505, the image processing unit 305 determines whether a document of the same type as the input document is registered in the extraction rule as a result of the document matching executed in S504. If the input document is not a registered document (NO in S505), the processing of this flowchart ends. If it is not a registered document, a new ID is added as described above, and the layout information of the text block detected in S502 is registered in the extraction rule. In this case, S402 does not recommend the file name and the metadata to the user, and the display control unit 301 accepts the input of the file name by the user. When the display control unit 301 receives an input from the user via the display / operation unit 123, the input file name is determined as the file name of the scanned image.

When a document of the same type as the input document is registered (YES in S505), the image processing unit 305 scans the image with the same document ID as the document of the extraction rule specified in S504 as the same type as the input document in S506. Give to.

In S507, the image processing unit 305 performs an index block estimation process for estimating the text block of the index of the item to be extracted (processed target) in the scanned image based on the extraction rule associated with the document ID assigned in S506. Run. A text block containing a character string (index) indicating items such as a title, a issuing company name, and a form number may be called an index block. The details of the index block estimation process will be described later.

In S508, the image processing unit 305 executes partial OCR on the index block group of each item estimated in S507, and extracts the character string corresponding to each index block as an index.

[About index block estimation process (S507)]
FIG. 8 is a flowchart of the index block estimation process of S507. The details of the index block estimation process will be described with reference to FIG. Hereinafter, the registered document refers to a document corresponding to the document ID given to the scanned image in S506 among the documents registered in the extraction rule acquired in S503. In the description of this flowchart, the registered document will be described as assuming that the document has the document ID “0001” of FIG.

In S800, the image processing unit 305 acquires the document identification information associated with the document ID assigned in S506 from the extraction rule. Then, the image processing unit 305 performs overall alignment with the entire text block in the scanned image and the entire text block of the registered document.

The input document of the scanned image acquired in S400 is a document of the same type as the registered document, and each item is printed at the same coordinates as the item of the registered document. However, the position of the text block on the scanned image and the position of the text block of the registered document may be misaligned due to the timing of printing and scanning or the difference depending on the device at the time of printing. Therefore, in this step, in order to reduce the influence of the deviation and improve the accuracy of the subsequent processing, the entire alignment is performed. In this embodiment, since the tilt correction is performed in S500 of FIG. 5, an example in which only the correction of shifting (translating) the entire text block on the scanned image is performed in the overall alignment of this step will be described. do.

In the overall alignment, the shift amount of how much the text block of the scanned image is shifted with respect to the text block of the registered document is calculated, and the coordinates are adjusted so that each text block of the scanned image is shifted by the shift amount. Make corrections.

FIG. 9 is a diagram obtained by cutting out a part of an image in which the text block of the scanned image and the text block of the registered document are drawn in the same coordinate system. A specific procedure for calculating the shift amount for overall alignment will be described with reference to FIG. In FIG. 9, the solid line rectangle indicates one text block 900 selected from the text block group in the scanned image, and the broken line rectangle indicates the text blocks 901 to 903 of the registered document surrounding the text block 900. Shown. Further, in FIG. 9, the alternate long and short dash line circle 904 indicates a range centered on the upper left apex of the text block 900 of the scanned image and having a certain distance as a radius.

In order to calculate the shift amount, a text block (called a pair block) of a registered document corresponding to each text block of the scanned image is determined. Here, the determination of the pair block of the text block of the scanned image will be described.

First, among the text blocks 901 to 903 of the registered document, a text block having the upper left vertex in the circle 904 centered on the upper left vertex of one text block 900 selected from the text block group in the scanned image is inserted. seek. In FIG. 9, the text blocks 901 and 902 correspond to each other. Next, the overlap rate between the text block 900 of the scanned image and the text blocks 901 and 902 of the registered document is obtained. For the overlap rate, the area of the common portion of both text blocks is calculated by combining the upper left vertices of the text block of the scanned image and the text block of the registered image. Then, the value is calculated by (area of common part) / (area of both text blocks, whichever is larger) and used as the overlap rate.

A text block of a registered document whose overlap rate satisfies a predetermined condition is defined as a pair block. The predetermined condition is, for example, that the overlap rate of the scanned image with the text block is equal to or greater than the value obtained by multiplying the maximum overlap rate by the coefficient α and equal to or greater than a predetermined threshold value. In this case, the coefficient α is for selecting a combination having an overlap rate close to the maximum overlap rate, and is set to a value less than 1.0 such as 0.5 to 0.8. Further, the predetermined threshold value defines the minimum line, and is set to a value less than 1.0 such as 0.3 to 0.7.

In FIG. 9, among the text blocks 901 and 902 of the registered document, only the text block 901 having a shape similar to that of the text block 900 of the scanned image is selected as the pair block. If there are other text blocks that satisfy the predetermined conditions, multiple pair blocks may be selected. In this way, for each of the pair block groups corresponding to one text block selected from the scanned image, the difference amount in the X direction and the Y direction of the upper left vertex from the text block selected from the scanned image ( Shift amount) is calculated. Then, the difference amount is voted in the shift amount histogram. The range of the histogram bins in this case may be arbitrary.

In the case of FIG. 9, for the text block 900, the difference amount (shift amount) in the X direction and the Y direction of the upper left vertex of the registered document from the text block 901 is calculated, and the shift amount is voted in the shift amount histogram. ..

The pair block group corresponding to the text block in the scanned image is determined, and the process of voting for the shift amount histogram is performed for all the text blocks of the scanned image. Then, the position of the maximum peak point in the finally obtained shift amount histogram is determined. The shift amount indicated by the determined position is used as the shift amount for overall alignment.

If there is concern about the influence of noise, smoothing may be applied to the generated shift amount histogram. Further, a local peak point other than the maximum peak point may be selected as a shift amount candidate, and the shift amount used for overall positioning may be selected from the candidates. For example, for each candidate of the shift amount, the coordinates of the text block of the scanned image are shifted, the similarity calculation similar to the document matching in S504 of FIG. 5 is performed, and the candidate having the highest similarity is finally shifted. It may be determined as a quantity.

By shifting the coordinates of each text block of the scanned image by the shift amount determined in the above procedure, the text block group of the aligned scanned image can be obtained. The method of aligning the text block is not limited to the above method. An example of performing only correction related to the shift (translation) of the entire scanned image has been described, but if a shift related to the magnification is expected as a shift between printing and scanning, not only the shift amount but also the shift in the magnification is taken into consideration. Alignment may be performed.

The scanned image or the text block group of the scanned image in the following steps shall refer to the entire aligned scanned image or the text block group.

Next, the index information of the registered document associated with the document ID given in S506 is acquired. Then, in S801, any one of the index items included in the index information is selected as the processing target, and S801 to S810 are repeated. Then, a process of estimating the text block of the item to be processed is performed from the text block group of the scanned image. When the processing for the item to be processed is completed, the item to be processed is selected again from the unprocessed items.

In S801, the image processing unit 305 selects one unprocessed index item from the items registered in the index information of the registered document and sets it as the item to be processed. In the present embodiment, any of the items of the title (title), the issuing company name (sender), the form number (number), and the total amount (total_price), which are stored in the index information of FIG. 7, is selected as the processing target. Will be done.

In S802, the image processing unit 305 acquires a "partial pattern" of the item to be processed. The partial pattern includes information on a partial layout (partial layout) of the text block included in the registered document and information on a range including the partial layout (partial pattern range).

FIG. 10A illustrates the position and size of the index block of each item in the registered document registered as the document ID “0001” in FIG. 7. The broken line rectangle in FIG. 10A represents the index blocks 1000 to 1003 of each item of the title, the form number, the total amount, and the issuing company name.

FIG. 10B is a diagram showing a partial pattern of the item of “sender”. The range represented by the rectangle of the alternate long and short dash line in FIG. 10B indicates the partial pattern range 1006 of the item of “sender”. The partial pattern range 1006 is determined using a preset value with reference to the index block 1003, which is a text block of the item "sender".

The text blocks 1004 and 1005 represent text blocks in which at least a part of the partial pattern range 1006 is included in the registered document. The partial layout in the registered document represented by the text blocks 1004 and 1005 and the index block 1003 is the partial layout of the item of the issuing company name. The partial layout is represented by a text block of the item to be processed and at least one text block other than the text block of the item to be processed. The layout is information representing the position information of each text block and the size of each text block.

As the information included in the partial pattern of the item of the issuing company name, the partial pattern range 1006 and the partial layout including the index block 1003 and the text blocks 1004 and 1005 are determined. In this way, the partial patterns corresponding to each item of the registration document are determined and stored.

Although details will be described later, in the present embodiment, a text block of an item to be processed in the scanned image is estimated by searching for a position in the scanned image whose arrangement is similar to or matching the partial layout.

FIG. 10 (c) is a diagram showing a partial pattern of the item of “title”. Similarly, for the title, a partial pattern range 1007 and a partial layout including the index block 1000 of the title and the text blocks 1001 and 1008-1013 included in the partial pattern range 1007 are determined as the partial pattern.

The size of the partial pattern range 1007 is different from that of the partial pattern range 1006 of FIG. 10B. In this way, the partial pattern size may be different depending on the nature of the item. Alternatively, as the size of the partial pattern range, a common size may be used for all items. The method of determining the size of the partial pattern range will be described in the second embodiment.

The partial pattern does not need to be determined each time the index extraction process performed after scanning the document document is executed. For example, when registering a document, a partial pattern may be determined for each item and stored in advance as a part of the extraction rule shown in FIG. 7. That is, in S802, it suffices to acquire the stored partial pattern of the item to be processed.

In the next S803 and S804, a position (XY candidate position) in the scanned image having a region having a high degree of coincidence with the partial layout of the item to be processed is determined. As a method of determining the XY candidate position, for example, the candidate position may be estimated by scanning a partial pattern with respect to the search range in the scanned image and calculating the degree of matching as in template matching. In the present embodiment, in order to suppress the amount of calculation, a candidate position in the Y direction in the search range is determined and the position in the Y direction (Y position) is narrowed down. Then, in each of the Y position candidate (Y candidate position) groups, a method of suppressing the amount of calculation by scanning a partial pattern in the X direction to determine the XY candidate position will be described.

In S803, the image processing unit 305 determines a Y candidate position group similar to the text block of the partial pattern of the item to be processed in the registered document from the text block group of the scanned image.

FIG. 11 is a diagram for explaining the determination process of the Y candidate position group. The explanation is made assuming that the item to be processed is the issuing company name (sender).

FIG. 11A is a diagram showing a partial pattern of the issuing company name (sender) in the registration document, and is the same diagram as in FIG. 10B. FIG. 11B is a scanned image, and the broken line rectangle represents a group of aligned text blocks. Further, the document indicated by the scanned image shown in FIG. 11B is a document determined to be the same type of document as the registered document “0001”, but the item line in the table structure is compared with that of the registered document of FIG. It shows an example where the number is increasing. Therefore, the index block 1101 of the issuing company name (sender) to be estimated in the scanned image is shifted downward as compared with the position of the index block 1002 of the issuing company name (sender) in the registration document. There is.

In FIG. 11C, one of the text blocks 1003 to 1005 representing the partial layout included in the partial pattern of the issuing company name, the text block 1003, is superimposed on the same coordinate system as the text block group of the scanned image. It is a figure. The determination of the Y candidate position group will be described with reference to FIG. 11C, focusing on the text block 1003 in the partial pattern.

The search range 1100 represented by the rectangular chain line in FIG. 11C represents a range to be searched for in order to determine the Y candidate position group of the item to be processed. The text blocks 1101 to 1109 represented by the broken line rectangles are the text blocks of the scanned image shown in FIG. 11B, in which the center of the rectangle is within the search range 1100.

To determine the Y candidate position group, first, one text block included in the partial layout (text block 1003 in FIG. 11C) is selected. Then, the selected text block is superimposed on the same coordinate system as the text block group of the scanned image, and the Y of the center of the rectangle with the text block of the scanned image (text blocks 1101 to 1109 in FIG. 11C) within the search range. Calculate the difference amount of each position. Then, the calculated difference amount is voted in the shift amount histogram in the Y direction. The range of bins in the shift amount histogram may be arbitrary.

FIG. 12 is a diagram showing an example of a shift amount histogram in the Y direction. FIG. 12A is a shift amount histogram after voting for the difference amount of the Y position between the text block 1003 of the partial pattern in FIG. 11C and the text block 1102 of the scanned image. h indicates the upper limit of the absolute value of the search range in the Y direction from the reference. Voting is performed at the position 1200 according to the difference amount in the Y direction between the text block 1003 and the text block 1102. Similarly, voting is performed according to the difference amount of the Y center between one text block included in the partial pattern and all the text blocks within the search range of the scanned image. This vote is made for all text blocks in the partial pattern. That is, for the text blocks 1004 and 1005 of the partial pattern, the difference amount of the Y center from the text blocks 1101 to 1109 in the search range is calculated and voted in the shift amount histogram. Then, the shift amount histogram in the Y direction is completed. If there is concern about the influence of noise, smoothing may be applied to the shift amount histogram generated in the Y direction.

FIG. 12B is a finally generated shift amount histogram in the Y direction. After the generation of the shift amount histogram is completed, the peak points as shown in the positions 1201 to 1206 in the histogram are determined, and the Y candidate position group is determined based on the shift amount in the Y direction according to the bin of each peak point.

The search range 1100 for determining the Y candidate position group in FIG. 11C is automatically determined with a preset value based on the position of the index block of the partial pattern. The size of the search range may be determined by using a range common to all items or according to the attributes of the items to be processed. For example, the title index block is often in a fixed position in the document. Therefore, when the item to be processed is a title, it is unlikely that the index block to be estimated from the search range is out of the search range even if the search range is narrowed, so the search range may be set narrow. By narrowing the search range, it is possible to prevent unnecessary candidate positions from being determined while reducing the amount of calculation. On the other hand, the position of the index block whose items are the total amount of money may change up and down according to the change in the number of item rows of the table structure in the document. Therefore, when the item to be processed is the total amount, the search range may be set wider up and down than other items.

In S804, the image processing unit 305 derives the degree of coincidence between the partial layout of the partial pattern and the text block group of the scanned image based on the respective Y candidate positions determined in S803.

FIG. 13 is a diagram showing a state in which the partial layout and the layout of the text block of the scanned image are overlapped when the partial layout of the item to be processed is superposed at a certain position in the scanned image. A method of deriving the degree of matching between the partial layout and the text block group of the scanned image will be described with reference to FIG.

In FIG. 13, the solid line rectangle is a text block 1003 to 1005 that constitutes a partial layout of the item to be processed. The dashed line rectangle represents the partial pattern range 1006. The dashed rectangle represents the text blocks 1101, 1104-1106, 1109 of the scanned image. The diagonally filled areas 1309 and 1310 represent areas where the text blocks 1003 to 1005 of the partial layout and the text blocks of the scanned image overlap.

The degree of matching Score between the partial layout and the text block of the scanned image is derived by the following equation (1).

In the above formula (1), R represents the full text blocks constituting the partial layout and N _R represents the total number of text blocks constituting the partial layout. In FIG. 13, R is a text block 1003-1005 and _NR is 3.

Correlation (r) is the degree of individual matching of one text block r constituting the partial layout. The individual agreement degree Correlation (r) of the text block r is derived by Eq. (2).

OverlappingQ is a set of text blocks of scanned images that overlap with the text block r. OverlapArea (r, q) is the area of the overlapping area of the text block r and the text block q of one of the text blocks of Overlapping Q. N _Overlapping Q represents the total number of Overlapping Q.

In FIG. 13, when r is the text block 1003, the Overlapping Q is only the text block 1105, and the Overlap Area (r, q) is the area 1309. When r is set to the text block 1005, the Overlapping Q corresponds to only the text block 1104, and the Overlap Area (r, q) corresponds to the area 1310. When r is the text block 1004, Correlation (r) is 0 because there is no corresponding Overlapping Q and N _{Overlapping Q is 0.}

Area_r is the area of the text block r, and Area_q is the area of the text block q.

In the derivation of the degree of matching by the equation (1), the larger the number of text blocks in the scanned image and the larger the area of the text blocks, the larger the value of the individual degree of matching Collrelation (r) may be derived. .. Therefore, for the degree of agreement Score, a penalty term PenaltyTerm may be added as shown in the following equation (1)'.

The penalty term PenaltyTerm in equation (1)'is derived by equation (3).

TotalArea_R is the total area of all text blocks that make up the partial layout.
In FIG. 13, it is the total area of the text blocks 1003 to 1005.

TotalArea_NonOverlappingQ is the sum of the areas of the text blocks of the scanned image existing in the partial pattern range that do not overlap with any of the text blocks constituting the partial layout. In the case of FIG. 13, it is the total area of the text blocks 1101, 1104, 1105, 1106, and 1109 of the text blocks 1101, 1104, 1105, 1106, and 1109 that do not overlap with the text blocks 1003 to 1005 in the partial pattern range 1006.

By providing a penalty term, it is possible to adjust so that the degree of matching is deducted when the text block in the scanned image exists in the range where the text block constituting the partial layout in the partial pattern range 1006 does not exist. can. Therefore, even when the number of text blocks constituting the partial layout is small, the degree of matching can be derived by utilizing the information in the area where the text blocks constituting the partial layout within the partial pattern range do not exist. The method of deriving the degree of coincidence is not limited to the derivation by the above equation, and it suffices if the degree of coincidence with the partial layout can be determined.

In S804, the image processing unit 305 places a partial pattern (partial layout and partial pattern range) so that the index block is located at any Y candidate position in the Y candidate position group determined in S803. Then, the image processing unit 305 scans the partial pattern in the X direction to derive the degree of coincidence at each position. The image processing unit 305 performs this for all Y candidate position groups.

FIG. 14 is a diagram showing the processing of this step at the Y candidate position of one of the Y candidate position groups determined in S803. In FIG. 14A, the solid line rectangle represents the text blocks 1003 to 1005 constituting the partial layout, and the alternate long and short dash line rectangle represents the partial pattern range 1006. The dashed square represents the text blocks 1101, 1105, 1106 of the scanned image, and the shaded area represents the overlapping region of the text block of the partial layout and the text block of the scanned image. Further, in FIG. 14, the processing in this step is shown to proceed in order from FIGS. 14 (a) to 14 (e), and the partial pattern is scanned in the X direction (from left to right) within the search range. However, it shows how to derive the degree of coincidence at each position. Similar processing is performed at each Y candidate position.

In S805, the image processing unit 305 determines the position where the degree of coincidence derived in S804 is maximum as the XY candidate position. For example, in the case of FIG. 14, the degree of coincidence of the partial pattern (partial layout) is maximized at the position shown in FIG. 14 (c). Therefore, the position of the text block 1003 indicating the index block included in the partial layout in FIG. 14C is determined as the XY candidate position.

In S806, the image processing unit 305 determines whether or not the degree of coincidence at the XY candidate positions determined in S805 is equal to or greater than a predetermined threshold value.

When the degree of coincidence is equal to or greater than the threshold value (YES in S806), the image processing unit 305 estimates the XY candidate position on the scanned image determined in S805 as the position where the text block (index block) of the item to be processed exists in S807. .. The image processing unit 305 performs a process of estimating the index block of the item to be processed in the scanned image based on the estimated position.

For example, when the index block of the item to be processed in the registered document is shifted to the XY candidate position in the scanned image, it is determined whether the overlapping text blocks in the scanned image satisfy a predetermined condition. The predetermined condition is, for example, that the overlap rate indicating the degree of overlap with the index block to be processed in the registered document is equal to or more than a predetermined value, and the distance of the upper left coordinate from the index block to be processed in the registered document is within a certain range. It is a condition that it is inside.

If it is determined that there is a text block satisfying a predetermined condition (YES in S807), the process proceeds to S808. In S808, the image processing unit 305 estimates that the text block of the scanned image determined in S807 to satisfy the predetermined condition is a text block (index block) including a character string indicating the item to be processed selected in S801.

If the degree of matching is less than the threshold value (S806 is NO) or if it is determined that there is no corresponding text block (S807 is NO), the process proceeds to S809. In S809, the image processing unit 305 determines that the text block corresponding to the item to be processed selected in S801 is not included in the scanned image. For example, if the character string corresponding to the item to be processed is not described in the predetermined area in the scanned image, or if the position is erroneously estimated in S804, the determination is made in S809.

In S810, the image processing unit 305 determines whether or not the process of estimating the index block has been completed for all the items registered in the index information of the registered document. If there is an unprocessed item, the process returns to S801.

If the processing for all the items is completed, the processing of this flowchart is completed and the process proceeds to S508. In S508, the image processing unit 305 executes OCR processing on the index block of each of the estimated items, and extracts the character string corresponding to each item as an index.

As described above, in the present embodiment, the index included in the scanned image is extracted by using a part of the layout of the text block. Therefore, according to the present embodiment, the index can be extracted even when the position of the index block included in the scanned image is different from that of the registered document due to a change in the description content in the input document or the like. Further, in the present embodiment, the type of the input document is specified by document matching, and the extraction rule associated with the document type is used. Therefore, even in the process of estimating the index block by the partial layout of the text block, it is possible to suppress the erroneous extraction of the index. Further, in the document matching and index block estimation processing, only the text block in the foreground area obtained as a result of the preprocessing of the OCR processing is used. Therefore, the index extraction process can be performed without incurring an extra calculation cost.

<Embodiment 2>
In the first embodiment, a method of determining the partial pattern range based on a preset value has been described. However, if the partial pattern range is set too wide, the position of the index block cannot be estimated appropriately when the layout changes only around the index block. On the other hand, when the partial pattern range becomes small, the number of text blocks constituting the partial layout may be determined to be small, and it becomes difficult to search for a region having a high degree of matching in the scanned image. Therefore, in the present embodiment, a method of determining the partial pattern range to an appropriate size will be described. In addition, this embodiment will be described mainly on the difference from the first embodiment. The parts not specified in particular have the same configuration and processing as in the first embodiment.

The number and layout of text blocks around the index block change depending on the type of document. Therefore, in the present embodiment, in order to determine the size of the partial pattern range, the area including the index block of the target item is gradually expanded, and the number of text blocks overlapping the area is counted. Then, the area when the number of overlapping text blocks exceeds a certain number is determined as the partial pattern range of the item.

FIG. 15 is a diagram for explaining a method of determining a partial pattern range in the present embodiment. In FIG. 15A, the solid line rectangle is the title index block 1000, and the alternate long and short dash line rectangle is an area for determining the partial pattern range of the title. The regions indicate the initial region 1500, the second stage region 1501, and the maximum region 1502, respectively. FIG. 15A shows how the area for determining the partial pattern range in the title item is changed stepwise. While gradually expanding the area from the initial area to the maximum area, the text blocks excluding the index block that overlaps the area are counted. Then, the area indicated by the rectangle of the alternate long and short dash line when the number of counted text blocks becomes a predetermined number or more is determined as the partial pattern range of the item. The predetermined number is preferably one or more. In this embodiment, it is assumed that the predetermined number is 5.

A method for determining the partial pattern range of the present embodiment will be specifically described. First, the number of text blocks that overlap the initial area 1500 even a little is counted. In this case, since there is no text block other than the index block 1000, the process proceeds to the next stage.

Next, the area is expanded and the text blocks that overlap the second stage area 1501 as much as possible are counted. FIG. 15B is a diagram in the case where the region for determining the partial pattern range is the second stage region 1501. As shown in FIG. 15B, the text blocks 1001 and 1008-1013 overlap with the second stage region 1501. Therefore, the number of text blocks overlapping the second stage area 1501 is counted as seven. Then, it is determined that the number of overlapping text blocks is 5 or more, which is a predetermined number. Therefore, the partial pattern range of the title is determined by the position and size indicated by the second stage region 1501. Therefore, the layout including the text blocks 1001, 1008-1013 and the index block 1000 whose partial pattern range includes at least a part is determined as the partial layout of the title.

Alternatively, the number of surrounding text blocks differs depending on the item, and the area where the layout of the text blocks does not change much depending on the description content differs. Therefore, for example, the size of the partial pattern range may be different depending on the attribute of the item. That is, the size of the partial pattern may be set in advance according to the attribute of the item.

When the item is a title, there is often no text block in the vicinity of the text block of the title. Further, the title is characterized in that it exists at the upper part of the document in which the layout of the text block does not change much due to the change in the description content of the document. Therefore, as shown in the partial pattern range 1007 of FIG. 10C, if the item is the title of the document, the entire image width fits in the X direction, and about a quarter of the image fits in the Y direction. The region may be determined as a partial pattern range.

As described above, in the present embodiment, the partial pattern range is determined according to the document. Therefore, the accuracy of the index block estimation process can be improved by the appropriate partial pattern range according to the document.

<Embodiment 3>
In the first embodiment, the position where the degree of matching derived by using the partial pattern is maximized is determined as the XY candidate position, and if the degree of matching of the XY candidate positions is equal to or higher than a predetermined threshold value, processing is performed based on the XY candidate position. The method of estimating the position of the index block of the target item was explained.

However, the input document may have a plurality of areas containing text blocks that are similar in arrangement to the partial layout of the registered document. When there are a plurality of areas similar to the partial layout in the input document, the method of the first embodiment may fail to estimate the index block of the item to be processed in the input document.

Therefore, in the present embodiment, a method of appropriately estimating the position of the index block in the input document will be described even when a plurality of areas similar to the partial layout of the item to be processed exist in the input document. In addition, this embodiment will be described mainly on the difference from the first embodiment. The parts not specified in particular have the same configuration and processing as in the first embodiment.

FIG. 16 is a flowchart for explaining the index block estimation process of S507 in the present embodiment. The details of the index block estimation process in the present embodiment will be described with reference to the flowchart of FIG. Since S1600 to S1604 are the same as S800 to S804, the description thereof will be omitted.

In S1605, the image processing unit 305 determines the XY position in the scanned image in which the degree of coincidence derived in S1604 is equal to or greater than a predetermined threshold value. As a result of this step, a plurality of XY positions may not be determined, but for convenience, the XY positions determined by this step are referred to as XY candidate position groups.

FIG. 17 is a diagram showing an example of a registered document in which a plurality of areas similar to a partial layout composed of an index block and a block around the index block exist. FIG. 17A is a diagram showing an example of a registered document. FIG. 17B is a diagram showing a partial pattern when the text block 1705 including the character string corresponding to the item of “Quotation Date” in the registration document of FIG. 17A is used as the index block. .. In FIG. 17B, the alternate long and short dash line rectangle indicates the partial pattern range 1700 of the "estimated date" item, and the text blocks 1701-1706 represented by the solid line rectangle are the partial layout of the "estimated date" item. Shows the text blocks that make up. In the description of the flowchart of FIG. 16, processing when "estimated date" is set as the item to be processed will be described.

FIG. 18 is a diagram for explaining an input document. FIG. 18A is a diagram showing an input document, and in the description of this flowchart, it is assumed that the index block estimation process is performed on the scanned image obtained as a result of scanning the input document. Further, the document similar to the input document of FIG. 18A will be described as assuming that the registered document of FIG. 17 is specified by the document matching of S504.

18 (b) to 18 (e) are images showing text blocks detected as a result of performing block selection processing on the scanned image of the input document of FIG. 18 (a), respectively. It is the figure which superposed the partial pattern of "estimated date". The rectangles in the respective figures of FIGS. 18 (b) to 18 (e) indicate partial patterns. That is, the solid line rectangle is a text block constituting the partial layout, and the alternate long and short dash line rectangle is the partial pattern range.

The positions of the partial patterns shown in FIGS. 18 (b) to 18 (e) are positions when the degree of coincidence derived in S1604 is equal to or greater than a predetermined threshold value. Therefore, among the text blocks represented by the solid rectangles constituting the partial layout, the XY positions 1801 to 1804 of the index block are determined as the XY candidate position groups as a result of the processing of this step.

In a document in which simple text block arrangements exist repeatedly as in the input document shown in FIG. 18A, if an index block exists in the repeatedly arranged text blocks, the degree of matching A plurality of XY positions where is equal to or greater than the threshold value are determined. Therefore, for the input document shown in FIG. 18A, the number of XY candidate position groups determined as a result of the processing of this step is 2 or more.

In S1606, the image processing unit 305 switches the processing according to the number of XY candidate position groups determined in S1605. If the number of XY candidate position groups is 1, the process proceeds to S1610, and if the number of XY candidate position groups is 0, the process proceeds to S1612. Since the processing of S1612 is the same as that of S809, the description thereof will be omitted.

If the number of XY candidate position groups is two or more, the process proceeds to S1607. In S1607, the image processing unit 305 acquires a similar position (group) that is a position in the registered document and the degree of coincidence with the partial layout of the item to be processed is equal to or more than a predetermined threshold value.

The partial layout included in the partial pattern of the item to be processed is superimposed on the position in the registered document to derive the matching degree of the text block, and the XY position in the registered document where the matching degree is equal to or higher than a predetermined threshold value is ". Determined as "similar position". The method of calculating the degree of coincidence between the text block in the registered document and the text block of the partial layout may be derived by the same method as in S1602 to S1604. That is, the degree of coincidence may be derived by the same procedure for the registered document instead of the input document.

FIG. 19 is a diagram for explaining similar positions in the registration document. FIG. 19 (a) is a diagram showing the same registration document as in FIG. 17 (a). 19 (b) to 19 (e) are images showing text blocks detected as a result of performing block selection processing on the scanned image of the registered document of FIG. 19 (a), respectively. It is the figure which superposed the partial pattern of "estimated date". The rectangles in the respective figures of FIGS. 19 (b) to 19 (e) indicate partial patterns. That is, the solid line rectangle is a text block constituting the partial layout, and the alternate long and short dash line rectangle is the partial pattern range.

The positions of the partial patterns in FIGS. 19B to 19E are the respective positions when the derived degree of coincidence becomes equal to or higher than a predetermined threshold value. Therefore, the XY positions of the index blocks among the text blocks constituting the partial layout are determined as the similar position groups 1901-1904. In this step, the position information of the similar position group of the item to be processed is acquired. The similar position group 1901-1904 also includes the XY position of the index block 1705 at the time of registration shown in FIG. 17B, as in the similar position 1902.

It is not necessary to perform the process of determining the similar position in the registered document in S1607. For example, when registering a document, a partial pattern may be determined for each item, then a similar position group may be determined, and information on the similar position group may be stored in advance as part of the extraction rule shown in FIG. That is, in S1607, a similar position group may be acquired as one of the stored extraction rules for the items to be processed.

In S1608, the image processing unit 305 associates the similar position group of the registered document acquired in S1607 with the XY candidate position group in the input document determined in S1605. Specifically, the XY candidate position groups sorted under the same conditions as the similar position group are associated with the similar position group sorted at the Y position in order from one side of the Y position, and further Y. Correspondence is performed in the order from the other side of the position.

FIG. 20 is a diagram for explaining the process of this step. The numerical values in the table are numerical values indicating symbols indicating positions in the document shown in FIG. 18 or FIG.

FIG. 20A is a diagram showing the correspondence when the numbers of the similar position group and the number of the XY candidate position groups match, as shown in FIGS. 18 and 19. Column 2001 is a group of similar positions sorted by Y position. Column 2002 is an XY candidate position group sorted by Y position, and is an XY candidate position group associated with a similar position group in column 2001 in order from the top of the Y position. Column 2003 is an XY candidate position group sorted by Y position, and is an XY candidate position group associated with a similar position group in column 2001 in order from the bottom of the Y position. In FIG. 20A, both the XY candidate position group in column 2002 and the XY candidate position group in column 2003 are associated with the same similar positions.

FIG. 20B is a diagram for explaining a method of associating the number of similar position groups with respect to the number of XY position groups in this step. For example, when the degree of coincidence with the registered document when the partial pattern is superimposed on the position of the registered document shown in FIG. 19E is less than the threshold value, and only the similar position groups 1901-1903 are acquired in S1607, The figure showing the correspondence is shown in FIG. 20 (b). Column 2011 is a group of similar positions sorted by Y position. Column 2012 is an XY candidate position group associated with the similar position group in column 2011 in order from the top of the Y position. Column 2013 is an XY candidate position group associated with the similar position group in column 2011 in order from the bottom of the Y position. In FIG. 20B, the XY candidate position groups corresponding to the similar position groups are different in the association from the top and the association from the bottom.

FIG. 20C is a diagram for explaining a method of associating the number of similar position groups with respect to the number of XY candidate position groups in this step. When the degree of coincidence with the input document when the partial pattern is superimposed on the position of the input document shown in FIG. 18E is less than the threshold value, and in S1605, only the XY positions 1801 to 1803 are determined as the XY candidate position group. 20 (c) is a diagram showing the correspondence of the above. Column 2021 is a group of similar positions sorted by Y position. Column 2022 is an XY candidate position group associated with the similar position group in column 2021 in order from the top of the Y position. Column 2023 is an XY candidate position group associated with the similar position group in column 2021 in order from the bottom of the Y position. The results of the mapping from the top and the mapping from the bottom are different. In the mapping from the top, the XY candidate position group corresponding to the similar position 1904 cannot be found, and in the mapping from the bottom, the XY corresponding to the similar position 1901. The result is that the candidate position group cannot be found.

In S1609, the image processing unit 305 determines one XY candidate position from the XY candidate position group determined in S1605 based on the result of the association performed in S1608.

As shown in FIG. 20A, the result of the association performed in S1608 may match the result of the association from the top and the result of the association from the bottom. In this case, among the XY candidate position groups, the XY position associated with the similar position indicating the position of the index block at the time of registration is determined as one XY candidate position. In the example of FIG. 20A, the XY position 1802 associated with the similar position 1902 indicating the position of the index block is determined as one XY candidate position.

On the other hand, as shown in FIGS. 20 (b) and 20 (c), the result of the association performed in S1608 may not match the result of the association from above and the result of the association from below. In this case, first, the XY position of the input document associated with the similar position indicating the position of the index block when the association is performed from above is determined. Further, when the association is performed from the bottom, the XY position of the input document associated with the similar position indicating the position of the index block is determined.

In the example of FIG. 20B, the XY position 1802 and the XY position 1803 associated with the similar position 1902 indicating the position of the index block are determined. In the example of FIG. 20C, the XY position 1802 and the XY position 1801 associated with the similar position 1902 are determined. Then, of the two determined XY positions, the one with the higher degree of coincidence derived in S1604 is determined as one XY candidate position in the XY candidate position group. It should be noted that one XY position may be selected from one of the two XY positions without using the degree of coincidence. For example, two XY positions may be displayed to receive an instruction from the user, and it may be used by remembering for each item whether to use the correspondence from the top or the correspondence from the bottom.

When one XY candidate position is determined from the XY candidate position group, the process proceeds to S1610. In S1610, similarly to the processing of S807, the XY candidate position is estimated as the position of the index block to be processed, and the index block of the item to be processed is estimated from the text block of the scanned image. Since S1611 is the same as S808 and S1613 is the same as S810, the description thereof will be omitted.

As described above, in the present embodiment, when there are a plurality of XY candidate positions whose matching degree is equal to or higher than the threshold value in the input document, the correspondence with the similar position group of the registered document whose matching degree with the partial pattern is equal to or higher than the threshold value. After making the attachment, one XY candidate position is determined. Therefore, the accuracy of the index block estimation process can be improved even when a plurality of areas similar to the partial layout consisting of the index block and the text block around the index block exist in the document.

<Other Embodiments>
In the above-described embodiment, an example in which the image forming apparatus 100 performs processing of each step of the flowchart of FIG. 4 by itself has been described. Alternatively, all or part of these processes may be performed by another image processing device on the system 105 having the function shown in FIG.

For example, the scanning process is executed by the image forming apparatus 100, and the scanned image is transmitted to the terminal 101 via the network. The terminal 101 has the same function as the image processing unit 305, and the terminal 101 may execute the index extraction process. In this case, the terminal 101 returns the index extraction result to the image forming apparatus 100, and the image forming apparatus 100 generates a file and transmits a file based on the acquired index extraction result.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Image forming apparatus 305 Image processing unit 111 CPU

Claims (23)

An acquisition method for acquiring image data of a manuscript,
A detection means for detecting a text block in an image indicated by the image data, and
From the information that defines the layout of the text block for each document group, a specific means for specifying the document corresponding to the image as a registered document based on a predetermined rule, and
A partial layout that is a layout including a text block corresponding to an item to be processed and at least one text block other than the text block corresponding to the item to be processed among the text blocks specified in the registered document. Based on, an estimation means for estimating a text block corresponding to the item to be processed in the image, and
An extraction means for extracting a character string in the estimated text block as a character string corresponding to the item to be processed, and
An image processing device characterized by having. The estimation means
By superimposing the partial layout on any position in the search range in the image, a degree of matching based on the size of the area where the text block included in the partial layout and the text block in the image overlap is derived. The image processing apparatus according to claim 1, wherein the estimation is performed. The estimation means
2. The claim 2 is characterized in that the degree of matching is derived with a predetermined area including a position in the image corresponding to the position of the text block corresponding to the item to be processed in the registered document as the search range. The image processing apparatus according to. The estimation means
Based on the difference in vertical position between the text block included in the partial layout and the text block in the search range, a vertical position group for estimating the text block corresponding to the item to be processed is determined. The image processing apparatus according to any one of claims 2 or 3, wherein the image processing apparatus is characterized by the above. The estimation means
The image processing apparatus according to claim 4, wherein the degree of coincidence at each position is derived when the partial layout is superimposed in the horizontal direction of the vertical position group within the search range. The estimation means
The image processing apparatus according to any one of claims 2 to 5, wherein the estimation is performed based on a position in the image in which the degree of matching is equal to or higher than a threshold value and the degree of matching is maximized. .. The estimation means
A candidate position in the image where the degree of coincidence is equal to or higher than the threshold value is determined, and the candidate position is determined.
When the number of the candidate positions is one, the candidate positions are determined as positions in the image for performing the estimation.
When the number of the candidate positions is two or more,
When the text block included in the partial layout is superimposed on any position in the registered document, the registered document whose degree of matching derived by the same method as the method for deriving the degree of matching is equal to or greater than the threshold value. The image according to claim 5, wherein the position in the image for performing the estimation is determined by acquiring the position in the image as a similar position and associating the candidate position with the similar position. Processing equipment. The estimation means
When the number of the candidate positions is two or more and the number of the candidate positions and the number of similar positions are the same.
As a result of associating the candidate positions arranged under the same conditions with the similar positions in order from one side, they are associated with the similar positions corresponding to the positions of the text blocks corresponding to the items to be processed. The image processing apparatus according to claim 7, wherein the candidate position is determined as a position in the image for performing the estimation. When the number of the candidate positions is two or more, and the number of the candidate positions and the number of similar positions are different.
As a result of associating the candidate positions arranged under the same conditions and the similar positions in order from one side, the candidate positions and the similar positions corresponding to the positions of the text blocks corresponding to the items to be processed are associated with each other. The first position indicated by the candidate position and
As a result of associating the candidate positions arranged under the same conditions with the similar positions in order from the other side, the candidate positions are associated with the similar positions corresponding to the positions of the text blocks corresponding to the items to be processed. The second position indicated by the candidate position was obtained, and
According to claim 7 or 8, the position of the first position and the second position that satisfies a predetermined condition is determined as a position in the image for performing the estimation. The image processing apparatus described. The estimation means
A text block corresponding to the item to be processed in the registered document is arranged at a position in the image determined based on the degree of matching, and a text block in the image that overlaps with the arranged text block is a predetermined value. The image processing apparatus according to any one of claims 2 to 9, wherein when the conditions are satisfied, the overlapping text blocks are estimated to be text blocks corresponding to the item to be processed in the image. The predetermined conditions are
The image according to claim 10, wherein the degree of overlap between the text block corresponding to the item to be processed and the overlapping text block is equal to or greater than a predetermined value and the distance between the vertices is within a certain range. Processing equipment. A predetermined range is set based on the text block corresponding to the item to be processed in the registered document.
The degree of agreement is
When the partial layout is superimposed on the image, the larger the area of the text block that does not overlap with the text block included in the partial layout among the text blocks included in the predetermined range in the image, the higher the degree of matching. The image processing apparatus according to any one of claims 2 to 11, wherein the image processing apparatus is adjusted so as to be lowered. The text block included in the partial layout is included in a predetermined range based on the text block corresponding to the item to be processed in the registered document and the text block corresponding to the item to be processed in the registered document. The image processing apparatus according to any one of claims 1 to 12, characterized in that it is determined by a text block. The predetermined range is
12. The image processing apparatus according to. The specific means
To identify the registered document based on the similarity of the layout of the text blocks by comparing the layout of the text blocks in the image detected by the detection means with the layout of the text blocks of the document group. The image processing apparatus according to any one of claims 1 to 14, which is characterized. 6. Image processing equipment. Further having a correction means for correcting the image or text blocks in the image,
The image processing apparatus according to any one of claims 1 to 16, wherein the estimation means makes the estimation based on a text block in the image based on the correction. The image processing apparatus according to claim 17, wherein the correction by the correction means includes at least one of tilt correction, rotation correction, and alignment with the registered document. The extraction means
The feature is that the text block corresponding to the item to be processed in the estimated image is subjected to OCR processing, and the character string obtained as a result of the OCR processing is extracted as the character string corresponding to the item to be processed. The image processing apparatus according to any one of claims 1 to 18. Further having a storage means for storing the rules for each document group for making the estimation,
The image processing according to any one of claims 1 to 19, wherein the estimation means acquires the rule of the registered document specified by the specific means and performs the estimation based on the rule. Device. The invention according to any one of claims 1 to 20, further comprising a setting means for setting the property of the image data based on the character string corresponding to the item to be processed extracted by the extraction means. Image processing equipment. The acquisition step to acquire the image data of the manuscript and
A detection step for detecting a text block in an image indicated by the image data, and
From the information that defines the layout of the text block for each document group, a specific step that identifies the document corresponding to the image as a registered document based on a predetermined rule, and
A partial layout that is a layout including a text block corresponding to an item to be processed and at least one text block other than the text block corresponding to the item to be processed among the text blocks specified in the registered document. Based on the estimation step of estimating the text block corresponding to the item to be processed in the image,
An extraction step of extracting a character string in the estimated text block as a character string corresponding to the item to be processed, and
An image processing method characterized by having. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 21.

Images