JP2023076051A - Information processing apparatus, program, and method for controlling information processing apparatus - Google Patents

Abstract

To use layout analysis or natural language analysis according to characteristics of pages, and automatically determine a page division position of scan data.SOLUTION: An information processing apparatus has determination means that determines if a first page satisfies a predetermined condition, when the determination means determines that the page satisfies the predetermined condition, decides whether to divide document data at a position between the first page and a second page by using information on a layout, and when the determination means does not determine that the page satisfies the predetermined condition, decides whether to divide the document data at the position between the first page and the second page by using a text.SELECTED DRAWING: Figure 5

Description

The present invention relates to an information processing apparatus, program, and control method for automatically determining division positions of a scanned document of multiple pages.

In recent years, a method of converting scan data obtained by reading a paper document with a scanner into a file of a predetermined format and transmitting the file to a storage server on a network for storage has been widely used. As a use case in which such a method is used, a multi-page paper document is collectively scanned and digitized, and the multi-page scanned data is divided and stored in a storage server.

Japanese Patent Application Laid-Open No. 2002-200002 discloses a technique for automatically dividing multiple pages of scan data into multiple documents. In Patent Document 1, only layout analysis, only natural language analysis, or both are applied to a scan data group to extract a feature amount of each scan data. A method of automatically determining the division position of the scan data (for example, division between the scan data of the 10th page and the scan data of the 11th page) is disclosed using the extracted feature amount.

JP-A-2006-59066

However, in the case of a document in which scan data with different features are mixed, there is a possibility that determination of division positions will be erroneous if determination is made using only one of the analysis methods as in Japanese Patent Application Laid-Open No. 2002-200018.

While automatic document segmentation by natural language analysis has higher accuracy than layout analysis, if the amount of text per page to be analyzed by natural language analysis is small, the accuracy of even natural language analysis is lowered. In other words, in the case of a document in which scanned pages suitable for automatic division by natural language analysis and scanned data suitable for automatic division by layout analysis are mixed, using only one analysis method leads to misjudgment of the division position. obtain.

On the other hand, if both layout analysis and natural language analysis are applied, the processing performance is degraded.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and it is an object of the present invention to automatically determine the page division position of scan data by selectively using layout analysis or natural language analysis according to the characteristics of each page. be.

a first page and a second page included in a document composed of a plurality of pages, extracting means for extracting text included in the first page;
acquisition means for acquiring information about the layout of the first page;
An information processing device having
having determination means for determining whether the first page satisfies a predetermined condition;
If the determining means determines that the predetermined condition is satisfied, the document is divided between the first page and the second page using the layout information acquired by the acquiring means. decide whether
If the determining means does not determine that the predetermined condition is satisfied, whether to split the document between the first page and the second page using the text extracted by the extracting means? is characterized by determining

According to the present invention, layout analysis or natural language analysis can be selectively used according to the characteristics of each page, and page division positions of scan data can be automatically determined.

It is a figure which shows the structural example of an image processing system. It is a figure which shows the hardware structural example of the processing terminal which comprises this system. It is a figure which shows the software structural example of the processing terminal which comprises this system. 4 is a flowchart for explaining the overall processing in the first embodiment; 9 is a flowchart showing details of document division position determination processing performed by an image processing unit according to the first embodiment; FIG. 3 is a diagram showing an example of a screen displayed by an image processing device or an information processing terminal; 9 is a flowchart for explaining the overall processing in the second embodiment; 10 is a flowchart showing details of document division position determination processing performed by an image processing unit according to the second embodiment;

EMBODIMENT OF THE INVENTION Hereinafter, the form for implementing this invention is demonstrated using drawing. It should be noted that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the solution of the invention.

[First embodiment]
<Overall composition>
FIG. 1 is a diagram showing an example of the configuration of an image processing system according to this embodiment. The image processing system has an image forming apparatus 110 , an image processing server 120 , an information processing terminal 130 and a storage server 140 . These devices and servers are interconnected by a network 150 and are communicable.

The image forming apparatus 110 of this embodiment can request image transmission of scan data to the storage server 140 via the image processing server 120 .

In addition, in the present embodiment, the image forming apparatus 110 is described as an example of a multifunction device having a scanning function, a printing function, a copying function, etc., but the image forming device 110 is not limited to the multifunction device. For example, any device having a scanning function can execute the processing of the present invention, which will be described later. Here, the scan function is a function of transmitting image data (scan data) generated by reading a document with a scanner provided in the image forming apparatus 110 to the outside. The print function is a function of printing print data received from the information processing terminal 130 or the like. A copy function is a function of obtaining a copy of a document by printing image data of the document read by a scanner.

Note that the image processing system of this embodiment includes the image forming apparatus 110, the image processing server 120, the information processing terminal 130, and the storage server 140, but is not limited to this. For example, the image forming apparatus 110 may also serve as the information processing terminal 130 and the image processing server 120 . Also, the image processing server 120 may be connected to a server on a LAN rather than on the Internet. Alternatively, the storage server 140 may be replaced by a mail server or the like, and the scanned image may be attached to an e-mail and transmitted.

<Hardware Configuration of MFP>
FIG. 2 is a diagram showing an example of the hardware configuration of the image processing system according to this embodiment. Image forming apparatus 110 has printer 201, scanner 202, operation unit 203, CPU 211, RAM 212, HDD 213, network I/F 214, printer I/F 215, scanner I/F 216, operation unit I/F 217, and extension I/F 218. . CPU 211 can exchange data with RAM 212 , HDD 213 , network I/F 214 , printer I/F 215 , scanner I/F 216 , operation unit I/F 217 and extension I/F 218 . In addition, the CPU 211 expands instructions (computer programs) read from the HDD 213 into the RAM 212 and executes the instructions expanded into the RAM 212, thereby controlling the execution of each process described later. Note that in the present embodiment, one CPU 211 uses one memory (RAM 212 or HDD 213) to execute each process shown in flowcharts described later, but the present invention is not limited to this. For example, multiple CPUs, multiple RAMs or HDDs may cooperate to execute each process.

The HDD 213 can store commands executable by the CPU 211, setting values used in the image forming apparatus 110, data related to processing requested by the user, and the like. The RAM 212 is an area for temporarily storing instructions read from the HDD 213 by the CPU 211 . The RAM 212 can also store various data necessary for executing instructions. For example, in image processing, the received data can be processed by developing it in the RAM 212 .

A network I/F 214 is an interface for network communication with devices in the image forming system. The network I/F 214 can transmit data reception to the CPU 211 and transmit data on the RAM 212 to the network 150 according to instructions from the CPU 211 . A printer I/F 215 is capable of transmitting print data to be printed to the printer 201 according to an instruction from the CPU 211 and transmitting the printer status received from the printer 201 to the CPU 211 .

A scanner I/F 216 transmits an image reading instruction from the CPU 211 to the scanner 202 and transmits image data received from the scanner 202 to the CPU 211 . Further, it is possible to transmit information on the status of the scanner received from the scanner 202 to the CPU 211 . The operation unit I/F 217 can transmit an instruction from the user through the operation unit 203 to the CPU 211 and can cause the operation unit 203 to display screen information for user operation.

An extension I/F 218 is an interface that allows an external device to be connected to the image forming apparatus 110 . The expansion I/F 218 has, for example, a USB (Universal Serial Bus) format interface. By connecting an external storage device such as a USB memory to the expansion I/F 218, the image forming apparatus 110 can read data stored in the external storage device and write data to the external storage device. It is possible.

The printer 201 can print image data received via the printer I/F 215 on paper, and can also transmit the status of the printer 201 to the printer I/F 215 . The scanner 202 can transmit image data obtained by reading a document (paper) placed on the scanner to the scanner I/F 216 according to an image reading instruction received via the scanner I/F 216 . The scanner 202 can also transmit the status of the scanner to the scanner I/F 216 . An operation unit 203 is an interface for giving various instructions to the image forming apparatus 110 based on user operations. For example, the operation unit 203 has a touch panel liquid crystal screen, displays an operation screen, and receives an operation from the user.

The image processing server 120 is composed of a CPU 221 , a RAM 222 , an HDD 223 and a network I/F 224 . The CPU 221 is in charge of controlling the entire apparatus, and is capable of controlling data transfer among the RAM 222 , HDD 223 and network I/F 224 . Further, the CPU 221 develops a control program (instruction) read from the HDD 223 in the RAM 222 and executes it. The HDD 223 of the image processing server 120 is a large-capacity storage unit that stores image data and various programs.

The information processing terminal 130 includes a CPU 231 , a RAM 232 , an HDD 233 , a network I/F 234 , an operation section I/F 235 and an operation section 236 .

The CPU 231 controls the entire apparatus, and can control data transfer among the RAM 232 , HDD 233 , network I/F 234 , operation unit I/F 235 , and operation unit 236 . In addition, the CPU 231 develops a control program (instruction) read from the HDD 233 in the RAM 232 and executes it. The operation unit I/F 235 is an interface that transmits to the CPU 231 a user's instruction input from the operation unit 236 and transmits information regarding an operation screen to be displayed to the operation unit 236 based on display control by the CPU 231 . An application for checking image data is installed in the information processing terminal 130. By executing the application, the image data of the image processing server is displayed, and the function of requesting the storage of the image data can be used. can be done. If the application for checking the image data is provided as a web application, the information processing terminal 130 executes the web application via the web browser to request image data display and image data storage. Any form is acceptable.

The storage server 140 is composed of a CPU 241 , a RAM 242 , an HDD 243 and a network I/F 244 . The CPU 241 is in charge of controlling the entire apparatus, and is capable of controlling data transfer among the RAM 242 , HDD 243 and network I/F 244 . Also, the CPU 241 develops a control program (instruction) read from the HDD 243 in the RAM 242 and executes it. The HDD 243 of the storage server 140 can store image data received from the image processing server 120 .

<Software configuration of image processing system>
FIG. 3 is a diagram showing an example of the software configuration of the image processing system according to this embodiment.

The image forming apparatus 110 has a controller 310 . The controller 310 includes a control section 311 , a storage section 312 , a communication section 313 , a display section 314 and a scanning section 315 .

The control unit 311 has functions of overall processing related to the functions of the controller 310 .

A storage unit 312 has a function of saving settings related to scanning and a function of saving a scanned image.

The communication unit 313 transmits the scanned image data and the setting information related to scanning to the image processing server 120 via the network 150 .

A display unit 314 displays a UI screen for accepting user operations. A scanning unit 315 receives a scanning request from the control unit 311 and executes scanning processing on the scanner 202 via the scanner I/F 216 . Also, the image data received by scanning is transmitted to the control unit 311 .

The image processing server 120 has an image processing service 320 . The image processing service 320 is composed of a control section 321 , a storage section 322 , a communication section 323 and an image processing section 324 . The control unit 321 has functions for overall processing related to the functions of the image processing service 320 . The storage unit 322 has a function of saving settings related to scanning, a function of saving image data received from the image forming apparatus 110, and a function of saving a learned document for matching image data.

The communication unit 323 has a function of receiving image data from the image forming apparatus 110 , a function of transmitting a processing result to the information processing terminal 130 , a function of receiving a scan image storage request from the information processing terminal 130 , and a function of transmitting an image to the storage server 140 . It has the function of sending data.

The image processing unit 324 performs character area analysis, OCR (Optical Character Recognition), image analysis such as image rotation and tilt correction, and image processing such as binarization. The image analysis processing includes document division position determination processing for determining the document division position by inputting the OCR results of the preceding and succeeding pages, learning based on the feature amount, and matching determination with the learned document.

<Overall processing flow>
FIG. 4 is a sequence diagram showing the flow of processing for converting an image scanned by the image forming apparatus 110 into a file and transmitting the file to the storage server 140. As shown in FIG. Here, the explanation will focus on exchanges between devices.

Although FIG. 4 shows that the image forming apparatus 110 exchanges analysis result acquisition and screen display with the image processing server 120, the information processing terminal 130 instead of the image forming apparatus 110 may execute these operations.

The control unit 311 of the image forming apparatus 110 receives a scan request from the user via the display unit 314 and executes scanning via the scanning unit 315 (S401). The control unit 311 of the image forming apparatus 110 transmits the scan data generated in S401 to the image processing server 120 via the communication unit 313 (S402).

The control unit 321 of the image processing server 120 receives the scanned image (scan data) transmitted in S402 via the communication unit 323, and performs document division position determination processing via the image processing unit 324 (S403). . Details of the processing will be described later.

When the control unit 311 of the image forming apparatus 110 detects the completion of processing from the image processing server 120 via the communication unit 313, it requests the image processing server 120 for the determination result, and transmits the determination result information of the document division position in S403. Acquire (S404).

The control unit 311 of the image forming apparatus 110 uses the determination result information acquired in S404 via the display unit 314 to perform document division for determining document division positions from a scan image group including a plurality of scan images. A position determination screen is displayed (S405).

The control unit 311 of the image forming apparatus 110 receives the correction of the document division position from the user on the document division position confirmation screen displayed in S405 via the display unit 314, and changes the document division position on the document division position confirmation screen. update. The details of the processing on the document dividing position determination screen will be described later (S406).

When the "Send" button is pressed on the division position determination screen, the control unit 311 of the image forming apparatus 110 transmits the determination result information updated in S406, the file name of each document, the file format, etc. via the communication unit 313. The information is transmitted to the image processing server 120 (S407).

Upon receiving the data transmitted in S407 via the communication unit 323, the control unit 321 of the image processing server 120 generates a plurality of files from the scan image group based on information such as the determination result information and the file name (S408). ). The control unit 321 of the image processing server 120 transmits the plurality of files generated in S408 to the storage server 140 via the communication unit 323 (S409).

FIG. 5 is a flow chart showing the details of document division position determination processing performed by the image processing unit 324 of the image processing server 120 in this system. This flow corresponds to S403 in FIG.

The image processing unit 324 acquires the scanned image group and information on the total number of pages of the scanned image group from the image forming apparatus 110 (S501). Page 1, page 2, and page 3 are assigned to each side in the order of scanning, and the scanned image group and the total number of pages of the scanned image group are stored in the storage unit 322 .

The image processing unit 324 determines whether there are two or more pages of the scan image group acquired in S501 (S502). If there are not two or more pages, the document division position cannot exist, so the document division position determination is skipped and the process advances to step S515. Otherwise, the process proceeds to S503.

The image processing unit 324 acquires the first page of the scan image group (S503). When proceeding from S503 or S514 to S504, the image processing unit 324 acquires the next page of the previously processed page. If the previous page is the first page, the second page is acquired (S504).

The image processing unit 324 acquires text information from the scanned images of the preceding and succeeding pages (S505). The preceding and succeeding pages here correspond to the scanned image acquired in S504 and the scanned image acquired in S503. The image processing unit 324 executes character area analysis and OCR (Optical Character Recognition) processing on all areas of the preceding and succeeding pages. Then, the character area and text are extracted, and the area number, coordinates (X coordinate, Y coordinate, width, height), text, and OCR certainty of the character area are stored in the storage unit 322 in association with the page number.

Table 1 is a table showing an example of the character area extraction result. As the result of character area extraction, a character area consisting of the area number, X coordinate, Y coordinate, width, and height of the character area is obtained. OCR confidence is a statistical measure of how correct the text data resulting from OCR is. The OCR confidence may be expressed as an average value of the confidence of each character or as a word (high, medium, low, etc.). Although the value range of the OCR certainty can take any range, it is set to a value of 0 to 1 in this embodiment.

Of the pages other than the first page, the scanned image positioned as the previous page has already been processed as the subsequent page in step S505 executed immediately before, so there is no need to perform the processing again. Specifically, when performing S505 in a situation where the first page is positioned as the previous page and the second page is positioned as the subsequent page, the processing of S505 is executed for the first and second pages. . After that, when the third page is newly acquired in S504 and the processing of S505 is executed, the processing of S505 has already been executed for the second page, so S505 is executed only for the newly acquired third page. process is executed.

The image processing unit 324 calculates text reliability for the scanned images of the preceding and following pages (S506). The text reliability is a criterion for determining whether or not the target scanned image is an image suitable for natural language analysis in S508, which will be described later. Specifically, it is calculated from the amount of text present in the scanned image and the OCR certainty obtained in S505.

For example, if the amount of text is less than a certain value, or even if the amount of text is a certain amount, if the average value of the OCR certainty is low, the reliability of the text decreases. In this case, on the premise that the importance of the text changes depending on the position of the character area, weighting may be performed on the text reliability in consideration of the coordinates of the character area. The image processing unit 324 associates the text reliability with the page number and stores it in the storage unit 322 .

Although the text reliability is calculated from the amount of text and the OCR certainty, it is not limited to this, and the text reliability may be calculated including information related to the image quality such as the resolution of the scanned image. . As in S505, the processing of S506 may be executed only for the first page and the page newly acquired in S504.

The image processing unit 324 determines whether the text reliability calculated in S506 is higher than a preset threshold (S507). If both the preceding and following pages are higher than the threshold, the process proceeds to S508, and if the text reliability of either page is lower than the threshold, the process proceeds to S510. Since the text reliability increases as the amount of text per page increases, normally most pages other than the cover page and the last page will have a text reliability higher than the threshold, and the process proceeds to S508.

If it is determined in S507 that the text reliability levels of both the preceding and succeeding pages are higher than the threshold, the image processing unit 324 performs natural language analysis (S508). In the natural language analysis, a feature amount to be input to the split confidence calculation model is generated from the text data, which is the OCR processing result of the preceding and succeeding pages acquired in S505. The division certainty calculated by the division certainty calculation model is used as an index for judging whether the document division position exists between the preceding and succeeding pages in the document division position judgment processing in S513.

The feature amount extracted from the text data that is the result of OCR processing may be changed according to the type of model that uses the feature amount. For example, if the split confidence calculation model is a known BERT (Bidirectional Encoder Representations from Transformers)-based model, it is necessary to generate tokens that can be input to the BERT from text data. Various known methods may be used, but for example, a method of combining character strings including the preceding and succeeding pages, further dividing the combined character string into words by bigram processing or morphological analysis processing, and tokenizing. Conceivable.

After the feature amount is generated by the natural language analysis, the image processing unit 324 calculates the division confidence (S509). Specifically, the split confidence is obtained by inputting the feature amount generated in S508 into the split confidence calculation model. In general, even in a judgment model (division confidence calculation model in this embodiment) that makes a binary judgment as to whether or not it is a document division position, the confidence that is a continuous value (division confidence in this embodiment) degree) expresses the determination result. It is common to perform a final judgment process (judgment as to whether or not it is a document division position) by performing threshold processing on this certainty.

Threshold processing will now be described. For example, machine learning outputs a degree of certainty of 0.87 (=87%) to the question whether this photo is of a cat. By setting a threshold value for that value, it is judged as a binary value of YES or NO. As a method of setting the threshold, there is a setting method in which "YES" (it is a cat) is determined when the degree of certainty is 0.7 or more. A threshold is set for the calculated certainty, and the determination result is binarized depending on whether it is larger or smaller than the threshold.

If it is determined in S507 that the text reliability of either the previous or subsequent page is lower than the threshold, the image processing unit 324 determines whether the text reliability of the subsequent page is higher than the threshold. If the text reliability of the subsequent page is higher than the threshold, the process advances to S513, and the layout analysis of S511 and the document classification of S512 are not performed.

The reason why the processing of S511 and S512 is not performed when the text reliability of the subsequent page is higher than the threshold (the text reliability of the previous page is lower than the threshold) is that the text reliability of the document is determined to be higher than the threshold. indicates that the amount of text is relatively large, it is not necessary to classify documents such as "cover" and "end of sentence", which have relatively small amount of text, in the document classification process of S512, which will be described later. Even if the processing of S511 and S512 is performed, there is a high possibility that it will be classified into "others" other than "cover" and "end of sentence". Document division positions of documents classified as "others" are checked by the user in step S406, so an increase in the number of documents classified as "others" leads to an increase in user load. Therefore, the document whose text reliability is higher than the threshold proceeds to S513 without performing the processing of S511 and S512.

On the other hand, if it is determined in S510 that the text reliability of the subsequent page is lower than the threshold, the process proceeds to S511.

The image processing unit 324 performs layout analysis on the scanned image of the subsequent page (S511). The feature amount acquired as a result of layout analysis is used for document classification in S512, which will be described later. The feature amount here includes, for example, the feature amount of a so-called document format such as character size, character spacing, line spacing, margins, and coordinates of character areas. However, it is not always necessary to perform a new layout analysis in S511, and the information of the X coordinate, Y coordinate, width, and height of the character area extracted in S505 can be used as the analysis result in S511. good. Also, the layout analysis is not limited to the above method, and other commonly known methods for extracting feature amounts that enable document classification may be used.

The image processing unit 324 performs document classification on the scanned image layout-analyzed in S511 (S512). Specifically, the scanned image is classified into one of "cover", "end of sentence", and "others". In general, the characteristics of the cover layout include "heading with large characters", "wide blank space", "character area close to the center", and "wide line spacing". On the other hand, the characteristics of the layout of the end page include "characters are biased toward the top of the page," "wide blank space at the bottom of the page," "characters are small and uniform relative to the cover page," and "constant and narrow line spacing." mentioned. Scanned images are classified into "cover", "end of article", and "others" based on the general characteristics of the cover layout and the layout of the end page.

As a specific classification method, each time the scanned image matches the above feature, it is added, and if the added value is higher than the threshold, it is classified. A method of classifying by comparing the degree of matching with the vector quantity of a typical cover page or the last page of the text can be considered. The classification result is stored in the storage unit 322 in association with the page number.

The image processing unit 324 performs document division position determination processing (S513). Threshold processing is performed on the division reliability obtained in the process of S509 to determine the document division position. If it exceeds a preset threshold, it is determined to be a document division position, and if it is less than the threshold, it is determined to be a non-document division position.

For the classification result acquired in the process of S512, the document division position is determined from the result of classification in S512 and the text reliability calculated in S506 (S513). A document whose back page is classified as "cover" is separated from the front page. A document whose back page is classified as "end of sentence" is not separated from the front page. However, only when the previous page is classified as "end of sentence", the determination of division is suspended. A document whose back page is classified as "other" also suspends the determination of division.

Table 2 shows the result of whether or not to divide the front and rear pages according to the combination of the classification results of the front and rear pages. However, the combination of the text reliability "high" and the text reliability "high" is determined by document division position determination processing using the division certainty obtained in S509.

Table 2 shows an example of the relationship between the preceding and succeeding pages determined by combining the classification results of the preceding and succeeding pages and the text reliability. "Split" in Table 2 indicates that the split position is between the previous and next pages, "-" indicates that the split position is not between the previous and next pages, and "Retain" indicates that the split position is between the previous and next pages. , and prompts confirmation on the document division position confirmation screen in S405. As shown in Table 2, the preceding and succeeding pages determined to have a text reliability of "low" in S507 are classified into one of "cover", "end of sentence", and "others". Depending on the combination of the classification result of the previous page and the classification result of the subsequent page, it is determined as "divided", "-(not divided)", or "suspended". These determination results are stored in the storage unit 322 as intermediate information.

The image processing unit 324 determines whether or not there is an unprocessed scanned image (S514). If there is an unprocessed scanned image, the process returns to S504. If all scanned images have been processed, the process advances to step S515.

The image processing unit 324 generates determination result information from the division determination result in S513 and the document classification result in S512 stored in the storage unit 322 (S515). The determination result information holds an ID number assigned to each divided image group, the number of pages constituting one image group, document classification results of individual documents, and the like. The generated determination result information is saved in the storage unit 322 . The above is the detailed processing of S403.

FIG. 6A is a diagram showing an example of a document dividing position determination screen displayed by the image forming apparatus 110 or the information processing terminal 130. FIG. Screen 610 is displayed in S405. A screen 610 displays a list of scanned images before transmission to the storage server 140 after completion of scanning and document division position determination processing. Further, on the screen 610, it is possible to confirm and correct the determination result of the document division position for the scan image group (determination result of S403).

On the screen 610, the scanned image is displayed in a divided state based on the determination result of S403. A screen 610 displays a file name 611 for each document group, a total number of pages 612 for each document group, a thumbnail image 613 of the document group, and a page number 614 . Furthermore, a document dividing line 615 exists on the right side of the last page of each document group, and the document dividing position can be corrected by dragging the document dividing line 615, for example.

The document division position correction method may be adjusted by dragging the thumbnail image 613 . Also, the thumbnail image 616 determined to be “suspended” in S403 is displayed so as to be distinguished from the other thumbnail images. For example, the thumbnail image 616 may be surrounded by a thick frame or displayed in a different color. A "send" button 617 is a button for confirming the document splitting position. Information such as the format is sent to the image processing server 120 .

A "batch confirmation" button 618 is a button for transitioning to a screen for collectively confirming and correcting document division positions. A list is displayed to select whether or not.

FIG. 6B is an example of a collective confirmation screen 620 displayed by the image forming apparatus 110 or the information processing terminal 130. As shown in FIG. The thumbnail image displayed on the collective confirmation screen 620 is the image of the page determined to be “suspended” when determining the document division position.

It is displayed when the “Confirm All” button 618 on the screen 610 is selected. On the collective confirmation screen 620, a list of pending thumbnail images 621 and tags 622 indicating the affiliation of the pending thumbnail images is displayed. When the reserved thumbnail image 621 is selected by the user, it is displayed as a reserved thumbnail image 623 whose cover has been determined in a manner distinguishable from surrounding documents. A “return” button 624 is a button for transitioning to the document dividing position determination screen 610 , and when the user presses it, the transition is made without saving the changes on the collective confirmation screen 620 . A “confirm” button 625 is a button for saving changes made on the collective confirmation screen 620 . When pressed by the user, a transition is made to a screen 610 to display a new group of images reconstructed from the selected image as the cover.

By carrying out the present invention according to the above processing procedure, documents with features that cannot be captured by layout analysis or natural language analysis coexist in the process of automatically determining document division positions from a group of scanned images containing multiple documents. It is possible to improve the accuracy of document segmentation for an image group. Furthermore, processing performance is improved by changing the analysis method according to the characteristics of the document. In addition, since the accuracy improvement can reduce the number of man-hours for correcting a portion where the automatic determination is incorrect, it is possible to reduce the user's load.

[Second embodiment]
In the first embodiment, document classification was performed based on feature amounts extracted by layout analysis. In this embodiment, it will be explained that accurate classification can be achieved by using matching with learned documents when classifying documents. Document learning is mainly performed on the document corrected by the user on the document division position determination screen. In the description of this embodiment, the description of the portions having the same configuration and processing procedure as those of the first embodiment will be omitted, and only the portions with differences will be described.

Differences from the first embodiment are the determination results of the flowcharts of FIGS. 7 and 8. FIG. FIG. 7 is a sequence diagram showing the flow of processing for converting an image scanned by the image forming apparatus 110 into a file and transmitting the file to the storage server 140. As shown in FIG. The difference from FIG. 4 is S701.

In S701, the image processing unit 324 learns the feature amount for the document corrected by the user based on the determination result information transmitted in S407. The main documents to be learned are the documents that were determined to be the "cover" or the "end" in S512, but were not placed on the cover or the end of the document group due to the user's correction in S406, and the documents that were determined to be "other" in S512. It is a document that has been determined but placed on the front cover or the end of the document group as a result of the user's correction in S406. Documents that are determined to be "cover" or "end of sentence" but are not placed on either the cover or the end of the document group are reclassified into "text".

On the other hand, documents that are determined to be "others" but are placed on the cover or at the end of the document group are reclassified into "cover" or "end of text". "Text" is the result of classification only in document learning in S701. Note that the documents to be learned may be documents other than those modified by the user. As a method of learning the feature amount, for example, the coordinate information of the character area, which is the character area extraction result extracted in S505, and the classification result are linked and stored. In addition, text information, which is the result of OCR processing, may also be stored. The learned document is stored in the storage unit 322 as a learned document. By linking and storing the feature amount and the reclassified classification result, it becomes possible to determine the classification result of the document that matches the feature amount without the document classification in S512.

FIG. 8 is a flow chart showing the details of the document division position determination process performed by the image processing unit 324 of the image processing server 120 in this system. This flow corresponds to S403 in FIG. Differences from FIG. 5 are S801, S802, and S803.

After layout analysis is performed on the scanned image in S511, the image processing unit 324 performs document matching on the scanned image subjected to layout analysis (S801). Document matching compares the feature amount learned in S701 with the feature amount of the scanned image to be processed, and determines whether or not the document is similar. For example, when the feature amount is coordinate information of a character region, determination is made based on whether or not the ratio of the overlapping area of the character regions to the area of the entire learned character region is equal to or greater than a threshold.

Moreover, if text information is also included as a feature amount, whether or not the text in the character area matches may also be added to the determination. In the first embodiment, even if the user corrects a misclassified document, the same determination will be made if a similar document is scanned. The image processing unit 324 learns the document modified by the user in S701, and acquires the classification result associated with the matched learned document from the storage unit 322 when the similar document is scanned. The classification result can be the classification result of the scanned image to be processed. The classified results are "cover", "end of sentence", and "text". The classification result is stored in the storage unit 322 in association with the page number.

As a result of executing document matching, the image processing unit 324 determines whether the scanned image is a learned document (S802). If it is determined that a learned document exists, the process advances to S803, and the document classification processing of S512 is not performed. If it is determined that the learned document does not exist, the process proceeds to S512.

Next, the processing of S803 will be described. In this embodiment, in addition to "cover", "end of sentence", and "others", "text" is added as a document classification in S701. Therefore, the number of combinations indicating the relationship between the preceding and succeeding pages increases to 20 in total. Table 3 is shown by adding differences to Table 2. As in Table 2, "Split" indicates that the split position is between the previous and next pages, "-" indicates that the split position is not between the previous and next pages, and "Hold" indicates that the split position is between the previous and next pages. This indicates that it is not possible to determine whether or not it is the division position, and prompts confirmation on the document division position confirmation screen in S405.

By carrying out the present invention according to the above processing procedure, in addition to the effect of the first embodiment, accurate classification can be achieved by using matching with already-learned documents. This makes it possible to further improve the accuracy of document segmentation and reduce the user's burden.

This completes the description of the present embodiment.

[Other Examples]
The object of the present invention is also achieved by executing the following processing. That is, a storage medium recording software program code for realizing the functions of the above-described embodiments is supplied to a system or device, and the computer (or CPU, MPU, etc.) of the system or device executes the program stored in the storage medium. This is the process of reading the code. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code and the storage medium storing the program code constitute the present invention.

110 image forming apparatus 120 image processing server 130 information processing terminal 140 storage server 311, 321 control unit 312, 322 storage unit 313, 323 communication unit 314 display unit 315 scanning unit 324 image processing unit

Claims (9)

a first page and a second page included in a document composed of a plurality of pages, extracting means for extracting text included in the first page;
acquisition means for acquiring information about the layout of the first page;
An information processing device having
having determination means for determining whether the first page satisfies a predetermined condition;
If the determining means determines that the predetermined condition is satisfied, the document is divided between the first page and the second page using the layout information acquired by the acquiring means. decide whether
If the determining means does not determine that the predetermined condition is satisfied, whether to split the document between the first page and the second page using the text extracted by the extracting means? An information processing apparatus characterized by determining The determination means is
2. The information processing apparatus according to claim 1, wherein it is determined whether one of the OCR certainty of said first page and the amount of text of said first page is below a predetermined threshold. the first page is included in a multi-page document;
If the determining means determines that the predetermined condition is satisfied, the first page is classified into a cover page, an end page, or other pages using the layout information. , determining whether to divide the first page from the second page based on the text reliability calculated based on the OCR reliability and the text amount, and the classification result of the first page. 3. The information processing apparatus according to claim 2, characterized by: When the text reliability of the first page is low and it is determined that the layout of the first page is the first page, the first page and the page located before the first page. 4. The information processing apparatus according to claim 3, wherein said document is divided between. between the first page and the page after the first page when the text reliability of the first page is low and the layout of the first page is determined to be the last page 5. The information processing apparatus according to claim 3, wherein the document is divided. If the first page cannot be classified as either a cover page or an end page using the layout information,
6. The method according to any one of claims 3 to 5, wherein a judgment result as to whether or not the division position is between the first page and a page positioned before or after the first page is suspended. The information processing device according to the item. If the judgment result is withheld,
7. The information processing apparatus according to claim 6, wherein the division position is determined by a user. a first page and a second page included in a document composed of a plurality of pages, an extracting step of extracting text included in the first page;
a obtaining step of obtaining information about the layout of the first page;
A control method for an information processing device having
a determination step of determining whether the first page satisfies a predetermined condition;
If the determining step determines that the predetermined condition is satisfied, the document is divided between the first page and the second page using the layout information acquired by the acquiring step. decide whether
If the determining step does not determine that the predetermined condition is satisfied, the text extracted by the extracting step is used to split the document between the first page and the second page, or A control method for an information processing device, characterized by determining: a first page and a second page included in a document composed of a plurality of pages, extracting means for extracting text included in the first page;
acquisition means for acquiring information about the layout of the first page;
A program executed by an information processing device having
having determination means for determining whether the first page satisfies a predetermined condition;
If the determining means determines that the predetermined condition is satisfied, the document is divided between the first page and the second page using the layout information acquired by the acquiring means. decide whether
If the determining means does not determine that the predetermined condition is satisfied, whether to split the document between the first page and the second page using the text extracted by the extracting means? A program executed by an information processing apparatus characterized by determining

Images