JP2024032563A - Information processing device, control method and program for information processing device - Google Patents

Abstract

An information processing device, a method for controlling the information processing device, and a program are provided. [Solution] Extract character area information from scanned image data of a form, compare it with learning data to determine similar forms, recognize and learn character strings in the corresponding character area, and use the learning data for each workflow. An information processing device for storing, at the time of learning a character region to be acquired in a workflow, displays a list of learned workflows with similar forms in step S901, and displays a character region to be extracted in a learned workflow selected from the list and information. A list of items is displayed in step S903, an association with information items in the learned workflow is accepted in step S904, and learning data of the workflow is carried over to the workflow in step S909. [Effect] It is possible to easily associate the extracted character area of a learned workflow with a new workflow on the display screen, and by inheriting the learning data, the effort of learning to set metadata is reduced. Can be done. [Selection diagram] Figure 9

Description

The present invention relates to an information processing apparatus and information processing apparatus that learns and extracts character areas from a scanned image displayed on an MFP or PC screen to set values in a workflow, which is a series of processing, and carries over the learning data to another workflow. The present invention relates to a control method and program for a processing device.

Conventionally, a system that manages forms by scanning paper forms and digitizing them, and setting the file name of the digitized form image based on the contents of the form and metadata used in a management device such as a cloud service. There is. As a method for setting the file name and metadata, there is a method of setting the file name and metadata based on the recognition result obtained by performing character recognition processing on the form image. In addition, by learning the character area used by the user in file name and metadata settings, it is possible to determine whether a form with a similar format has been learned, and to use the characters used in the file name and metadata when scanned. It becomes possible to present columns to the user. In addition, by defining scan processing, setting file names and metadata values, sending data to an information processing device, etc. as a workflow, multiple settings can be made as a series of processes for each usage.

In the above system, each workflow uses different information from the form as a file name or metadata, so it is necessary to learn the character areas to be extracted for each workflow. Furthermore, when changing the information processing device that manages the form, when using a different information processing device for each purpose, or when it is necessary to change the character area to be extracted, it is necessary to create a new workflow. At that time, it is necessary to newly learn each item for each form format. For example, when migrating a cloud service used for form management to a new cloud service, the metadata items used may change. If you want to use a character string in a character area that was previously learned as metadata ``Name'' as metadata ``Title'' for a new cloud service, you will have to relearn the form for each form even if you want to use the same information. Therefore, if many formats of forms have already been learned in the previous workflow, it is necessary to relearn the same number of formats, which becomes a burden on the user.
Patent Document 1 discloses a technique for inheriting first metadata to another second metadata. A file defining items of first metadata is prepared, and a correspondence relationship between the first metadata and other second metadata that uses the same information is described in the file. When a UI for setting metadata based on this definition file is displayed and a value is set for the first metadata, the same information is given to the second metadata.

Japanese Patent Application Publication No. 2008-134915

However, in the method described in Patent Document 1, it is necessary to prepare a definition of the correspondence between metadata as a file, and the defined file must be modified every time an item of metadata changes, which imposes a workload on the user. . Furthermore, if you attempt to associate metadata with each other based only on the metadata name, there is a possibility that the association will be incorrect if there are multiple similar metadata names or if there are a large number of metadata items.
In the present invention, when setting a file name and metadata from the contents of a scanned form, if the same character area has already been learned in another workflow, it is possible to easily carry over the learning data and set the file name and metadata. The purpose is to enable the setting of In addition, when transferring file names and metadata, it is possible to check which information in the form has been selected by the transfer source, and to check whether the transfer destination is the same information as the information you want to obtain. reduce.

The present invention provides an extraction means for extracting character area information from scanned image data of a form, a determination means for comparing the extracted character area information with learning data to determine a similar form, and An information processing apparatus comprising: a recognition means for recognizing a character string in a character region corresponding to a character region; a learning means for learning the character region; and a storage means for storing learning data for the character region for each series of processing. When learning character regions to be acquired in the first series of processing, a first display means for displaying a list of a series of learned processes for similar forms of scanned images; a second display means for displaying a list of learned character regions to be extracted and information items of the second series of processes by selecting a learned second series of processes from a list of processes; a receiving means for accepting an association between an item of information to be extracted in the learned second series of processing and an item of information to be transmitted in the first series of processing for learning; The method is characterized by having a duplication means for duplicating the learning data of the series of processes into the learning data of the first series of processes.

According to the present invention, it is possible to easily associate the extracted character area of a learned workflow with a new workflow on the display screen, and by taking over the learning data, the effort of learning for setting metadata is reduced. It can be reduced.

Diagram showing the overall configuration of this system MFP hardware configuration diagram Hardware configuration diagram of client PC and MFP cooperation service Software configuration diagram of this system Sequence diagram showing the flow of processing between each device Diagram showing an example of a screen displayed by the MFP or client PC Diagram showing an example of a screen displayed by the MFP or client PC Flowchart of image analysis processing performed by the image processing unit Data structure of data saved in analysis result saving processing performed by the image processing unit Flowchart showing details of learning data transfer process Learning data transfer screen displayed by MFP or client PC Learning data transfer screen displayed by MFP or client PC Learning data transfer screen displayed by MFP or client PC Learning data transfer screen displayed by MFP or client PC Learning data transfer screen displayed by MFP or client PC

EMBODIMENT OF THE INVENTION Hereinafter, the form for implementing this invention is demonstrated using drawings. Note that the following embodiments do not limit the claimed invention, and not all combinations of features described in the embodiments are essential to the solution of the invention.

[First embodiment]
<System configuration>
FIG. 1 is a diagram showing the overall configuration of an information processing system. The information processing system includes an MFP (Multifunction Peripheral) 110, a client PC 111, an MFP cooperation service 120, and an external cloud service 130. The MFP 110 and the client PC 111 are communicably connected to various devices that provide various services on the Internet via a LAN (Local Area Network).
The MFP 110 is a multifunction device having multiple functions such as a scanner and a printer, and is an example of an image processing device. The client PC 111 is an information processing device such as a desktop terminal or a mobile terminal that can receive external cloud services via the Internet.
The MFP cooperation service 120 is a service that performs image analysis on a scanned image received from the MFP 110 and has a function of forwarding a request from the MFP 110 to another service, and is executed by an information processing device such as a server.
The cloud service 130 is an external cloud service that performs cloud storage and document management. It is a service that allows you to save files over the Internet, set metadata for files, and check metadata by acquiring files with a web browser, and is executed by an information processing device such as a server.
The information processing system of this embodiment has a configuration including an MFP 110, a client PC 111, an MFP cooperation service 120, and a cloud service 130, but is not limited thereto. For example, the MFP 110 may also serve as the client PC 111 and the MFP cooperation service 120. Furthermore, the MFP cooperation service 120 may be located on a LAN instead of on the Internet. Further, the cloud service 130 may be replaced with a mail server or the like, and the scanned image may be attached to an e-mail and sent.

<MFP hardware configuration>
FIG. 2 is a hardware configuration diagram of the MFP 110. The MFP 110 includes a control section 210, an operation section 220, a printer section 221, a scanner section 222, and a modem 223.
Control unit 210 controls the overall operation of MFP 110.
The CPU 211 reads a control program stored in the ROM 212 and executes and controls various functions of the MFP 110 such as reading/printing/communication.
The RAM 213 is used as a main memory of the CPU 211, a temporary storage area such as a work area, etc. In this embodiment, it is assumed that one CPU 211 uses one memory (RAM 213 or HDD 214) to execute each process shown in a flowchart described later, but the present invention is not limited to this. For example, each process may be executed by making multiple CPUs, multiple RAMs, or HDDs work together.

The HDD 214 is a large capacity storage unit that stores image data and various programs.
The operation unit I/F 215 is an interface that connects the operation unit 220 and the control unit 210.
The operation unit 220 is equipped with a touch panel, a keyboard, etc., and receives operations/inputs/instructions from the user.
The printer I/F 216 is an interface that connects the printer section 221 and the control section 210. Image data for printing is transferred from the control unit 210 to the printer unit 221 via the printer I/F 216 and printed on a recording medium.
Scanner I/F 217 is an interface that connects scanner section 222 and control section 210.
The scanner unit 222 reads a document set on a document table (not shown) or an ADF (Auto Document Feeder), generates scanned image data, and inputs the data to the control unit 210 via the scanner I/F 217. The MFP 110 can print out (copy) the image data generated by the scanner unit 222 from the printer unit 221, and can also send a file or email.
Modem I/F 218 is an interface that connects modem 223 and control unit 210.
The modem 223 performs facsimile communication of image data with a facsimile device on the PSTN.
The network I/F 219 is an interface that connects the control unit 210 (MFP 110) to the LAN. The MFP 110 uses the network I/F 219 to transmit image data and information to various services on the Internet, and to receive various information.
The hardware configuration of the MFP 110 described above is an example, and the MFP 110 may include other configurations as necessary, or may not include some of the configurations.

<Hardware configuration of client PC and MFP cooperation service>
FIG. 3 is a hardware configuration diagram of the client PC 111 or the MFP cooperation service 120. The client PC 111 or the MFP cooperation service 120 is an information processing device that includes at least a CPU 311, a ROM 312, a RAM 313, an HDD 314, and a network I/F 315.
The CPU 311 controls the overall operation by reading a control program stored in the ROM 312 and executing various processes.
The RAM 313 is used as a temporary storage area such as a main memory and a work area of the CPU 311.
The HDD 314 is a large capacity storage unit that stores image data and various programs.
The network I/F 315 is an interface that connects the client PC or the MFP cooperation service 120 to the Internet. The MFP cooperation service 120 and the cloud service 130 receive processing requests from other devices (such as the MFP 110) via the network I/F 315, and transmit and receive various information.

<Software configuration of information processing system>
FIG. 4 is a software configuration diagram of the image processing system according to this embodiment.
(MFP)
The software of the MFP 110 is roughly divided into two parts: a native function section 410 and an additional function section 420. Each unit included in the native function unit 410 is provided as standard in the MFP 110, whereas the additional function unit 420 is an application that is additionally installed in the MFP 110. The additional function unit 420 is an application based on Java (registered trademark), and can easily add functions to the MFP 110. Note that other additional applications (not shown) may be installed in the MFP 110.
The native function section 410 includes a scan execution section 411 and an image data storage section 412.
The additional function section 420 includes a display control section 421, a scan instruction section 422, and a cooperation service request section 423.
The display control unit 421 displays a UI screen for accepting user operations on a liquid crystal display unit having a touch panel function of the operation unit 220 of the MFP 110. For example, input of authentication information for accessing the MFP cooperation service 120, selection of a workflow which is a series of processing, operations for starting a scan, and displaying a UI screen such as a preview screen.
The scan instruction unit 422 requests the scan execution unit 411 to perform a scan process along with scan settings according to a user instruction input via the UI screen.

The scan execution unit 411 receives a scan request including scan settings from the scan instruction unit 422. The scan execution unit 411 generates scan image data by reading the document placed on the document platen glass with the scanner unit 222 via the scanner I/F 217 in accordance with the scan request. The generated scan image data is sent to the image data storage unit 412. The scan execution unit 411 sends a scan image identifier that uniquely indicates the saved scan image data to the scan instruction unit 422. The scanned image identifier is a number, symbol, alphabet, or the like (not shown) for uniquely identifying an image scanned by the MFP 110. The image data storage unit 412 stores the scan image data received from the scan execution unit 411 in the HDD 214 in association with the scan image identifier.
The scan instruction unit 422 acquires scan image data corresponding to the scan image identifier received from the scan execution unit 411 from the image data storage unit 412. The scan instruction unit 422 requests the cooperation service request unit 423 to instruct the MFP cooperation service 120 to process the acquired scan image data.
The cooperation service request unit 423 requests the MFP cooperation service 120 for various processes. For example, requests such as login, analysis of scanned images, and transmission of scanned images are made. Exchanges with the MFP cooperation service 120 use protocols such as REST and SOAP, but other communication means may also be used.
(MFP cooperation service)
The MFP cooperation service 120 includes a request control section 431, an image processing section 432, a workflow management section 433, a cloud service access section 434, a data management section 435, and a display control section 436.
The request control unit 431 is on standby in a state where it can receive a request from an external device. Upon receiving a processing request, it instructs the image processing unit 432, workflow management unit 433, cloud service access unit 434, and data management unit 435 to perform processing as appropriate according to the request.

The image processing unit 432 performs character region analysis on the image, character recognition processing on the analyzed character region (hereinafter referred to as OCR (Optical Character Recognition) processing), similar form determination, and image recognition such as image rotation and tilt correction. Perform processing and image processing. Similar document determination will be explained in detail later in the processing of S509 to S510 in FIG.
The workflow management unit 433 manages a workflow that defines a series of processing methods and settings created by the user in advance from scanning to sending to an external cloud service. Specifically, it manages a series of settings such as scan reading settings, cloud service destination settings, storage directory settings, and file name setting rules definitions.
The cloud service access unit 434 requests processing from the cloud service. Cloud services generally use protocols such as REST and SOAP to expose various interfaces for storing files in cloud services, setting metadata, and retrieving stored files. The cloud service access unit 434 operates the cloud service using the published cloud service interface.
The data management unit 435 holds user information, various setting data, etc. managed by the MFP cooperation service 120.
The display control unit 436 receives a request from a web browser running on another terminal (not shown) such as a PC or mobile connected via the Internet, and generates screen configuration information (HTML, CSS, etc.) necessary for screen display. )return it. The user checks user information registered in the MFP cooperation service 120 and changes settings for scanning via a screen displayed on a web browser.
Although FIG. 4 describes an example of a configuration in which the additional function unit 420 is installed in the MFP 110, the present invention is not limited to this configuration, and the function of the additional function unit 420 may be included in the client PC 111.

<Overall processing sequence>
FIG. 5 is a sequence diagram showing the flow of processing between devices when an image scanned by the MFP 110 is converted into a file and transmitted to the cloud service 130. Here, we will mainly explain the exchange between each device. Although FIG. 5 shows that the MFP 110 interacts with the MFP cooperation service 120, the client PC 111 may execute the acquisition of analysis results, screen display, learning instructions, etc., which will be described later, instead of the MFP 110.
In a normal state, the MFP 110 displays a main screen on the touch panel on which buttons are arranged for implementing each function provided.
By installing an additional application (hereinafter referred to as "scan application") on the MFP 110 for sending scanned forms to a cloud storage service, buttons for using the functions of the application will be displayed on the main screen of the MFP 110. When this button is pressed, a screen for transmitting the scanned form to the cloud service is displayed, and the process shown in the sequence of FIG. 5 is performed.
In S501, the scan application displays a login screen for entering authentication information for accessing the MFP cooperation service 120.
In S502, the user inputs authentication information for login, and the MFP 110 transmits the information to the MFP cooperation service 120 as a login request.
In S503, the MFP cooperation service 120 verifies whether the user name and password included in the login request are correct, and if correct, returns an access token to the MFP 110. Various subsequent requests made from the MFP 110 to the MFP cooperation service 120 are issued together with this access token, and the user to be processed can be specified using this information. User authentication is performed using generally known methods (Basic authentication, Digest authentication, authorization using OAuth, etc.).
In S504, the MFP cooperation service 120 generates a list of workflows that define scan read settings created by the user in advance, destination settings for sending to which cloud service, storage directory settings, and file name setting rules. Send to MFP 110.

In S505, the MFP 110 displays the list of received workflows on the workflow selection screen. Thereafter, the user selects a workflow from the workflow selection screen, places the paper form to be scanned on the document table glass or ADF, and presses the "Start Scan" button.
In S506, the MFP 120 performs scanning to generate image data of the paper form in electronic format.
Then, in S507, the MFP 120 transmits an analysis request for the scanned image to the MFP cooperation service 120 together with the image generated by the scanning process.
In S508, when the MFP cooperation service 120 receives the scan image analysis request at the request control unit 431, it requests the image processing unit 432 of the MFP cooperation service 120 to analyze the image. After that, the MFP cooperation service 120 returns "processId", which is an identifier uniquely indicating the analysis requested to the MFP cooperation service 120, to the MFP 110 without waiting for the end of the image analysis process.
After receiving the request, the MFP cooperation service 120 performs image analysis processing in the image processing unit 432.
In the image analysis process, first in S509, the image processing unit 432 analyzes a character area existing within the image.
Subsequently, in S510, the image processing unit 432 performs similar form determination processing using information on the arrangement of character areas within the form. Specifically, the text area placement information of images scanned in the past using the workflow used is compared with the text area placement information of the currently scanned image, and it is determined whether the images have similar text area placements. . Information on past scan images used in this determination is saved and accumulated through the process of S518, which will be described later.
Subsequently, in S511, the image processing unit 432 performs character recognition processing (OCR processing) on the analyzed area based on the determination result. Note that details of the OCR processing in S509 to S511 will be explained with reference to FIG.
The MFP 110 uses the "processId" received in the response of S507 in S512 to periodically (for example, every several hundred milliseconds to several milliseconds) inform the MFP cooperation service 120 of the processing status of the image analysis of "processId". Check. Although not shown in the figure, the process of S512 is continued until a response indicating the completion of image processing from the MFP cooperation service 120 is obtained (until the timing of S513). Upon receiving the request to confirm the processing status in S512, the MFP cooperation service 120 confirms the processing status of "processId" and returns a response.

In the response, a character string indicating the current processing status is stored in "status". For example, when "status" is "processing", it indicates that the MFP cooperation service 120 is "processing", and when "status" is "completed", it indicates that the processing is "completed". Note that if the process "failed", other statuses such as "failed" may be returned. In addition, the response upon completion of processing (when "status" is "completed") includes information such as the results of analyzing the scanned image and scan settings as well as the status.
In subsequent S513, when the MFP 110 detects the completion of the process, in S514 it acquires result information included in the response from the URL where the result information is stored.
Then, in S515, an attribute setting screen for setting a file name and metadata is displayed using the result information obtained in S514. FIG. 6B shows an example of an attribute setting screen displayed on the MFP 110. Details of the attribute setting screen will be described later.
On the attribute setting screen, set the file name and metadata for the scanned form based on the workflow settings, and press the [OK] button 623.
In S516, information on the character area used to set the file name and metadata is sent to the MFP cooperation service 120.
The request control unit 431 of the MFP cooperation service 120, which received the learning request in S517, requests the image processing unit 432 to perform form learning.
In S518, the image processing unit 432 saves the information on the character area of the entire image and the input information on the character area used by the user for the file name and metadata, which was received in S516.
Further, if the user is modifying a character string extracted by OCR processing on the attribute setting screen, the request control unit 431 requests the image processing unit 432 to learn to modify the OCR processing result in S519.
In S520, the image processing unit 432 saves the correction information of the OCR processing result. Details of S520 will be described later.

Thereafter, in S521, the MFP 110 sends the scanned image and information such as a file name and metadata to be set after converting the scanned image into a file to the MFP cooperation service 120.
When the MFP cooperation service 120 receives the request, it starts file generation processing and returns to the MFP 110 that the request has been successfully received. When the MFP 110 receives the transmission response, it ends the process and returns to the workflow selection screen in S505.
In S522, the MFP cooperation service 120 obtains information on the file format to be sent to the cloud storage 130 from the scan settings of the workflow, which is a series of processes registered in the MFP cooperation service. Then, a file is generated from the scanned image based on the settings.
In S523, the file name received in S521 is set in the file generated in S522, and the file is sent to the cloud service 130 set as the destination in the workflow settings. The cloud service 130 replies to the MFP cooperation service 120 that the transmission has been received, and the sequence ends.

(MFP, client PC display screen)
FIGS. 6A and 6B are diagrams showing examples of screens displayed by the MFP 110 or the client PC 111.
FIG. 6A is a diagram showing an example of a scanned form list screen 600. On this screen, it is possible to view a list of forms that have undergone scanning and image analysis processing and have not yet been sent to the cloud service 130. This screen also includes a list of scanned forms 601, a send button 602, an edit button 603, and a delete button 604.
A list of scanned forms 601 displays a list of forms for which scanning and image analysis (S506 to S511) have been completed. The scanned form list 601 includes a form name 605, a destination 606, a status 607, and a workflow 608. The form name 605 is an identifier that uniquely identifies the name of the form. The destination 606 is the name of the cloud service 130 to which the form file is sent. Status 607 indicates the result of similar document determination for a document, and either "unlearned" or "learned" is displayed. "Unlearned" means that it has been determined that no similar forms exist in the used workflow, and "Learned" means that it has been determined that similar forms exist. In the workflow 608, the name of the workflow selected and executed in S504 is displayed. For example, if one of the workflows to be sent to cloud service A is created with the name "Workflow A1", "Workflow A1" is displayed. Furthermore, if the scan settings, file name, and metadata setting rules are different when sending to cloud service A, it can be created separately as "Workflow A2", and "Workflow A2" is displayed.
The send button 602 is a button for sending a form to the cloud service 130. Select any form from the scanned form list 601 (it may be grayed out by selecting it with a mouse, etc.) and press the send button 602 to send it to the cloud service 130 displayed in the destination 606. and send it. Note that if the transmission is successfully completed, the form will be deleted from the list.
The edit button 603 is a button for moving to an attribute setting screen 610 in FIG. 6B, which will be described later. By selecting an arbitrary form from the scanned form list 601 and pressing the edit button 603, the screen moves to the attribute setting screen 610 of the selected form.
The delete button 604 is a button for deleting a form. By selecting an arbitrary form from the scanned form list 601 and pressing the delete button 604, the selected form can be deleted.

FIG. 6B is a diagram showing an example of the attribute setting screen 610.
The file name setting area 611 is an area for setting a file name based on rules defined by the user when creating a workflow. Figure 6B shows that when creating a workflow, character strings in the form, such as "{company name}_{document number}" are used to select the string that corresponds to "{company name}" and "{document number}". This is an example in which the file name is determined by substituting "{company name}_{form number}".
Displayed in the file name setting area 611 are an item name 612 indicating information to be extracted from the form, a text field 613 for inputting information for the item, and a cutout image 614 of a character area to be described later. If it is determined in S510 that there is no similar document and the document is an unlearned document, the text area is not selected, the text field is empty, and the cutout image is displayed without being set. If it is determined in S510 that there is a similar document, a text field 613 and a cutout image 614 are set using the character area selected in the similar document.
The file name display section 615 displays file names based on the file name setting rules set by the user. Further, if the value of the text field 613 corresponding to the item name 612 has been input, the item name written in the file name display section 615 is replaced with the value of the text field 613.

The metadata setting area 616 is an area for setting metadata values to be sent to the cloud service. Similar to the file name setting area 611, the metadata setting area 616 displays an item name 612M, a text field 613M, and a cutout image 614M of the character area. The item name 612M in the metadata setting area 616 displays the metadata name set in the destination cloud service. Furthermore, if multiple pieces of metadata can be set in the destination cloud service, the item name 612M, text field 613M, and character area cutout image 614M may each be displayed as many times as the number of pieces of metadata. Further, if there is no metadata setting in the destination cloud service, the metadata setting area 616 may not be provided. The learned workflow list 617 displays a list of workflows that are learning similar forms that have the same shape information as the currently displayed form, and transfers learning information for each item name 612. The detailed learning information transfer process will be explained with reference to FIG. 9 and FIGS. 10A to 10E. Preview area 618 displays the scanned image. When you hover over a text area in an image, shapes such as lines and frames and colors are added to the text area. Further, the text field 613 displays the character string of the character area. Further, a cutout image of the character area is displayed in the cutout image 614. Each time the mouse-over character area changes, the text field 613 and cutout image 614 change.

The [Delete text area] button 619 deletes the characters corresponding to the set text area. [Enlarge preview] button 620 increases the enlargement magnification of the image displayed in the preview area. [Reduce preview] button 621 Reduces the magnification of the image displayed in the preview area. When enlarging and reducing, the display position is adjusted so that the coordinates of the center of the preview area are the same as before enlarging and reducing. The [Initial Preview Display] button 622 displays the initial magnification when you move the display position of the preview image by swiping or change the display magnification by pressing the [Enlarge Preview] button 620 or [Reduce Preview] button 621. Return to position. The [OK] button 623 transmits the scanned image together with the file name and metadata set on the attribute setting screen to the MFP cooperation service 120, and executes a learning process (S516 to S520) for modifying the form and OCR processing results. At this timing, learning processing for correcting OCR processing results, which will be described later, is also executed. When the transmission is completed, the screen returns to the scanned form list screen 600.

- Image analysis processing FIG. 7 is a flowchart showing details of the image analysis processing performed by the image processing unit 432 in this system. This flow corresponds to S508 to S511 in FIG.
First, in S701, character areas of an input image are analyzed to obtain a group of character areas within a form.
Subsequently, in S702, similar form determination is performed to estimate similar forms by comparing with learning data. These steps have been explained with reference to FIG. 5, so a detailed explanation will be omitted.
As a result of the determination, if a similar document is found in S703 (Yes in S703), the process proceeds to S704, and if no similar document is found (No in S703), the process proceeds to S708.
(If similar documents are found)
In S704, the character area of the target form corresponding to the character area registered in the found similar form is acquired.
Then, in S705, character recognition processing is performed on the corresponding character area obtained in S704, and a character string is extracted. Through this process, file names and metadata that are considered desired by the user for the target form can be presented based on the file name setting rules and metadata setting rules for similar forms.
Thereafter, in S706, the analysis results are added to the learned form storage area.
Finally, in S707, the request control unit 431 is notified as a learned form, and this flow ends.
(If no similar form is found)
In S708, character recognition processing is performed on the entire character area of the form, and character strings are extracted. The reason why the character recognition process is performed on the entire character area is because it is determined that no similar form exists and the area that is expected to be used for setting the file name and metadata is unknown.
Thereafter, in S709, the analysis results are added to the unlearned form storage area.
In S710, the request control unit 431 is notified as an unlearned form, and this flow ends.

(data structure)
FIG. 8 is a diagram showing an example of a data structure saved in the analysis result saving process performed by the image processing unit 432 in this system.
First, FIG. 8(a) is a diagram showing an example of the outline of the data structure of the analysis result. There are roughly three storage areas here, which are classified based on the results of the similar form determination process in S509. Specifically, a group of documents for which it is determined that no similar documents exist are saved in the same storage area, and when it is determined that similar documents exist, the most similar documents are saved in the same storage area. . Each workflow has a storage area as shown in Figure 8(a).

Next, FIG. 8(b) is a diagram showing an example of a detailed data structure of the analysis result. The analysis result includes character area information analyzed in S704 of FIG. 7 and character string information extracted in S705 or S708. The root "formList" represents a list of forms, and the analysis results regarding a plurality of forms are stored as an array in units as explained in FIG. 8(a). Each form has “formID”, “imageWidth”, “imageHeight”, and “regions”.
“formID” is an identifier that is unique in this system and is given to a form.
“imageWidth” indicates the number of pixels in the X direction (horizontal direction) of the analyzed image.
“imageHeight” indicates the number of pixels in the Y direction (vertical direction) of the analyzed image.
“regions” includes the arrangement information of character regions analyzed from the analyzed image and the arrangement of character information.
Next, the information in "regions" will be explained.
“rect” indicates the coordinates of one extracted character area. "x" indicates the X coordinate of the upper left corner of the region, "y" indicates the Y coordinate of the upper left corner of the region, "width" indicates the number of pixels in the X direction of the region, and "height" indicates the number of pixels in the Y direction of the region.
“text” indicates a character string extracted as a result of character recognition performed by OCR on the character area of “rect”. This "rect" and "text" information is included for the entire character area in the analyzed scan image.

- Handover process of learning data FIG. 9 is a flowchart showing details of the process of handing over the learning data of the character area to be extracted, which has been learned in another workflow, to the workflow being executed in S516 to S520 in this system. The flow of learning data handover processing will be explained using examples of attribute setting screens shown in FIGS. 10A to 10E. Note that this processing flow is realized by the CPU 311 of the MFP cooperation server 120 loading a program stored in the ROM 312 into the RAM 313 and executing it.
FIG. 10A is an example of an attribute setting screen when workflow A1 to be sent to cloud service A is executed, and similar forms of scanned images have already been learned. Workflow A1 has the item names "Company Name" and "Form Number" to set the file name, and cloud service A has the metadata Name, so it has "Name" as the item name.
FIG. 10B is an example of an attribute setting screen when executing workflow B1 for sending to cloud service B. The same image as the document scanned in FIG. 10A is scanned, and similar documents have not been learned. Workflow B1 has the item name "Company Name" to set the file name, and cloud service B has the item name "Title" because it has metadata Title. This embodiment will be described using an example in which the learning data of the character areas of "Company Name" and "Name" learned in Workflow A1 is inherited to "Company Name" and "Title" of Workflow B1. Note that FIGS. 10C, 10D, and 10E are examples of screen transitions when learning data is transferred on the attribute setting screen.
In S901, the CPU 311 acquires a workflow that is learning a form similar to the scanned image displayed on the current preview screen 1001 in FIG. 10C.
In S902, the CPU 311 displays the list of acquired workflows in the pull-down list 1002 of the learned workflow list, allowing selection of the workflow to which the learning data is to be inherited. In addition, the options in the pull-down list 1002 include an item that does not take over from the workflow, considering the case where the workflow is not taken over. If the workflow is selected in S902 (Yes in S902), the process advances to S903, and if the option of not taking over is selected (No in S902), the process advances to S906.

FIG. 10D is an example when workflow A1 is selected in S902.
In S903, the CPU 311 adds shapes and colors such as lines and frames to the preview screen for character areas extracted by each item name of the workflow selected in S902. This allows the character area to be visually confirmed by highlighting it as shown in the learned character area 1003 in FIG. 10D. Further, a learned item list 1004, which is a list of item names of the workflow selected in S902, is displayed for each setting item of the workflow. By mousing over the inherited item name 1005 in the learned item list 1004, the inherited item name 1005 and the character area 1006 corresponding to that item are displayed with the same highlighted display different from the learned character area 1003. . This makes it possible to clearly see the correspondence between the inherited item name 1005 and the character area 1006.
In S904, the CPU 311 accepts that the displayed inheritance item 1005 has been selected for association by a click or touch operation. If the selection for associating the handover item 1005 is accepted (Yes in S904), the process advances to S905, and if the selection of the handover item 1005 is not accepted (No in S904), the process advances to S906.
In S905, the CPU 311 selects the corresponding character area 1006 from the preview screen, and displays the text field 613 and the cutout image 614 of the character area associated with the inheritance item 1005, as described in FIG. 6B. indicate.
In S906, the CPU 311 selects a character area from the preview screen using the method described in FIG. 6B, and sets a text field 613 and a cutout image 614 of the character area in association with the item name.

FIG. 10E is an example when the same text area as "Company Name" of Workflow A1 is selected in the "Company Name" item and the same text area as "Name" of Workflow A1 is selected in the "Title" item in S904 and S905. It is.
In S907, the CPU 311 performs learning when the [OK] button 1007 in FIG. 10E is pressed.
In S908, the CPU 311 determines whether the character area selected for each item name is the same character area as the learned workflow selected in S902. If the workflow being executed uses the same character area as the learned workflow (Yes in S908), the process advances to S909. If not (No in S908), the process advances to S910 and learns the character area to be extracted from the form.
In S909, the CPU 311 is executing all types of form shape information (character area arrangement information, etc.) for determining similar forms from the training data of the learned workflow, and character string information of the character area to be extracted. Replicate the information that workflow items extract. As a result, if the learned workflow is similar to a previously learned form, the same character area can be extracted even if the currently executed workflow has not been learned. In the example of FIG. 10E, the learning data for the item name "Company Name" of Workflow A1 is duplicated for the item name "Company Name" of Workflow B1. Then, even if the workflow B1 has never scanned a similar document, if the workflow A1 has already been trained, the character area will be automatically selected. Similarly, the learning data for the item name "Title" of the workflow B1 "Name" of the workflow A1 is duplicated. Therefore, if you want to obtain the same information from a form even if the metadata name set for each cloud service is different, by performing the transfer process, learning data other than similar forms of the scanned form will also be inherited, so you can create a new form. There is no need to re-learn each shape. As a result, even if, for example, 50 types of forms are transferred from a learned workflow to another workflow, the 50 types of forms will be in a learned state by performing the transfer process once.

(Other examples)
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

110 MFP
120 MFP cooperation service
130 Cloud Services

Claims (7)

an extraction means for extracting character area information from the scanned image data of the form;
a determination unit that determines similar forms by comparing the information with learning data based on the information of the extracted character area;
recognition means for recognizing a character string in a character area corresponding to the similar form;
a learning means for learning the character area;
An information processing device comprising a storage means for storing learning data of the character area for each series of processing,
a first display means for displaying a list of a series of processes that have been learned for similar forms of scanned images when learning character areas to be obtained in the first series of processes;
When the user selects a learned second series of processes from the list of learned processes, a list of learned character regions to be extracted and information items of the second series of processes is displayed. a second display means for displaying;
a reception unit that accepts an association between an item of information to be extracted in the learned second series of processes and an item of information to be transmitted in the first series of processes to perform learning;
An information processing apparatus comprising: duplication means for duplicating the learning data of the second series of processes into the learning data of the first series of processes based on the association. 2. The information processing apparatus according to claim 1, wherein the first display means displays a series of learned processes for similar forms in a pull-down list. 3. The information processing apparatus according to claim 2, wherein the information on the character area is arrangement information on the character area and information on a character string. 4. The information processing apparatus according to claim 3, wherein the extracted information is transmitted together with the scanned image to an external service connected via a network. The information processing apparatus according to claim 4, further comprising a scanner section. an extraction step of extracting character area information from the scanned image data of the form;
a determination step of comparing the extracted character area information with learning data to determine similar forms;
a recognition step of recognizing a character string in a character area corresponding to the similar form;
a learning step of learning the character area;
A method for controlling an information processing device, comprising: storing learning data of the character area in a storage means for each series of processing,
a first display step of displaying a list of a series of processes that have been learned for similar forms of the scanned image when learning character areas to be obtained in the first series of processes;
When the user selects a learned second series of processes from the list of learned processes, a list of learned character regions to be extracted and information items of the second series of processes is displayed. a second display step of displaying;
a reception step of accepting an association between an item of information to be extracted in the learned second series of processes and an item of information to be transmitted in the first series of processes for learning;
A method for controlling an information processing apparatus, comprising: a duplication step of duplicating learning data for a second series of processes into learning data for a first series of processes based on the association. A program for causing a computer to execute the method for controlling an information processing apparatus according to claim 6.

Images