JP6850323B2 - Information processing device, its control method and program - Google Patents

Description

The present invention relates to a mobile terminal, an information processing system, and a processing method and program thereof that perform entry checking using captured images.

Conventionally, there is a mechanism in which form data is transmitted from a user terminal to a form server, the form server checks whether there are any deficiencies in the entered contents, and the check result is returned to the user terminal.

Patent Document 1 describes an electronic processing system in which the form server performs all processing such as determining the content of form data transmitted from a user terminal, correcting an error by an operator's operation, and creating a processing result with a message of the correction result. Technology is described.

Japanese Unexamined Patent Publication No. 6-28275

However, in the system configuration of Patent Document 1, in order to check the entry deficiency, the form data must be sent to the form server, and the entry deficiency check cannot be performed in an environment where communication with the form server is difficult. In addition, even a minor defect check such as omission of entry causes communication between the user terminal and the form server, and there is a problem that it takes time to complete a complete form.

On the other hand, when character recognition is performed on an image taken by a user terminal and an entry deficiency check is performed, the processing time for the entry deficiency check becomes longer on a user terminal whose specifications are not higher than those of a server or a computer that is normally used. Technical challenges arise. In addition, although it is a simple entry deficiency check, it is necessary to wait for the entry deficiency check in the system of the user terminal, which may reduce the work efficiency. Since the user wants to check the image immediately after shooting, there is a problem that it is not convenient to wait for even a few seconds.

Therefore, an object of the present invention is to provide a mechanism for streamlining the entry check on a form.

A information processing device for achieving the object of the present invention, based on the result of the process according to the character recognition for the image data, and checking means for checking documents according to the image data, check processing by the confirmation unit a verification status information acquisition unit that acquires the confirmation status information indicating whether has been performed, the supplementary image information acquisition means for acquiring supplementary image information relating to image data to be appended to the image data, and the previous Kiga image data and output means for outputting the combined and acquired supplementary image information by the verification status information acquired confirmed status information by acquisition means the supplementary image information acquiring means, wherein the Ru with the.

According to the present invention, there is an effect that the entry check on the form can be made more efficient.

The figure which shows an example of the structure of the form system 100 of this invention The figure which shows an example of the hardware composition of the information processing apparatus 101 Functional block diagram showing an example of the information processing device 101 and the server 102 A flowchart showing an example of the entire process of acquiring the check result of incompleteness in the information processing device 101 and transmitting it to the server 102. Flow chart showing an example of entry deficiency check processing performed by the information processing device 101 A flowchart showing an example of processing for receiving a photographed image of an application form, an incidental image, an entry deficiency check flag, a login user name, and a form name on the server 102. A flowchart showing an example of a process for displaying a list of application forms received by the server 102. A flowchart showing an example of a process of adding an image incidental to a captured image of an application form in the information processing device 101. The figure which shows an example of the deficiency confirmation cover sheet which covers the application form in order to visually confirm the deficiency check of the application form with the information processing apparatus 101 in an easy-to-understand manner. The figure which shows an example of the screen which displays the application form photographed image by covering the deficiency confirmation cover sheet in order to make it easy to visually check the entry deficiency in the information processing apparatus 101. The figure which shows an example of the screen which displays the entry defect check result in the information processing apparatus 101. The figure which shows an example of the table which manages a form and a deficiency confirmation cover sheet in a server 102 and an information processing apparatus 101. A diagram showing an example of a table that manages the transmission time of the application form received by the server 102, the sender, the form ID, and the entry deficiency check flag. The figure which shows an example of the screen which displays the table managed in FIG. 14 on a server 102. The figure which shows an example of the screen which displays the application form received by the server 102 Figure showing an example of a table that manages the entry deficiency check recognition area and deficiency check method for each form The figure which shows an example of the screen which confirms and displays the image of the application form transmitted by the information processing apparatus 101. The figure which shows an example of the screen which confirms and displays the incidental image of the application form transmitted by the information processing apparatus 101. The figure which shows an example of the screen which performs the information setting preset in the information processing apparatus 101. The figure which shows an example of the hardware configuration of a server 102

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

As shown in FIG. 1, in the present embodiment, the form system 100 (information processing system) has a configuration in which an information processing device 101 (mobile terminal) and a server 102 are communicably connected by a network 103. The information processing device 101 captures an image of the form using a camera or the like, and checks for deficiencies in the form entry. The server 102 manages the recognition area and the recognition condition of the entry contents necessary for checking the entry deficiency of the form for each form identification information. The information processing device 101 is a portable terminal such as a mobile phone, a smartphone, a tablet terminal, and a PDA terminal, and an application of a browser or a form system is installed in the information processing device 101. Communication is possible. Needless to say, the server 102 may be a general-purpose computer.

Hereinafter, an example of a hardware configuration applicable to the information processing apparatus 101 shown in FIG. 1 will be described with reference to FIG.

In FIG. 2, 201 is a CPU that comprehensively controls each device and controller connected to the system bus 204. Further, the ROM 203 or the external memory 212 is required to realize a function executed by a BIOS (Basic Input / Output System) or an operating system program (hereinafter, OS) which is a control program of the CPU 201, and each server or each PC. Various programs etc. are stored.

Reference numeral 202 denotes a RAM, which functions as a main memory of the CPU 201, a work area, and the like. The CPU 201 realizes various operations by loading a program or the like necessary for executing the process from the ROM 203 or the external memory 212 into the RAM 202 and executing the loaded program.

Further, 205 is an input controller, which controls the input from the touch sensor 210. Reference numeral 206 denotes a video controller, which controls the display on a display such as the display 211. Although the display 211 is described in FIG. 2, the display may be not only a CRT but also another display such as a liquid crystal display. These are used by the user of the information processing apparatus 101 as needed.

The 207 is a memory controller, which can be used in an external storage device (hard disk (HD)) for storing boot programs, various applications, font data, user files, edit files, various data, etc., a flexible disk (FD), or a PCMCIA card slot. Controls access to external memory 212 such as compact flash® memory connected via an adapter.

Reference numeral 208 denotes a communication I / F controller, which connects and communicates with an external device via the network 103, and executes communication control processing on the network. For example, communication using TCP / IP is possible.

The CPU 201 enables display on the display 211 by, for example, executing an outline font expansion (rasterization) process in the display information area in the RAM 202. Further, the CPU 201 makes it possible to receive a user's touch operation instruction on the touch sensor 210 on the display 211.

Various programs described later for realizing the present invention are recorded in the external memory 212, and are executed by the CPU 201 by being loaded into the RAM 202 as needed. Further, definition files, map data, and the like used when executing the above program are also stored in the external memory 212, and detailed description of these will be described later.

Reference numeral 209 denotes a captured image input controller, which controls input from the camera 215 via a captured image input processing unit 214 (corresponding to a photographing means). It is possible to recognize the captured image input from the camera 215.

Next, the hardware configuration of the information processing apparatus applicable to the server 102 shown in FIG. 1 will be described with reference to FIG.

In FIG. 20, 2001 is a CPU that comprehensively controls each device and controller connected to the system bus 2004. Further, the ROM 2002 or the external memory 2011 is necessary to realize the BIOS (Basic Input / Output System) and the operating system program (hereinafter, OS), which are the control programs of the CPU 2001, and the functions executed by each server or each PC. Various programs described later are stored.

The 2003 is a RAM, which functions as a main memory, a work area, and the like of the CPU 2001. The CPU 2001 realizes various operations by loading a program or the like necessary for executing a process from the ROM 2002 or the external memory 2011 into the RAM 2003 and executing the loaded program.

Further, 2005 is an input controller, which controls input from a keyboard (KB) 2009, a pointing device such as a mouse (not shown), or the like. 2006 is a video controller that controls the display on a display such as a CRT display (CRT) 2010. Although it is described as CRT2010 in FIG. 20, the display may be not only the CRT but also another display such as a liquid crystal display. These are used by the administrator as needed.

2007 is a memory controller that connects to a hard disk (HD), flexible disk (FD), or PCMCIA card slot that stores boot programs, various applications, font data, user files, edit files, various data, etc. via an adapter. Controls access to external memory 2011 such as compact flash (registered trademark) memory and database.

Reference numeral 2008 denotes a communication I / F controller, which connects and communicates with an external device via a network (for example, 103 shown in FIG. 1), and executes communication control processing on the network. For example, communication using TCP / IP is possible.

The CPU 2001 enables display on the CRT 2010 by, for example, executing an outline font expansion (rasterization) process in the display information area in the RAM 2003. In addition, the CPU 2001 enables user instructions on the CRT 2010 with a mouse cursor (not shown) or the like.

Various programs described later for realizing the present invention are recorded in the external memory 2011, and are executed by the CPU 2001 by being loaded into the RAM 2003 as needed. Further, a definition file and various information tables used when executing the above program are also stored in the external memory 2011.

FIG. 3 is a block diagram showing an example of the functional configuration of the information processing device 101 and the server 102. The information processing device 101 will be described as a mobile terminal including a photographing means.

The image acquisition unit 301 is a functional unit that acquires a second image that is transparent so that the entry check portion corresponding to the first image obtained by photographing by the photographing means can be visually recognized.

The image generation unit 302 is a functional unit that generates a third image for entry check by using the first image and the second image.

The display unit 303 is a functional unit that displays a third image generated by the image generation unit 302 so that the user can enter and check the entry. Further, it is a functional unit that displays the first image and displays the number of incidental images of the first image obtained by photographing by the photographing means together with the first image. Further, when the recognition result is acquired by the recognition result acquisition unit 307 described later, it is a functional unit that displays information indicating that the recognition result has been obtained together with the third image. Further, it is a functional unit that switches from the third image to the first image and the recognition result image generated according to the recognition result and displays the recognition result when there is an instruction to display the recognition result.

The output unit 304 is a functional unit that outputs the first image as an entry check completed when an output instruction is given after the display unit 303 displays the third image. Further, it is a functional unit that outputs the first image and the incidental image described later together. It is also a functional unit that outputs a first image including header information.

The incidental image storage unit 305 is a functional unit that stores the incidental image obtained by photographing by the photographing means when the user gives an instruction to photograph the incidental image.

The definition information storage unit 306 is a functional unit that stores definition information including a recognition area and recognition conditions for recognizing the entered contents from the first image.

The recognition result acquisition unit 307 is a functional unit that acquires a recognition result based on the first image and the definition information.

The recognition processing execution unit 308 is a functional unit that executes recognition processing based on the first image and the definition information in the background of displaying the third image.
The information acquisition unit 309 is a functional unit that acquires header information including checker information related to the entry check.

The output image generation unit 310 is a functional unit that generates a first image including header information.

The receiving unit 311 is a functional unit that receives the first image.
The output image registration unit 312 is a functional unit that registers the first image in a hard disk, database, or the like.
The list display unit 313 is a functional unit that displays a list of cases corresponding to the first image.

FIG. 4 is a flowchart showing an example of an overall process (hereinafter referred to as an entry deficiency check result acquisition / transmission process) in which the information processing apparatus 101 acquires the entry deficiency check result and transmits it to the server 102.

Further, FIG. 19 is an example of a screen for setting information preset by the information processing apparatus 101. On the initial setting screen 1901, when the store name, the person in charge, and the telephone number are input and the OK button is pressed, the information is saved in the information processing device 101. If the application form to be photographed is decided in advance as in the initial setting screen 1902, the form definition information for character recognition and the form ID for specifying the cover sheet are input. You may. This initial setting is set by the entry deficiency check application.

In step S401, the information processing device 101 starts the entry deficiency check application and logs in. By pressing the initial setting button (not shown) when the entry deficiency check application is started, the above-mentioned initial setting screen is displayed and the initial setting is appropriately executed by the user's operation.

When the press of the application form shooting button (not shown) is detected in step S402, the camera 215 is activated to shoot the application form for checking the entry deficiency. Here, the four corners of the form such as the application form are detected, and when the focus is on, the shooting is executed. By this execution, the photographed image corresponding to the application form is acquired.

In step S403, it is determined whether or not there is a form that matches the captured image. Whether or not there is a matching form is determined by comparing the photographed image and the form image in the order of data in the local form table 1200 (FIG. 12) using a form recognition technique or the like, and identifying the matching form. As described above, when the form ID is set in advance, it may be determined whether or not the form ID is in the form table. Alternatively, the form may be specified by extracting the form ID from a predetermined area of the captured image. If it is determined in step S403 that there is no matching form, the process is transferred to step S404, and if it is determined that there is a matching form, the process is transferred to step S406.

If there is no matching form table in step S404, an error message indicating that the application form taken in step S405 is not a deficiency check compatible form is displayed and the application is terminated. If the form is not compatible, the process may shift to re-shooting after displaying an error message. If there is a matching form table, the cover sheet (image) corresponding to the photographed application form is acquired from the external memory 212 with reference to the form and the form table (FIG. 12) in step S406. The form table 1200 and the corresponding images (form image, cover sheet) are stored in the external memory 212. That is, it shows an example of the process of acquiring the second image which corresponds to the first image obtained by photographing and is transmitted so that the entry check part which is a cover sheet can be visually recognized.

In step S407, the photographed image of the photographed application form and the cover sheet (FIG. 9) for confirming deficiencies corresponding to the photographed application form acquired in step S404 are overlaid to generate a check image. That is, it shows an example of the process of generating the third image for entry check using the first image and the second image.
This generated image is displayed on the entry deficiency checking screen (Fig. 10-1). That is, it shows an example of the display process of displaying the generated third image so that the user can fill in and check it.
This cover sheet may be a cover sheet that includes a description of the checked portion so that the user can easily check.
In the cover sheet of FIG. 9, the check points are transparently processed, and when the check points are superimposed on the captured image, it is possible to easily confirm the entry of the places to be checked for deficiencies. It is possible to make the part other than the check part transparent, but in order to make the check part easier for the user to see, the transmittance is made lower than the check part or the color is changed (for example, set to gray). It is desirable to make an image that looks like this.

In step S408, in order to initialize the deficiency check process, the entry deficiency check completion flag is turned off, and the OK button flag and the cancel button flag are turned off.

In step S409, an entry deficiency check processing flow (described later in FIG. 5) is performed. This entry deficiency check process is executed in the background displaying the entry deficiency check in progress screen shown in FIG. 10-1. That is, an example of executing the recognition process based on the first image and the definition information is shown in the background of the display of the third image.

In step S410, the entry deficiency check flag, the OK button flag, and the cancel button flag, which are the recognition results, are acquired. It is confirmed in the entry deficiency check processing flow (FIG. 5) whether the OK button flag is ON, and if it is ON, the process proceeds to the incidental image addition processing flow (described later in FIG. 8) in step S422. The case where the OK button flag is ON is a case where the deficiency check is completed by visually checking the deficiency using the overlaid image on the entry deficiency checking screen shown in FIG. 10-1. .. When the process of step S422 is completed, the information processing device 101 transmits the photographed image of the application form, the incidental image, the form name, the login user name, and the defect check completion flag OFF to the server 102 in step S423 to end the process.

In step S410, it is confirmed whether the OK button flag is ON in the entry deficiency check processing flow (FIG. 5), and if it is OFF, it is confirmed whether the cancel button flag is ON in step S411. If the cancel button is ON, the app will be closed.

If the cancel button flag is OFF in step S411, the entry deficiency check completion screen (FIG. 10) is displayed in step S412 to the effect that the entry deficiency check is completed and the confirmation button is displayed (recognition result information is displayed). -Change to -2). When the cancel button flag is OFF, the entry deficiency check completion flag is ON. In other words, it is the case where the entry deficiency check is completed by the recognition by the system instead of the visual inspection in the background.
In other words, information indicating that the recognized result is obtained together with the third image when the recognized result is acquired based on the first image and the definition information and the recognized result is acquired (for example,). This is an example of the process of displaying the confirmation button).

In step S413, it is determined which of the OK button, the cancel button, and the confirmation button is pressed on the entry deficiency check completion screen (FIG. 10-2). Exit the app when the cancel button is pressed. When the OK button is pressed, the process transitions to the incidental image addition processing flow (described later in FIG. 8) in step S422. When the process of step S422 is completed, the information processing device 101 transmits the photographed image of the application form, the incidental image, the form name, the login user name, and the deficiency check completion flag ON to the server 102 in step S424 to end the process. If there is a defect, the confirmation button is pressed to confirm it, and then the transmission is controlled. When the confirmation button is pressed, the process proceeds to step S414.

In step S414, the information processing apparatus 101 searches the entry deficiency check table 1610 using the form ID of the captured image identified in step S403 as a search key, and puts the RAM 202 in the result column 1611 of the check ID = i of all the acquired records. The value of the variable result [i] stored in is registered in each. In the present embodiment, the variable result [i] = [OK, NG, OK, NG, OK] indicating the check result of the form ID = 1000 in the result column 1611 of 5 records of the form ID = 1000 of the entry deficiency check table 1610. The value of is registered.

In step S415, the information processing apparatus 101 closes the entry deficiency check completion screen (FIG. 10-2) from the display 211, and displays the captured image display on the entry deficiency check result screen (FIG. 11). That is, when there is an instruction to display the recognized result by pressing the confirmation button, the process of switching from the third image to the recognition result image generated according to the recognized result as the first image and displaying it. An example is shown.

In step S416, the information processing apparatus 101 determines whether or not NG exists in the variable result [i].

If NG does not exist, in step S417, the information processing apparatus 101 displays the entry deficiency check result screen (no deficiency) 1110 with the message "There was no entry deficiency."

If NG exists, in step S418, the information processing apparatus 101 displays the entry deficiency check result screen (with deficiency) 1120 in which the message "There is an entry deficiency." Is written.

In step S419, the information processing apparatus 101 searches the entry deficiency check table 1610 using the form ID = the form ID of the captured image and the result column 1611 = NG as search keys, and the starting point coordinates (X, Y) of the entry deficiency check recognition area. ) And end point coordinates (X, Y).

In step S420, the information processing apparatus 101 draws a rectangle having the starting point coordinates (X, Y) and the ending point coordinates (X, Y) of the acquired entry deficiency check recognition area diagonally on the captured image, and fills it with red ( 1121). At this time, the rectangle is set to be transparent. As a result, the deficient area can be displayed in an identifiable manner.

In step S421, it is determined which button is pressed on the entry deficiency check result screen. If the cancel button is pressed, the application is terminated, and if the OK button is pressed, the incidental image addition process (described later in FIG. 8) of step S422 is performed. After step S422, in step S424, the information processing device 101 transmits a photographed image of the application form, an incidental image, a form name, a login user name, and a deficiency check completion flag ON to the server 102, and ends the process. That is, it shows an example of the process of outputting the information related to the entry check and the first image when the output instruction is given after the display of the third image. Further, when there is an incidental image, an example of a process of outputting the first image and the incidental image together is shown. Needless to say, if there is no incidental image, the incidental image is not transmitted.
The transmission may be performed for each application form, or if communication with the server 102 is not possible, the data is stored in the information processing device 101, and when communication is possible, data related to a plurality of application forms is transmitted. May be configured to transmit.

FIG. 5 is a flowchart showing an example of the entry deficiency check process performed by the information processing apparatus 101.

Further, FIG. 16 is an example of a table that manages an entry deficiency check recognition area and a deficiency check method for each form.

The form ID, the check ID attached to the check location, the recognition condition of the entry deficiency check, the start point coordinates (X, Y) and the end point coordinates (X, Y) of the entry deficiency check recognition area are stored in the entry deficiency check table 1610. There is. That is, it is an example of storing definition information including a recognition area and recognition conditions for recognizing the entered contents from the first image.

In addition, although the result column 1611 for storing the entry deficiency check processing result exists in the "entry deficiency check table 1610", the result column 1611 is not used yet in the entry deficiency check process of FIG. It is null.

In step S501, the entry deficiency check table 1610 is searched using the form ID of the captured image specified in step S404 as a search key, and the entry deficiency check record is acquired.

In step S502, 1 is assigned to the variable i and stored in the RAM 202.

In step S503, the start point (X coordinate, Y coordinate) and end point (X coordinate, Y coordinate) of the entry deficiency check recognition area and the entry deficiency check recognition condition are acquired from the i-th entry deficiency check record.

In step S504, the value of the entry deficiency check recognition condition is determined.

If the entry deficiency check recognition condition is "essential", it is an item that must be entered. Therefore, in step S505, it is determined whether or not characters or symbols are entered in the check recognition area of the captured image. Although this determination process uses an analysis technique such as OCR, the description thereof will be omitted because it is a known technique.

When a character or a symbol is entered, in step S506, “OK” is assigned to the variable result [i] that stores the check result in the RAM 202.

If no character or symbol is entered, "NG" is assigned to the variable result [i] in step S507.

When the entry deficiency check recognition condition is "circled", one of the plurality of options is surrounded by a circle. Therefore, in step S508, whether or not the check recognition area of the captured image is marked with a circle. Is determined. This determination process uses an analysis technique such as OCR, but since it is a known technique, the description thereof will be omitted. In this embodiment, the presence or absence of the circle mark is determined, but the determination is not limited to the circle mark, and the check mark or other figure may be determined.

In addition, in the above, the number of circles is set to "1", but if the entry deficiency check table 1610 is provided with "upper limit value" and "number of checks" columns and exceeds the upper limit value, "NG", the number of marks If is other than the "number of checks", it may be determined as "NG". In addition, even if the entry deficiency check table 1610 is provided with a "number of digits" column, it is judged as "OK" if the entered characters are the permitted number of digits, and "NG" if the entered characters are not permitted. Good. In addition, a "character attribute" column may be provided in the entry deficiency check table 1610, and OK / NG may be determined based on the entered character attributes such as "number", "hiragana", "katakana", and "alphanumeric characters". ..

Further, when the entered character cannot be recognized in the analysis result by OCR or the like, it may be determined as "NG". Further, the value of the variable result [i] is not limited to "OK" and "NG", and may have values such as "unclear", "unrecognizable", and "contradiction".

If a circle is entered, "OK" is assigned to the variable result [i] in step S509.

If the circle is not entered, "NG" is assigned to the variable result [i] in step S510.

In step S511, the variable i is incremented in order to check for deficiencies in the next area. In step S512, it is determined whether or not there is an entry deficiency check record of check ID = i. If it does not exist, the entry deficiency check completion flag is turned ON in step S517, and this entry deficiency check process ends. If it exists, it is determined in step S513 whether or not the OK button has been pressed on the entry deficiency checking screen (FIG. 10-1), and if the OK button has been pressed, the OK button flag is turned ON in step S515. Return to (1) in FIG. If the OK button is not pressed, the process proceeds to step S514. Since the OK button and the cancel button are pressed by the user at an arbitrary timing, the determination of step S513 and step S514 is performed at the pressed timing.

In step S514, it is determined whether or not the cancel button has been pressed on the entry deficiency check screen (FIG. 10-1), and if the cancel button has been pressed, the cancel button flag is turned ON in step S516, and (FIG. 4). Return to 1). If the cancel button has not been pressed, the process returns to step S503 and continues to check for deficiencies. The case where the cancel button is pressed is the case where the entry deficiency check is completed.

This is the end of the explanation of the entry deficiency check process shown in FIG. By this process, the check result for the entry deficiency check record corresponding to the form ID of the captured image is stored in the variable result [i]. Specifically, in this embodiment, the value of [OK, NG, OK, NG, OK] is stored in the variable result [i].

FIG. 6 is an example of a flow in which the server 102 receives the image and the attribute information transmitted from the information processing apparatus 101 in step S424.

In step S601, the server 102 receives the image of the application form, the incidental image, the entry deficiency check completion flag (OFF, ON), the login user name, and the form name.

In step S602, the received image (image of the application form for which entry deficiency has been checked) and ancillary images are stored in a predetermined folder, and the reception date and time, the form name, the login user name, and the reception date and time, the form name, and the login user name are stored in the reception application list table 1300 (FIG. 13). Add a record of entry deficiency check completion flag (OFF / ON). As a result, the information checked for deficiencies in the information processing device 101 is registered. After that, the operator on the server 102 side performs a detailed check, and if there is any deficiency, the person in charge corresponding to the login user name is called. The telephone uses, for example, the telephone number entered in the header of the received image.

FIG. 7 is an example of a flow in which the server 102 displays the image and the attribute information received from the information processing device 101, and the operator confirms whether or not there is any omission.

In step S701, the reception application list table (FIG. 13) is read and the reception application list screen (FIG. 14) is displayed. At this time, if the server 102 receives more than a predetermined number of application forms within a predetermined time (for example, if 20 or more application form images are received within 5 minutes), the application form system is inadequate. It is decided that the application forms that are not checked are displayed with priority, and the reception application form list screen 1420 is displayed. Further, when the number of application forms of a predetermined number or less is received within the predetermined time, the reception application list screen 1410 is displayed. In the mode in which the application form without the deficiency check is preferentially displayed, the server 102 generates a list sorted by "incomplete" for the deficiency check and displays it on the reception application list screen.

In step S702, it is determined whether or not the end button has been clicked on the reception application list screen (FIG. 14), and if the end button is pressed, the confirmation process of the application form ends. If the end button is not clicked, it is determined in step S703 whether the form name is clicked.

It is determined in step S703 whether the form name has been clicked, and if it has not been clicked, the process returns to step S702. If the form name is clicked, the application form received in step S704 is displayed on the reception application form confirmation screen (FIG. 15), and the operator on the server side visually checks for deficiencies.
At this time, if the application form for which the deficiency check has been completed is clicked on the reception application list screen (Fig. 14), the reception application form confirmation screen (filled in deficiency check completed) 1510 is displayed, and the application form for which the deficiency check has not been completed is clicked. If so, the reception application confirmation screen (incomplete entry check completed) 1511 is displayed. Therefore, the operator on the server side can easily confirm whether or not the application form has already been checked for deficiencies in the system.

In step S705, the server-side operator determines whether or not there is a deficiency in the application form.
If there are any deficiencies, notify the sender (logged-in user) described in the header of the image of the application form of the deficiencies, inform them that the application form will be photographed and resent after filling in, and proceed to step 708. To do. If there are no deficiencies, the sender (logged-in user) described in the header of the application form is notified that there are no deficiencies, and the process proceeds to step 708.
Since the telephone number is described in the header as an example, it is assumed that the sender is contacted by telephone in this embodiment, but the header is different from the sender such as the e-mail address and the URL of the system. You may also describe the means of contact. In addition, when the deficiency contact button (not shown) is pressed on the reception application confirmation screen of FIG. 15, a window (screen including a text box) for entering the deficiency content is displayed, and when the deficiency content is input, the sender (not shown) It is also possible to adopt a configuration in which transmission is performed to the information processing device 101 of the person in charge).

When the completion button is pressed on the reception application confirmation screen (FIG. 15) in step S708 to close the screen, the record of the application form displayed in step 704 is deleted from the reception application list table (FIG. 13) in step S709. It should be noted that the completion flag may be set and managed as confirmed without deleting.

FIG. 8 is an example of a processing flow in which the information processing apparatus 101 adds an incidental image incidental to the captured image of the application form.

In step S801, the transmission image confirmation screen (FIG. 17) is displayed, and the incidental image number and the number of incidental images are set to 0.

In step S802, the store name, the person in charge, the person in charge telephone number, the number of incidental images, the entry deficiency check completion status based on the result column of the entry deficiency check table 1610, etc. set on the initial setting screen (FIG. 19) (these are headers). Get information). Based on this acquired information, it is added as a header image to the captured image of the application form and displayed. Specifically, based on the acquired store name, person in charge name, person in charge telephone number, number of incidental images, and completion status of entry deficiency check, an image is created in a predetermined size and inserted at the top of the image of the application form. That is, it shows an example of the process of displaying the number of incidental images of the first image obtained by photographing together with the first image. Further, it shows an example of the process of acquiring the header information including the checker information related to the entry check and generating the first image including the header information.
An image including this header information is transmitted to the server 102 as an image that has been checked for deficiencies.
The information to be added to the header is not limited to the above information, as any item can be added. Further, although the telephone number is described as an example of the means for the operator of the server 102 to contact the sender, another means for contacting the sender such as an e-mail address or the URL of the system should be described in the header. It doesn't matter.

In step S803, it is determined which button is pressed on the transmission image confirmation screen (FIG. 17). If the cancel button is pressed, the process returns to (2) of FIG. 4 and the application form is rephotographed in step S402. The send button is pressed, and if there is an incidental image, the incidental image number / number of incidental images information is added to the header of each incidental image, and this processing flow ends.
If the incidental document image shooting button is pressed, an incidental image such as a driver's license required for the application form is photographed in step S804. The captured incidental image is stored in the memory. That is, it shows an example of the process of storing the incidental image obtained by shooting when the user gives an instruction to shoot the incidental image.

In step S805, the incidental image taken in step S804 is displayed on the incidental image confirmation screen (FIG. 18), and the number of incidental images is incremented by 1.

In step S806, the number of incidental images is added to the incidental image number of the captured incidental image.

In step S807, it is determined which button is pressed on the incidental image confirmation screen (FIG. 18). When the OK button is pressed, the screen returns to the transmitted image confirmation screen (FIG. 17).
If the cancel button is pressed, the number of incidental images is reduced by -1 in step S808, the incidental image number of the captured incidental image is cleared, and the screen returns to the transmitted image confirmation screen (FIG. 17).
If the re-shooting button is pressed, the number of incidental images is reduced by -1 in step S809, and the process returns to step S804 in order to clear the incidental image number of the captured incidental image and re-shoot the incidental image.
The above is an example of the processing flow of capturing and adding an image attached to the captured image of the application form.

As described above, according to the present embodiment, it is possible to improve the efficiency of entry check using an image taken by a mobile terminal. In addition, the entry check can be easily performed by the time when the entry check process by image recognition is completed, and the convenience can be improved. By enabling a simple entry check using a cover sheet, it is possible to efficiently proceed with the work of a large number of application forms.

Furthermore, if you are uncertain about the simple entry check (visual check) using the cover sheet, you can wait for the system entry deficiency check before performing it, and the checker (user) selects which one to use. Therefore, it is more convenient and the efficiency of business can be improved.

In addition, by adding the contact information of the end user as header information to the application form and the incidental image, the operator on the server side can find the entry deficiency and easily contact the checker.

Furthermore, since the operator on the server side can grasp whether the application form sent by the system has already been checked for deficiencies, the deficiencies are checked with priority given to those that have not been checked for deficiencies in the system. be able to.

As described above, the recording medium on which the program that realizes the functions of the above-described embodiment is recorded is supplied to the system or the device, and the computer (or CPU or MPU) of the system or the device stores the program in the recording medium. Needless to say, the object of the present invention can be achieved by reading and executing.

In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium on which the program is recorded constitutes the present invention.

Recording media for supplying programs include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon. A disk or the like can be used.

Further, by executing the program read by the computer, not only the function of the above-described embodiment is realized, but also the OS (operating system) or the like running on the computer is actually operated based on the instruction of the program. Needless to say, there are cases where a part or all of the processing is performed and the processing realizes the functions of the above-described embodiment.

Further, after the program read from the recording medium is written in the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the function expansion board is based on the instruction of the program code. It goes without saying that there are cases where the CPU or the like provided in the function expansion unit performs a part or all of the actual processing, and the processing realizes the functions of the above-described embodiment.

Further, the present invention may be applied to a system composed of a plurality of devices or a device composed of one device. It goes without saying that the present invention can also be applied when it is achieved by supplying a program to a system or an apparatus. In this case, by reading the recording medium in which the program for achieving the present invention is stored into the system or the device, the system or the device can enjoy the effect of the present invention.

The form of the above program may consist of object code, program code executed by an interpreter, script data supplied to an OS (operating system), and the like.

Further, by downloading and reading a program for achieving the present invention from a server 102, a database, or the like on a network by a communication program, the system or device can enjoy the effects of the present invention. It should be noted that all the configurations in which each of the above-described embodiments and modifications thereof are combined are also included in the present invention.

100 Form system 101 Information processing device 102 Server 103 Network

Claims (10)

A confirmation means for checking documents related to image data based on the result of processing related to character recognition for image data, and
Confirmation status information acquisition means for acquiring confirmation status information indicating whether or not the check process has been performed by the confirmation means, and
Ancillary image information acquisition means for acquiring ancillary image information related to the image data incidental to the image data, and
And output means for outputting the combination of the supplementary image information acquired by the supplementary image information acquiring unit and the acquired verification status information by the pre-outs image data the verification status information acquisition unit,
The information processing apparatus characterized by Ru with a. The output means is characterized in that, when instructed to add the incidental image data to the output image data, the output means outputs the incidental image information related to the incidental image data to be added. The information processing apparatus according to 1. The collateral image information processing apparatus according to claim 1 or 2, characterized in that the number of image data to be attached. Equipped with a shooting means to obtain image data by shooting by the shooting unit
The confirmation status information acquisition unit, according to claim 1, characterized in that to obtain the confirmation status information and against the image data obtained by said imaging means indicating whether the check processing by the confirmation means has been performed The information processing apparatus according to any one of items 3 to 3. When a new shooting instruction is given by the shooting means to the output image data, the output means outputs the incidental image information related to the incidental image data obtained by the shooting by the shooting means. The information processing apparatus according to claim 4. Further equipped with a reception means for accepting the user's check processing of documents related to the image data. The information processing apparatus according to any one of claims 1 to 5, wherein the information processing apparatus is obtained. The receiving means uses the image data and the confirmation status information acquisition means by the output means. The acquired confirmation status information and the incidental image information acquired by the incidental image information acquisition means are combined. The feature is that the check process by the user is accepted on the screen that is output together. The information processing device according to claim 6. Display a list of documents that have not been checked by the user by the reception means. With additional display means
The display means is the confirmation status information acquisition means in the list of displayed documents. Claim 6 or 7 characterized by displaying the confirmation status information related to the acquired document. The information processing device described in. Based on the results of processing according to the character recognition for the image data, a verification step to check the documents related to said image data,
A confirmation status information acquisition step for acquiring confirmation status information indicating whether or not the check process has been performed by the confirmation step, and a confirmation status information acquisition step.
Ancillary image information acquisition step for acquiring ancillary image information related to the image data incidental to the image data, and
An output step of outputting together with supplementary image information acquired by the supplementary image information acquiring step and the obtained confirmed status information by the verification status information acquisition step and before Kiga image data,
A control method for an information processing device, which is characterized by executing. Computer,
A confirmation means for checking documents related to image data based on the result of processing related to character recognition for image data, and
Confirmation status information acquisition means for acquiring confirmation status information indicating whether or not the check process has been performed by the confirmation means, and
Ancillary image information acquisition means for acquiring ancillary image information related to the image data incidental to the image data, and
And output means for outputting the combination of the supplementary image information acquired by the supplementary image information acquiring unit and the acquired verification status information by the pre-outs image data the verification status information acquisition unit,
A program to make it work.

Images