JP6708935B2 - Information processing apparatus, processing method thereof, and program - Google Patents

Description

The present invention relates to an information processing device that reads an item included in an object, a processing method thereof, and a program.

Conventionally, there is a mechanism in which form data is transmitted from a user terminal to a form server, the form server side checks whether or not there are any deficiencies in the entered contents, and returns the check result to the user terminal.

In Patent Document 1, electronic processing in which all processing such as determination of entry contents of a form transmitted from a user terminal, error correction by operator operation, creation of a processing result with a correction result message added, and the like is performed on the form server side The technology of the system is described.

JP-A-6-28275

However, in the system configuration of Patent Document 1, when the form transmitted from the user terminal causes an error due to the entered contents, the data of the entire form must be retransmitted.

On the other hand, when checking for incompleteness in a captured image obtained by capturing an image using an image capturing device, characters or symbols may be deleted due to unclearness of captured images, crushing of characters or symbols, fading, overlapping, paper stains, dust, etc. It may not be read correctly and a "read error" may occur. This "reading error" is difficult to eliminate even if the entire form is re-imaged, and may be eliminated by enlarging the error location and re-imaging. However, in this case, since the error location is enlarged and re-imaged, a part of the form is imaged instead of the entire form, and thus it is not possible to specify which form is being imaged.

Therefore, it is an object of the present invention to provide a mechanism for reading an item included in an object after identifying the object to be imaged.

The present invention is an information processing apparatus including an imaging device, said obtained from electronic data captured using the imaging device, based on the identification information of the imaged object, determining an item read from the electronic data a reading item determining means, while performing the display of the image based on the electronic data, and item reading means for reading an item determined by the electronic wherein the data reading item determining means, read by the item reading means item If there is an item for which the value read by the item reading unit is determined to be abnormal, based on the management unit that manages the condition for determining whether the value of is normal or abnormal, and the condition that is managed by the management unit, An instruction receiving unit that receives a selection of whether to capture electronic data obtained by imaging the entire object or enlargement of a portion related to the item determined to be abnormal in the object and capturing electronic data taken. Is characterized by .

According to the present invention, it is possible to provide a mechanism for reading an item included in an object after identifying the object to be imaged.

The figure which shows an example of a structure of the form system 100 of this invention. The figure which shows an example of the hardware constitutions of the information processing apparatus 101. Block diagram showing an example of the functional configuration of the information processing apparatus 101 Flowchart showing an example of the entry deficiency check process in the form system 100. Flowchart showing an example of entire entry deficiency check processing in the form system 100 Flowchart showing an example of partial entry deficiency check processing in the form system 100 The figure which shows an example of the transmission image confirmation screen displayed on the information processing apparatus 101. The figure which shows an example of the entry deficiency check result screen (no entry deficiency) displayed on the information processing apparatus 101. The figure which shows an example of the entry deficiency check result screen (with entry deficiency) displayed on the information processing apparatus 101. The figure which shows an example of the defective part re-imaging screen displayed on the information processing apparatus 101. The figure which shows an example of the table which manages the form form in the form system 100. The figure which shows an example of the table which manages the deficiency check recognition area and check method for each form in the form system 100. The figure which shows an example of the result list of the entry deficiency check process in the form system 100. The flowchart which shows an example of the entry deficiency check process in the form system 100 of 2nd Embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[First Embodiment]
As shown in FIG. 1, the form system 100 according to the present embodiment has a configuration in which an information processing apparatus 101 and a server 102 are communicably connected via a network 103.

The information processing apparatus 101 captures an image of a form using a camera or the like and sends it to the server 102. The server 102 manages, for each form identification information (hereinafter, referred to as a form ID), a recognition area and recognition condition of the content of the contents, which is necessary for checking the incompleteness of the received form image.

In this embodiment, the server 102 performs the entry defect check, but the present invention is not limited to this. The information processing apparatus 101 receives the recognition area and the recognition condition from the server 102, and the information processing apparatus 101 receives the recognition area and the recognition condition. You may check for incompleteness.

The information processing apparatus 101 is a portable terminal such as a mobile phone, a smart phone, a tablet terminal, a notebook PC, and a PDA terminal, in which a browser or a form system application is installed, and wireless communication is performed via the network 103. , And can communicate with the server 102.

Hereinafter, an example of a hardware configuration applicable to the information processing apparatus 101 shown in FIG. 1 will be described with reference to FIG.

In FIG. 2, reference numeral 201 denotes a CPU, which centrally controls each device and controller connected to the system bus 204. Further, the ROM 203 or the external memory 212 is necessary to realize a control program of the CPU 201, such as a BIOS (Basic Input/Output System), an operating system program (hereinafter, OS), and a function executed by each server or each PC. Various programs and the like are stored.

A RAM 202 functions as a main memory and a work area of the CPU 201. The CPU 201 implements various operations by loading a program or the like required for executing the processing from the ROM 203 or the external memory 212 into the RAM 202 and executing the loaded program.

An input controller 205 controls the input from the touch sensor 210. A video controller 206 controls display on a display device such as the display 211. Although the display 211 is shown in FIG. 2, the display may be not only a CRT but also another display such as a liquid crystal display. These are used by the user of the information processing apparatus 101 as needed.

A memory controller 207 is installed in an external storage device (hard disk (HD)) for storing a boot program, various applications, font data, user files, edit files, various data, a flexible disk (FD), or a PCMCIA card slot. It controls access to an external memory 212 such as a CompactFlash (registered trademark) memory connected via an adapter.

A communication I/F controller 208 connects and communicates with an external device via the network 103, and executes communication control processing on the network. For example, communication using TCP/IP is possible.

It should be noted that the CPU 201 enables display on the display 211, for example, by executing outline font rasterization processing in a display information area in the RAM 202. Further, the CPU 201 can receive a touch operation instruction from the user on the touch sensor 210 on the display 211.

Various programs and various tables to be described later for realizing the present invention are recorded in the external memory 212, loaded into the RAM 202 as needed, and executed by the CPU 201.

A captured image input controller 209 controls an input from the camera 215 via the captured image input processing unit 214. The captured image input from the camera 215 can be recognized and captured.

FIG. 3 is a block diagram showing an example of the functional configuration of the information processing apparatus 101 and the server 102.

The capturing unit 301 is a functional unit that captures a form as electronic data by capturing an image with an image capturing device.

The management unit 302 is a functional unit that manages form form definition information including a recognition region of a form and a recognition condition.

The entry deficiency determining unit 303 uses the recognition area and the recognition condition managed by the management unit 302 to determine whether the content entered in the recognition area included in the electronic data captured by the capturing unit 301 is normal or not. It is a functional unit that determines an error.

The re-capturing unit 304 is a functional unit that captures electronic data by re-imaging the form with an imaging device when the entry deficiency determining unit 303 determines an error.

The same form determination unit 305 determines whether the form included in the electronic data captured by the capture unit 301 and the form included in the electronic data captured by the recapture unit 304 are the same form. Is a functional unit for determining.
Equipped with
When the same form determination unit 305 determines that the same form is the same form, the entry defect determination unit 303 determines that the entry defect determination unit 303 included in the electronic data captured by the recapture unit 304 has an error. The content entered in the recognition area is a functional unit that determines normality or error.

The re-capturing unit 304 recognizes all or part of the form necessary for determining the same form by the same form determining unit 305, and the recognition area determined to be an error in the recognition area managed by the management unit 302. It is a functional unit that captures a moving image including an area as electronic data.

The guide display unit 306 is a functional unit that displays a guide for capturing the whole or part of the form and the recognition area determined to be an error so as to capture a moving image.

The imaging determination unit 307 is a functional unit that determines whether or not the entire or part of the form and the recognition area determined to be an error have been imaged according to the guide displayed by the guide display unit 306.

The guide display unit 306 is a functional unit that sequentially displays a guide for capturing the entire or part of the form that has not been captured and the recognition region determined to be an error when the image capturing determination unit 307 determines that the image has been captured. is there.

The one-sheet form determination unit 308 is a functional unit that determines whether the form captured by the recapture unit 304 as a moving image is one form.

The notification unit 309 is a functional unit that causes the re-capturing unit 304 to perform re-imaging or notifies the user of an error when the one-sheet form determination unit 308 determines that the form is not one.

When the entry deficiency determination unit 303 determines that an error has occurred, the instruction receiving unit 310 captures a moving image in which an area including the recognition region determined to have the error is enlarged as compared to the time of capturing the form, or the entire form. Is a functional unit that accepts a selection of whether to image.

The capturing unit 311 is a functional unit that captures an object by using an image capturing device as electronic data.

The object identification information identifying unit 312 is a functional unit that identifies identification information for identifying an object from the electronic data captured by the capturing unit 311.

The read item determination unit 313 is a functional unit that determines the item to be read from the electronic data based on the identification information of the object identified by the object identification information identification unit 312.

The item reading unit 314 is a functional unit that displays an image based on the electronic data captured by the capturing unit 311 and reads the item determined by the read item determining unit 313 from the electronic data.

The re-capturing unit 315 is a functional unit that captures the object as electronic data by re-imaging the object using the imaging device.

The same object determining unit 316 determines whether the object included in the electronic data captured by the capturing unit 311 and the object included in the electronic data captured by the recapturing unit 315 are the same object. Is a functional unit for determining.

The item reading unit 314 is a functional unit that reads the item determined by the read item determining unit 313 from the electronic data captured by the recapturing unit 315 when the same object determining unit 316 determines that the object is the same object. is there.

The management unit 317 is a functional unit that manages a condition for determining whether the value of the item read by the item reading unit 314 is normal or abnormal.

The value determination unit 318 is a functional unit that determines whether the value read by the item reading unit 314 is normal or abnormal based on the condition managed by the management unit 317.

The item reading unit 314 is a functional unit that reads an item determined to be abnormal by the value determination unit 318.

The guide display unit 319 is a functional unit that displays a guide for capturing an image of an item determined to be abnormal by the value determination unit 318.

The guide display unit 319 is a functional unit that displays a guide for capturing an image of an item that has been determined to be abnormal by the value determination unit 318 and that has not been captured.

When there is an item determined to be abnormal by the value determination unit 318, the instruction receiving unit 320 takes in electronic data of an image of the entire object or magnifies an image of the item of the object determined to be abnormal. It is a functional unit that accepts a selection as to whether to capture the electronic data.

The notification unit 321 notifies the user of an error when the object identification information identification unit 312 identifies the identification information of the object and then the object identification information identification unit 312 identifies identification information different from the identification information. It is a department.

FIG. 4 is a flowchart showing an example of the entry deficiency check process in the form system 100.

The entry deficiency check result screen (no entry deficiency) 800 in FIG. 8 is an example of the entry deficiency check result screen displayed on the information processing apparatus 101, and is a case where there are no entry deficiencies. "There were no incomplete entries." is displayed at the top of the screen, and the captured form image is displayed below it.

9 is an example of an entry defect check result screen displayed on the information processing apparatus 101, and is a case where there is an entry defect. “There are deficiencies in the entry of ○ areas.” is displayed at the top of the screen, and an image in which the deficiencies of entry in the captured form image (902, 903) are painted in red is displayed below it.

It should be noted that the display of the deficient entry areas 902 and 903 is not limited to the filling with red, and the display can be made by the user such as filling with a color other than red, enclosing with a frame, blinking, enlarging, and reverse display. I wish I had it. Further, the identification method is not limited to the display identification method, and may be an audio identification method such as reading out an incomplete portion by voice.

FIG. 11 is a diagram showing an example of a table that manages form forms in the form system 100. The form table 1100 is stored in the external memory 212 of the server 102 and manages “form ID”, “form name”, and “file name of form image” of the form form managed by the server 102.

The records of the form table 1100 are registered by the system administrator of the form system 100, and the system administrator performs maintenance using a maintenance screen (not shown). Specifically, when a form form is newly created, a new record is registered in the form table 1100. When the form image file is changed due to the layout change or wording change of the form, the value of the form image in the form table 1100 is updated to the changed form image file.

The deficiency check process of FIG. 4 will be described.

In step S401, the information processing apparatus 101 accepts login to the form system performed by the user of the form system. Note that, in this entry defect checking process, the form application serving as the user interface may be an application installed in the information processing apparatus 101 or may be an application such as a Web application displayed on a browser.

In step S<b>402, the information processing apparatus 101 uses the camera 215 to capture an image of a completed form that is to be checked for incompleteness, and stores the captured image (hereinafter, captured image) in the external memory 212. That is, step S402 is a step showing an example of a process of capturing the form as electronic data by capturing an image with the image capturing device. Further, step S402 is a step showing an example of a process of capturing an object as electronic data by capturing an image of the object using the image capturing device. In this embodiment, the camera 215 of the information processing apparatus 101 is used to obtain a captured image. However, an image captured using a digital camera or an image scanned using a scanner is used as the information processing apparatus. It may be taken into 101.

In step S403, the information processing apparatus 101 sets the variable “form ID”=NULL and the variable “group ID”=“0” and stores them in the RAM 202. The “group ID” is a variable that indicates which part of the form was enlarged and re-imaged when the form was re-imaged.

In step S404, the information processing apparatus 101 transmits the “captured image”, the “form ID”, and the “group ID” to the server 102.

Specifically, before transmitting to the server 102, the information processing apparatus 101 displays the transmission image confirmation screen 700 (FIG. 7), and when the depression of the transmission button 704 is accepted, the “captured image 701” and the “form ID” are displayed. Send the "group ID". In this example, it is assumed that there is a portion where the entered content is determined to be unclear due to a weak writing pressure, a problem that the light is reflected at the time of image capturing and the image cannot be captured neatly (702 in FIG. 7, 703).

Note that when this step S404 is executed after step S427 or step S429 described later, the “form ID” of the captured image will be described later because the captured image re-captured in step S425 or step S428 described later is transmitted. It is specified in step S407. Therefore, the value of the "form ID" included in the entry defect check result list received in step S413 is set as the "form ID" and is transmitted to the server 102.

In step S405, the server 102 receives the “captured image”, the “form ID”, and the “group ID”. Note that, as will be described later in step S427, when re-imaging is performed with a moving image, the "group ID" may be in a list format.

In step S406, the server 102 determines the value of the group ID, the process proceeds to step S407 if it is “0”, and proceeds to step S409 if it is other than “0”.

In step S407, the server 102 identifies the form ID of the form of the captured image. That is, step S407 is a step showing an example of a process for identifying the identification information for identifying the object from the captured electronic data. Specifically, all the records in the form table 1100 are searched, and the form image extracted from the external memory 212 is compared with the captured image using the file path stored in the “form image” column to complete the deficiency. The form ID of the form form to be checked is specified. The form image extracted from the external memory 212 is compared with the captured image, and when the images match at a ratio of a predetermined value or more, it is determined that the captured image is the form form of the form image extracted from the external memory 212, and the image is captured. The form ID of the form is specified to be the form ID corresponding to the form image matched in the image comparison.

It should be noted that the two images used for this image comparison may be compared as they are unprocessed, or the images may be compared after being subjected to image processing such as contour extraction in order to improve the accuracy of the image comparison. Good.

In this embodiment, the form image to be checked for incompleteness of the form is specified by comparing the captured image with the form image managed by the server 102. However, the form ID is not limited to this method. It is also possible to use a method in which a form ID of the form is specified by reading a bar code or a number indicating the number on the form and analyzing the captured image or reading the captured image. In the present embodiment, it is assumed that the “shopping credit application form” of form ID=FORM01 is specified from the captured image.

In step S408, the server 102 performs the entire entry deficiency check process (FIG. 5). That is, step S408 is a step showing an example of a process for determining whether the content written in the recognition area included in the captured electronic data is normal or error, using the managed recognition area and the recognition condition. Is.

Here, FIG. 5 will be described.

FIG. 5 is a flowchart showing an example of the entire entry deficiency check process in the form system 100.

FIG. 12 is a diagram showing an example of a form definition table that manages the entry deficiency check recognition area and check method for each form in the form system 100. The form definition table 1200 is stored in the external memory 212 of the server 102 and is checked with a “form ID”, a “group ID (identifying a group including a plurality of recognition areas)” and a “(recognizing area is identified in the entry defect check)”. The "ID", "recognition condition (for checking deficiency of entry)", "starting point coordinates (X, Y) of (recognizing area)" and "end point coordinates (X, Y)" are managed.

That is, the form definition table 1200 is an example of means for managing the form definition information including the recognition area and the recognition condition of the form. Further, the form definition table 1200 is an example of a unit that manages a condition for determining whether the value of the read item is normal or abnormal.

In this embodiment, the server 102 manages the form table 1100 and the form definition table 1200, and the server 102 checks the deficiency of entries, but the present invention is not limited to this. 1100 and the form definition table 1200 may be managed, and the information processing apparatus 101 may check for incompleteness.

The entire entry deficiency check process of FIG. 5 will be described.

In step S501, the server 102 searches the form definition table 1200 using the “form ID” specified in step S407 as a search key, and acquires the form definition information list. That is, step S501 is a step showing an example of a process of determining the item to be read from the electronic data based on the identification information of the specified object.

In step S502, the server 102 substitutes 1 for the variable i and stores it in the RAM 202.

In step S503, the server 102 repeats steps S504 to S509 by the number of records in the form definition information list acquired in step S501.

In step S504, the server 102 acquires the starting point (X coordinate, Y coordinate) and end point (X coordinate, Y coordinate) of the recognition area and the recognition condition from the i-th record of the form definition information list.

In step S505, the server 102 determines the value of the acquired recognition condition. If the recognition condition=“essential”, the process proceeds to step S506, and if the recognition condition=“circled”, the process proceeds to step S507.

In step S506, the server 102 determines whether a character or symbol has been entered because the recognition area of the captured image corresponds to a required entry item. If characters or symbols are entered, it is determined as "OK" and the process proceeds to step S509, and if no characters or symbols are entered, it is determined as "NG" and the process proceeds to step S508. It should be noted that although this determination processing uses an analysis technique such as OCR, the description thereof is omitted because it is a known technique.

In step S<b>507, the server 102 determines whether or not the circle mark is entered because the recognition area of the captured image corresponds to an item that encloses one of the plurality of options with the circle mark. If the circle is entered, it is determined to be "OK" and the process proceeds to step S509. If the circle is not entered, it is determined to be "NG" and the process proceeds to step S508. Although this determination processing uses an analysis technique such as OMR, the description thereof is omitted because it is a known technique. Although the presence/absence of a circle mark is determined in the present embodiment, the present invention is not limited to the circle mark and may be a determination of a check mark or another figure.

Although the number of circles is “1” in the above description, it is not limited to “1”, and columns such as “upper limit number of checks” and “number of checks” are provided in the form definition table 1200. If the number of circles exceeds the "check upper limit number", "NG" may be determined, and if the number of circles is other than "check number", "NG" may be determined.

In this embodiment, the recognition condition is of two types, "essential" and "encircled", but is not limited to this, and may be "digit number" or "character attribute". In that case, even if a “digit number” column is provided in the form definition table 1200 and it is determined that the entered character is “OK” if the number of digits is permitted and “NG” if the number of digits is not permitted. Good.

Further, the form definition table 1200 is provided with a “character attribute” column, and it is determined as “OK” or “NG” by the attribute of the entered character, for example, “number”, “hiragana”, “katakana”, “alphanumeric character”, or the like. Good.

Further, if the written character cannot be recognized as a result of analysis by OCR or the like, it may be determined as “NG”.

In addition, the determination result of whether the recognition area conforms to the recognition condition is not limited to two, “OK” and “NG”, and values such as “contradiction”, “unclear”, and “unrecognizable” can be used. You may have it.

In step S508, the server 102 inserts the i-th record of the form definition information list into the entry deficiency check result list 1300 (FIG. 13). The entry deficiency check result list 1300 of FIG. 13 is a list having as elements the form definition information record in which the entry deficiency check result=“NG”.

As shown in FIG. 13, the entry failure check result list 1300 has “error details” items, and check entry failures such as “required”, “circled”, “contradiction”, “unclear”, and “unrecognizable”. The reason why the result is judged to be “NG” is recorded. The records 1303 and 1304 for which “error details”=NULL will be described later in the description of step S411.

From the above, steps S505 to S507 are steps showing an example of processing for determining whether the read value is normal or abnormal based on the managed condition.

In step S509, the server 102 increments the value of the variable i.

In step S510, when the record of the form definition information list acquired in step S501 remains, steps S504 to S509 are repeated.

This is the end of the description of the entire entry deficiency check process in FIG. By the entire entry deficiency check process of FIG. 5, the form definition information record in which the entry deficiency check result=“NG” is registered in the entry deficiency check result list 1300.

Returning to the description of the entry deficiency check process in FIG.

In step S409, the server 102 performs a partial entry deficiency check process (FIG. 6). That is, step S409 shows an example of a process of determining whether the content entered in the recognition area determined to be an error included in the captured electronic data is normal or error, when it is determined that the forms are the same. It is a step. Further, step S409 is a step showing an example of a process of reading the item determined to be abnormal. Here, FIG. 6 will be described.

FIG. 6 is a flowchart showing an example of partial entry deficiency check processing in the form system 100. The flowchart of the entire entry deficiency check process in FIG. 5 is replaced with the partial entry deficiency check process, and the same steps as those in FIG. By the processing of FIG. 6, the entry defects of the re-imaging partial form are checked.

In step S601, the server 102 repeats steps S602 to S510 by the number of “group IDs” acquired in step S405.

In step S602, the server 102 searches the form definition table 1200 using the “form ID” received in step S405 and the “group ID” targeted in step S601 as search keys, and acquires the form definition information list. .. That is, step S602 is a step showing an example of a process of determining the item to be read from the electronic data based on the identification information of the specified object.

The subsequent steps S502 to S510 are the same as those described with reference to FIG.

However, the recognition area in steps S506 to S507 is determined according to the distance from the guide frame image included in the captured image. Specifically, in the case of a captured image in which the recognition area of the group ID=2 is enlarged, the rectangle surrounded by the guide frame included in the captured image is the starting point coordinates (100, 100) from the form definition record 1201 of the form definition table 1200. ) And the end point coordinates (450, 300) can be regarded as a rectangle having diagonal points.

Further, when the size of the rectangle surrounded by the guide frame is horizontal=3500 pixels, vertical=2000 pixels, and the group ID=2 and the check ID=8 are checked, the recognition area is the record of the form definition table 1200. Since 1203 is the starting point coordinates (225, 250) and the ending point coordinates (350, 300), the recognition area of step S506 is from the upper left of the rectangular image surrounded by the guide frame,
Starting point ((225-100)*3500/(450-100), (250-100)*2000/(300-100)=(1250 pixels, 1500 pixels),
End point ((350-100)*3500/(450-100), (300-100)*2000/(300-100)=(2500 pixels, 2000 pixels)
It is a rectangle with two points diagonally separated by a distance of.

This can solve the problem that the server 102 that has received the re-captured image cannot specify which part of the form is the enlarged image and cannot determine the entry content. That is, since it is possible to specify which part of the form is the enlarged image, it is possible to perform the entry defect check using the re-imaged image.

In this embodiment, the recognition area is determined according to the distance from the guide frame image included in the captured image that is enlarged and re-imaged. However, the recognition area is not limited to this. A method of memorizing the rate and determining the recognition area using the enlargement rate, a method of re-imaging only the defective portion and using the entire re-imaging image as the recognition area instead of re-imaging in a group of recognition areas And so on.

In step S506, the server 102 determines whether a character or symbol has been entered because the recognition area of the captured image corresponds to a required entry item. If characters or symbols are entered, it is determined as "OK" and the process proceeds to step S509, and if no characters or symbols are entered, it is determined as "NG" and the process proceeds to step S508. It should be noted that although this determination processing uses an analysis technique such as OCR, the description thereof is omitted because it is a known technique.

This is the end of the description of the partial entry deficiency check process of FIG. By the partial entry deficiency check process of FIG. 6, the entry deficiency of the re-imaged partial form is checked, and the result of the entry deficiency check of the re-imaged partial form is “NG” in the entry deficiency check result list 1300. The form definition information record is registered. Returning to the description of the entry deficiency check process in FIG.

In step S410, the server 102 acquires the guide frame coordinates required for reimaging in step S425 described later. Specifically, the "form ID" received in step S405 or specified in step S407, the "group ID" included in the entry defect check result list 1300 generated in step S408 or step S409, and the "check" The form definition table 1200 is searched by using “ID”=NULL as a search key, and the coordinates of the recognition area group are acquired.

Here, the form definition table 1200 is set with “form ID”=“FORM01”, “group ID”=2 and 3 from the entry defect check result list 1300 of FIG. 13 and “check ID”=NULL as search keys. By searching, the form definition records 1201 and 1202 are acquired.

In step S411, the server 102 adds the guide frame coordinates acquired in step S410 to the entry defect check result list 1300. Here, the form definition records 1201 and 1202 acquired in step S410 are added to the entry defect check result list 1300 (1303 and 1304).

In step S412, the server 102 transmits the entry deficiency check result list 1300 to the information processing apparatus 101.

In step S413, the information processing apparatus 101 receives the entry deficiency check result list 1300.

In step S414, the information processing apparatus 101 determines whether or not any deficiency is detected in the entry deficiency check. Specifically, it is determined whether or not a record exists in the entry deficiency check result list 1300. If it is determined that there is no defect, the process proceeds to step S415, and if it is determined that there is a defect, the process proceeds to step S416.

In step S415, the information processing apparatus 101 displays the entry defect check result screen (no entry defect) 800 (FIG. 8). Specifically, the captured image 801 imaged in step S402 is displayed on the display 211, the message "There is no deficiency of entry" is superimposed, and the entry deficiency check process of FIG. 4 is ended.

In step S416, the information processing apparatus 101 displays the entry deficiency check result screen (entry deficiency) 900 (FIG. 9). Specifically, the captured image 901 captured in step S402 is displayed on the display 211, the message "There is a deficiency in the entry" is superimposed, and the entry deficient area is identified and displayed (902 and 903 in FIG. 9). ). It should be noted that the identification display of the incomplete entry area is performed from the record (1301, 1302) in the incomplete entry check result list 1300 acquired in step S413 from the start point coordinates (X, Y) and the end point coordinates (X, Y) of the “recognition area”. Is obtained, and a rectangle having the two points on the diagonal is drawn on the captured image and painted in red. As a result, it is possible to distinguishably display the area where the entry is incomplete.

Further, in the entry deficiency check result screen (entry deficiency) 900, a “recapture image button 904” and a “recapture image (video) button 905” of the flawed portion are displayed, and depending on the content of the entry deficiency check, Allow the user to select a reimaging method.

In step S417, the information processing apparatus 101 receives a user operation. If the "reimage entire image button 904" has been pressed, the process proceeds to step S428, and if the "immediately reimage image (moving image) button 905" proceeds to step S418.

That is, in step S417, when it is determined that an error has occurred, a selection is made as to whether to capture a moving image in which the region including the recognition region determined to have the error is enlarged compared to when capturing the document, or to capture the entire document. Is a step showing an example of a process of accepting. If there is an item determined to be abnormal, step S417 acquires electronic data obtained by imaging the entire object, or acquires electronic data obtained by enlarging the image of the object determined to be abnormal. It is a step showing an example of a process of receiving a selection of whether to include.

Specifically, when the user determines that an error has occurred due to the contents entered on the form such as a mistake, the user presses the "reimage entire image button 904" after correcting the contents entered. When an error occurs because the captured image is unclear, that is, when the user determines that the error can be resolved by enlarging and re-imaging, the user “reimaging the defective portion (video) button 905”. Press. As a result, when re-imaging the form, the user cannot freely select whether the error part is enlarged and re-imaging as described above, or the entire form is re-imaging because the error part is re-filled. Can be resolved.

In this embodiment, the user selects "re-imaging the entire image" or "re-imaging the defective portion", but the present invention is not limited to this. If the value of “Details” is “Required”, “Circle”, or “Inconsistency”, consider the entered contents as the cause of the error such as a mistake, display a message to correct the form and re-image the whole, When the value of “detail” is “unclear” or “unrecognizable”, the image capturing is considered to be the cause of the error, and only the defective portion is enlarged and re-imaged. The processing device 101 may determine whether to “re-image the entire image” or “re-image the defective portion”.

In step S418, the information processing apparatus 101 displays the defective portion re-imaging screen 1010 (FIG. 10).

The image of the form displayed on the defective portion re-imaging screen 1010 of FIG. 10 is a real-time display of the image of the form captured by the camera 215.

In step S419, the information processing apparatus 101 starts capturing a moving image. That is, step S419 is a step showing an example of a process of re-imaging the object using the imaging device and capturing the object as electronic data.

In step S420, the information processing apparatus 101 displays the guide frame 1011 or the guide message 1012 for re-imaging the entire form. That is, step S420 is a step showing an example of a process of displaying a guide for capturing the whole or part of the form and the recognition area determined to be an error so as to capture a moving image. This can solve the problem that the user does not know which part of the form should be re-imaged.

In step S421, the information processing apparatus 101 compares the image captured according to the guide displayed in step S420 with the captured image captured in step S402, and determines whether the form is the same. If it is determined that the forms are the same, the process proceeds to step S422, and if it is determined that the forms are different, an error message such as "The form is different from the one checked for incompleteness" is displayed and the process returns to step S420.

That is, step S421 is a step showing an example of processing for determining whether or not the form included in the captured electronic data and the form included in the captured electronic data are the same form. Further, step S421 is a step showing an example of a process of determining whether or not the object included in the electronic data captured first and the object included in the electronic data captured later are the same object. Is. Further, step S421 is a step showing an example of a process of reading an item from the captured electronic data when it is determined that they are the same object.

In the present embodiment, the entire form is re-imaged in step S420 and compared with the captured image captured in step S402, but the present invention is not limited to this method. The “applicant” column may be enlarged and re-imaged, and may be compared with the image in the “applicant” column of the captured image captured in step S402. Alternatively, the serial number or the like that identifies the form may be captured in step S420 to be compared with the serial number portion of the captured image captured in step S402. That is, as long as it can be determined that they are the same form, the form may be the whole or a part.

As a result, it is possible to solve the problem of not knowing whether the form is the same as the form for which the entry deficiency check was previously performed when the error location is enlarged and re-imaged, and the re-imaging after the entry deficiency check is preferably performed. A mechanism can be provided.

In step S422, the information processing apparatus 101 repeats steps S423 to S425 by the number of defective recognition area groups and the number of groups by which steps S423 to S425 are repeated. Specifically, the process is repeated for the number of “group IDs” included in the entry defect check result list 1300 received in step S413 and without duplication. Here, since the records 1301 and 1302 of the entry defect check result list 1300 of FIG. 13 are two of “group ID”={2, 3}, steps S423 to step for “group ID”=2 and 3 Perform S425 respectively.

In step S423, the information processing apparatus 101 detects whether or not the form has been changed. If it is detected that the form has been changed, the process returns to step S420, and if it has not been detected, the process proceeds to step S424. . That is, it is determined whether the form has been changed (replaced) while the moving image is being captured.

That is, step S423 is a step showing an example of processing for determining whether or not the moving image-captured form is one form. Further, step S423 is a step showing an example of a process of notifying the user of an error when identification information different from the identification information is identified after the identification information of the object is identified.

Specifically, if a ruled line or a character of a form is used as a mark and the mark moves smoothly (continuously) in the captured image, it is determined that the form has not been changed, and the mark is the captured image. It suddenly disappears from the inside, the form image enters the captured image from a direction different from the movement of the information processing apparatus 101 (using the gyro function or the GPS function), and the image being captured is in any range of the form. If it is recognized that the end of the form has been reached and the form image continues, the form is determined to have been changed (replaced) during the moving image capturing.

In addition, in the present embodiment, when it is detected that the form is changed, the process returns to step S420 and the moving image capturing is performed again. However, the present invention is not limited to this. You may display an error message such as "Please slowly image one form." That is, step S423 is a step showing an example of the process of re-imaging again or notifying the user of an error when it is determined that the form is not one.

In step S424, the information processing apparatus 101 displays the guide frame 1021 or the guide message 1022 for re-imaging only the entry area group having an entry error. That is, step S424 is a step showing an example of processing for displaying a guide for capturing the whole or part of the form and the recognition area determined to be an error so as to capture a moving image. Further, step S424 is a step showing an example of a process of sequentially displaying a guide for imaging the whole or part of the form that has not been imaged and the recognition area determined as an error. Further, step S424 is a step showing an example of a process of displaying a guide for capturing an image of an item determined to be abnormal. Further, step S424 is a step showing an example of a process of displaying a guide for capturing an image of an item which has been determined to be abnormal and which has not been imaged.

Note that this guide frame has the starting point coordinates (X, Y) and the ending point coordinates (X, Y) of the records (1303, 1304) of “check ID”=NULL included in the entry defect check result list 1300 received in step S413. While maintaining the aspect ratio of ), the display is enlarged to fit on the display 211. As a result, it is possible to solve the problem that the user does not know which part of the form should be enlarged and how much to re-image.

For example, when “group ID”=2 (“(2) applicant” column), the vertical and horizontal directions of the starting point coordinates (100, 100) and the ending point coordinates (450, 300) of the record 1303 of the entry defect check result list 1300. The guide frame 1021 is displayed on the defective portion re-imaging screen 1020 by enlarging the display 211 while keeping the ratio. If "group ID" = 3 ("(3) Workplace" column), the aspect ratio of the starting point coordinates (100, 400) and the ending point coordinates (450, 700) of the record 1304 of the entry defect check result list 1300. While holding, the guide frame 1021 is displayed on the defective portion re-imaging screen 1020 by enlarging the display 211 to fit the display 211.

The image of the form displayed on the defective portion re-imaging screen 1020 of FIG. 10 is an image of the form captured by the camera 215 displayed in real time.

In step S425, the information processing apparatus 101 re-images the recognition area group including the defective portion with a moving image. That is, step S425 is a step showing an example of a process of reading an item from the electronic data while displaying an image based on the captured electronic data.

In addition, in this embodiment, the recognition region group including the defective portion is re-imaged by the moving image, but the present invention is not limited to the moving image, and for example, the image of the form captured by the camera 215 is analyzed to include the defective portion. When the recognition area group is determined to be within the guide frame as a trigger, the information processing apparatus 101 automatically captures a still image, or a mark is printed on the form for each recognition area group, and the camera 215 captures the mark. The information processing apparatus 101 is triggered by the fact that it is determined that the recognition area group including the defective portion has been expanded to a predetermined size by analyzing the image of the printed form and measuring the size of the mark. A method such as an image capturing method may be used.

That is, step S425 is a step showing an example of a process of capturing the form as electronic data by re-imaging the form with the imaging device when it is determined that there is an error. Further, step S425 is a step showing an example of a process of determining whether or not the whole or part of the form and the recognition region determined to be an error are imaged according to the displayed guide.

In the re-imaging, the image including the guide frame 1021 displayed in step S424 is also captured. This can solve the problem that when the form transmitted from the user terminal has an error due to the determination of the entry content, the entry content of the entire form must be re-imaged and re-transmitted. In addition, “reading error” occurred when the characters or symbols could not be read correctly due to blurring of captured images, crushing, blurring, overlapping of characters and symbols, dirt on paper, dust, etc. It is possible to solve the problem that is difficult to solve.

Steps S423 to S425 are repeated by the number of defective recognition area groups. That is, in steps S419 to S426, the moving image including the whole or part of the form necessary for determining that the form is the same and the recognized recognition area determined to be an error among the managed recognition areas are captured. Is a step showing an example of a process of importing as electronic data.

In step S426, the information processing apparatus 101 finishes capturing the moving image and stores the captured moving image in the external memory 212 as electronic data. In the present embodiment, the captured moving image is stored in the external memory 212 and is used in the subsequent entry defect check (step S409). However, the present invention is not limited to this method, and an error portion may be detected from the captured moving image. Alternatively, the captured image may be extracted as a still image, the still image may be stored in the external memory 212, and the still image may be used for checking the entry defects.

In step S427, the information processing apparatus 101 sets a re-imaging group ID list in the group ID. For example, in the case of the “group ID”={2,3} re-imaged, the group ID={2,3}.

In step S428, the information processing apparatus 101 re-images the entire form (not shown).

In step S429, the information processing apparatus 101 sets the group ID=“0”.

This is the end of the description of the entry deficiency check process in FIG.

In this embodiment, the recognition area group including the defective portion is re-imaged. However, the present invention is not limited to this. A method of re-imaging only the defective portion, and a recognition area group including the defective portion are re-imaged. May be trimmed only in the recognition area of the defective portion.

As described above, it is possible to solve the problem of not knowing whether the form is the same as the form for which the entry deficiency check was previously performed when the error location is enlarged and re-imaged, and the re-imaging after the entry deficiency check is preferably performed. A mechanism can be provided. In addition, by using this moving image capturing, by capturing a position where the same form is determined and an error position as a series of moving images, it is ensured that the image is the same form as the form in which the entry defect check was performed previously. be able to.

In addition, even when the error location is enlarged and re-imaged, the form is identified, so that it is possible to solve the problem that it is not possible to identify which form is being imaged. , It is possible to provide a mechanism for reading the items included in the object.

In the present embodiment, the moving image is re-captured, but the present invention is not limited to this method. Instead of re-imaging with a moving image, a still image is captured when the image acquired by the imaging device fits in the guide frame. By doing so, a method of re-imaging the whole or part of the form and the error part may be used.

In addition, it is not essential to store the captured moving image. For example, the image captured by the camera 215 is not stored in the external memory 212, and it is determined that the recognition area is within the guide frame based on the image. A method may be used in which the image is analyzed in real time by using as a trigger to perform character recognition and check processing. At this time, the image captured by the camera 215 does not have to be displayed on the display 211, but it is preferable to display it because it is possible to know where the image is captured and analyzed.

When the image captured by the camera 215 is displayed on the display 211, the image displayed on the display 211 and the image on which character recognition and check processing are performed need not be the same image. That is, the image captured by the camera 215 may be displayed on the display 211, and character recognition and check processing may be performed using the image at a slightly delayed timing. Conversely, character recognition and check processing may be performed using the image captured by the camera 215, and the image may be displayed on the display 211 at a slightly delayed timing. In addition, character recognition and check processing may be performed using an image captured by the camera 215, and an image in which a guide, a message, or the like is superimposed on the image may be displayed on the display 211.

[Second Embodiment]
The first embodiment is a mechanism for suitably performing re-imaging after character recognition by imaging a place for identifying a form and a place for performing character recognition, and the second embodiment is for a place for identifying a form. By imaging the portion where the character recognition is performed, the form to be imaged is identified, and then the items included in the form are read. That is, the first embodiment is a use case of enlarging and capturing an item when re-imaging an error portion, and the second embodiment is not limited to re-imaging and is a use case of enlarging and capturing an item.

FIG. 14 is a flowchart showing an example of the entry deficiency check processing in the form system 100 according to the second embodiment. Note that in the second embodiment, the flowchart of FIG. 4 in the first embodiment is replaced with FIG. 14, and the same steps as those in the first embodiment are given the same step numbers and the description thereof is omitted.

For the sake of simplicity of the second embodiment, it is assumed that the information processing apparatus 101 receives the recognition area and the recognition condition from the server 102, and the information processing apparatus 101 performs the filling error check in FIGS. 5 and 6.

In step S1401, the information processing apparatus 101 receives a user operation. If the "reimage entire image button 904" is pressed, the process returns to step S402, and if the "immediately reimage image (moving image) button 905" is selected, the process proceeds to step S419.

In step S1402, the information processing apparatus 101 compares the image captured in step S419 (moving image or still image cut out from the moving image) with the captured image captured in step S402 to determine whether or not they are the same form. To judge. If it is determined that the forms are the same, the process proceeds to step S409, and if it is determined that the forms are different, an error message such as "The form is different from the one checked for incompleteness" is displayed and the process returns to step S419. That is, step S1402 is a step showing an example of a process of determining whether the form included in the captured electronic data and the form included in the captured electronic data are the same form.

It should be noted that in the present embodiment, the image taken in step S402 and the image taken in step S419 are compared to determine whether or not they are the same form, but the present invention is not limited to this method. Alternatively, a method may be used in which a barcode or a number indicating a form ID is written on a form, and the form ID of the form is specified by reading at the time of image capturing or analysis of the captured image to determine whether or not the same form is used. Good.

Step S402 in FIG. 14 is a step showing an example of a process of capturing an object as electronic data by capturing an image of the object using the image capturing device.

Step S407 in FIG. 14 is a step showing an example of a process for specifying the identification information for identifying the object from the captured electronic data.

Step S409 of FIG. 14 (more precisely, step S602 of FIG. 6) is a step showing an example of a process of determining the item to be read from the electronic data based on the identification information of the specified object.

Step S409 in FIG. 14 (more precisely, step S506 and step S507 in FIG. 6) is a step showing an example of a process of reading an item from the electronic data while displaying an image based on the captured electronic data. is there.

This is the end of the description of FIG.

In the present embodiment, the object to be imaged is a form, but the form is not limited to a form, and a document, slip, printed matter, leaflet, poster, signboard, answer sheet, questionnaire, menu, table, product, product package. , Objects such as animals, plants, living things, landscapes, nature, celestial bodies, buildings, photographs, objects, images, moving images, etc. that can be imaged using a camera.

As described above, it is possible to provide a mechanism for reading an item included in an object after identifying the object to be imaged.

As described above, the recording medium recording the program that realizes the functions of the above-described embodiments is supplied to the system or apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by reading and executing.

In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium recording the program constitutes the present invention.

As a recording medium for supplying the program, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a DVD-ROM, a magnetic tape, a non-volatile memory card, a ROM, an EEPROM, and a silicon. A disk or the like can be used.

Further, not only the functions of the above-described embodiments are realized by executing the program read by the computer, but also the OS (operating system) or the like running on the computer is actually executed based on the instructions of the program. It goes without saying that a case where a part or all of the processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Furthermore, after the program read from the recording medium is written in the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the function expansion board is instructed based on the instruction of the program code. Needless to say, this also includes the case where the CPU or the like included in the function expansion unit performs some or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

Further, the present invention may be applied to a system including a plurality of devices or an apparatus including one device. Further, it goes without saying that the present invention can be applied to the case where it is achieved by supplying a program to a system or an apparatus. In this case, by reading the recording medium storing the program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

The form of the program may be an object code, a program code executed by an interpreter, script data supplied to an OS (operating system), or the like.

100 Information Processing System 101 Information Processing Device 102 Server 103 Network

Claims (9)

An information processing apparatus including an imaging device,
Read item determination means for determining an item to be read from the electronic data, based on identification information of the imaged object obtained from the electronic data taken in using the imaging device,
An item reading unit that reads the item determined by the reading item determination unit from the electronic data while displaying an image based on the electronic data.
Management means for managing conditions for judging whether the value of the item read by the item reading means is normal or abnormal;
When there is an item whose value read by the item reading unit is determined to be abnormal based on the condition managed by the managing unit, electronic data obtained by imaging the entire object is acquired, or An information processing apparatus, comprising: an instruction receiving unit that receives a selection of whether to capture electronic data obtained by enlarging a portion related to an item determined to be abnormal. A capturing means for capturing the object as electronic data by capturing an image of the object using the image capturing device;
Object identification information identifying means for identifying identification information for identifying the object from the electronic data captured by the capturing means;
A value determining unit that determines whether the value read by the item reading unit is normal or abnormal based on the condition managed by the managing unit;
The information processing apparatus according to claim 1, wherein the item reading unit reads an item determined to be abnormal by the value determining unit. Re-acquisition means for re-imaging the object using the imaging device to capture it as electronic data,
The same object determining means for determining whether or not the object included in the electronic data captured by the capturing means and the object included in the electronic data captured by the recapturing means are the same object. And further,
The item reading means,
If it is determined that the identical object by the same object determination unit, to claim 2, characterized in that to read the item determined by the reading item determining means from the electronic data captured by the reuptake means The information processing device described. The information processing apparatus according to claim 2, further comprising a guide display unit that displays a guide for capturing an image of an item determined to be abnormal by the value determination unit. The guide display means,
The information processing apparatus according to claim 4, further comprising: displaying a guide for capturing an image of an item that has not been imaged among the items determined to be abnormal by the value determining unit. The guide display means displays a guide for imaging an item determined to be abnormal by the value determination means while displaying an image based on the electronic data. Information processing equipment. After the object identification information specifying unit specifies the identification information of the object, when the object identification information specifying unit specifies the identification information different from the identification information, a notification unit for notifying the user of an error is further provided. The information processing apparatus according to any one of claims 2 to 6, characterized in that. A processing method in an information processing apparatus including an imaging device, comprising:
The information processing device is
A read item determination step of determining an item to be read from the electronic data, based on identification information of an imaged object, which is obtained from electronic data captured by using the imaging device;
An item reading step of reading the item determined in the reading item determining step from the electronic data while displaying an image based on the electronic data, and determining whether the value of the item read in the item reading step is normal or abnormal Management steps for managing conditions,
If there is an item whose value read by the item reading step is determined to be abnormal based on the condition managed by the managing step, electronic data obtained by imaging the entire object is captured, or An instruction receiving step of receiving a selection as to whether to capture electronic data obtained by enlarging a portion related to the item determined to be abnormal, and executing the instruction receiving step. A program executable by an information processing device including an imaging device,
The information processing device,
Read item determination means for determining an item to be read from the electronic data, based on identification information of the imaged object obtained from the electronic data taken in using the imaging device,
An item reading unit that reads the item determined by the reading item determination unit from the electronic data while displaying an image based on the electronic data.
Management means for managing conditions for judging whether the value of the item read by the item reading means is normal or abnormal;
When there is an item whose value read by the item reading unit is determined to be abnormal based on the condition managed by the managing unit, electronic data obtained by imaging the entire object is acquired, or A program for functioning as an instruction receiving unit that receives a selection of whether to capture electronic data obtained by enlarging a portion related to the item determined to be abnormal.

Images