JP6712045B2 - Information processing system, its processing method, and program - Google Patents

Description

The present invention relates to an information processing system for notifying a deficiency of a read document, a processing method thereof, and a program.

Conventionally, there is a mechanism in which a completed document is read, a deficiency in the entry is checked, and the check result is displayed on the information processing device.

Patent Document 1 describes a technique of an electronic processing system that checks the content of application data and determines an unacceptable error such as a format error or omission of entry of an item.

JP-A-6-28275

When the application form is checked for defects, it is necessary to present the result of the defect check to the applicant and request correction of the application form. However, in the system configuration of Patent Document 1, when the deficiency check of the application forms is performed by a plurality of people at once, there is a risk that the application content and the deficiency check result are presented to a person different from the applicant of the application form.

As a specific example of the case where the incompleteness check of the application forms of multiple people is carried out at once, in a shopping mall or an event hall, a plurality of shopping credit application forms etc. filled in by each customer are collectively read and checked. Conceivable. In that case, there is a problem in that the applicant and the applicant must be informed of the details of the application and the result of the defect check in order to prevent leakage of personal information.

Also, even if the application form is checked by one person instead of multiple people, the risk of presenting the application details and the result of the defect check to a person different from the applicant of the application form due to spoofing etc. Therefore, in this case as well, there is a problem that the applicant must be sure to present the application content and the defect check result.
In addition to the shopping credit application form described above, specific examples of cases in which identity verification is performed using an identification card include application forms such as a mortgage and life insurance. In addition, from January 2016, in administrative procedures using My Number, identity verification has come to be performed using identity verification documents.

Therefore, an object of the present invention is to provide a mechanism for facilitating the identity verification of a document applicant.

The present invention is an information processing system for managing the image data of the captured document using an information processing apparatus, and image data identification that includes the applicant's face image of the document, prior Symbol A storage unit that stores the document image data captured by using the information processing apparatus and the identification certificate image data in association with each other, and a face image that specifies a face image from the identification certificate image data stored in the storage unit. specifying means, in the case of checking the document image data, the document image data stored in the storage means, has been identified from this the document image data and rather with corresponding identification image data by the face image identification unit characterized in that it comprises a Viewing control means for displaying both controlled on the screen of the information processing apparatus and a face image.

According to the present invention, it is possible to provide a mechanism for facilitating identification of a document applicant.

The figure which shows an example of a structure of the information processing system 100 of this invention. The figure which shows an example of the hardware constitutions of the information processing apparatus 101. Block diagram showing an example of the information processing system 100 The flowchart which shows an example of the process which displays the check result of incompleteness in the information processing apparatus 101. Flowchart showing an example of the entry defect check processing performed by the server 102 The figure which shows an example of the screen which displays the entry deficiency check result list in the information processing apparatus 101. The figure which shows an example of the screen which displays the deficiency check result and identification card in the information processing apparatus 101. The figure which shows an example of the table which manages a document form in the server 102. The figure which shows an example of the table which manages the incompleteness check for each form in the server 102. The figure which shows an example of the table which manages various certificates in the server 102. The figure which shows an example of the table which manages the picked-up image in the information processing apparatus 101 or the server 102. The figure which shows an example of the table which links|relates and manages a captured image, a form, an applicant, and a face image file in the information processing apparatus 101 or the server 102. The figure which shows an example of the table which manages the entry deficiency check result list for every application form image in the information processing apparatus 101 or the server 102. The flowchart which shows an example of the process which displays the check result of an entry defect in the information processing apparatus 101 which concerns on 2nd Embodiment. The flowchart which shows an example of the process which performs a face-matching determination in the information processing apparatus 101 which concerns on 2nd Embodiment. The figure which shows an example of the screen which displays the entry deficiency check result list (identification card identification error) in the information processing apparatus 101 which concerns on 2nd Embodiment. The figure which shows an example of the screen which displays the entry deficiency check result list (face-to-face photo) in the information processing apparatus 101 which concerns on 2nd Embodiment. The figure which shows an example of the screen which displays the entry deficiency check result list (face matching determination error) in the information processing apparatus 101 which concerns on 2nd Embodiment.

[First Embodiment]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

As shown in FIG. 1, an information processing system 100 according to this embodiment has a configuration in which an information processing apparatus 101 and a server 102 are communicably connected via a network 103. The information processing apparatus 101 captures an image of a document using a camera or the like, and checks the document for incompleteness. The server 102 manages, for each document form identification information, a recognition area and recognition condition of the entry content necessary for checking the entry deficiency of the document.

The information processing apparatus 101 is a portable terminal such as a mobile phone, a smart phone, a tablet terminal, a notebook PC, and a PDA terminal, in which a browser or a document system application is installed, and the network 103 is provided by wireless communication or wired communication. It is possible to communicate with the server 102 via.

In the present embodiment, the information processing device 101 and the server 102 communicate with each other to check the incompleteness and display the result. However, the information processing device 101 has all the functions of the server 102 in the present embodiment. The present invention can be implemented even if the server 102 is not provided. However, in the following description, assuming a more general-purpose document system, the functions of the information processing apparatus 101 and the server 102 will be described separately.

Hereinafter, an example of a hardware configuration applicable to the information processing apparatus 101 shown in FIG. 1 will be described with reference to FIG.
In FIG. 2, reference numeral 201 denotes a CPU, which centrally controls each device and controller connected to the system bus 204. Further, the ROM 203 or the external memory 212 is necessary to realize a control program of the CPU 201, such as a BIOS (Basic Input/Output System), an operating system program (hereinafter, OS), and a function executed by each server or each PC. Various programs and the like are stored.

A RAM 202 functions as a main memory and a work area of the CPU 201. The CPU 201 implements various operations by loading a program or the like required for executing the processing from the ROM 203 or the external memory 212 into the RAM 202 and executing the loaded program.

An input controller 205 controls the input from the touch sensor 210. A video controller 206 controls display on a display device such as the display 211. Although the display 211 is shown in FIG. 2, the display may be not only a CRT but also another display such as a liquid crystal display. These are used by the user of the information processing apparatus 101 as needed.

A memory controller 207 is installed in an external storage device (hard disk (HD)) for storing a boot program, various applications, font data, user files, edit files, various data, a flexible disk (FD), or a PCMCIA card slot. It controls access to an external memory 212 such as a CompactFlash (registered trademark) memory connected via an adapter.

A communication I/F controller 208 connects and communicates with an external device via the network 103, and executes communication control processing on the network. For example, communication using TCP/IP is possible.

It should be noted that the CPU 201 enables display on the display 211, for example, by executing outline font rasterization processing in a display information area in the RAM 202. Further, the CPU 201 can receive a touch operation instruction from the user on the touch sensor 210 on the display 211.

Various programs to be described later for implementing the present invention are recorded in the external memory 212, and are executed by the CPU 201 by being loaded into the RAM 202 as necessary. Further, the definition file, map data, etc. used when the above program is executed are also stored in the external memory 212, and a detailed description thereof will be described later.
A captured image input controller 209 controls input from the camera 215 via the captured image input processing unit 214. It is possible to recognize a captured image input from the camera 215.

FIG. 3 is a block diagram showing an example of the functional configuration of the information processing system 100.
The image acquisition unit 301 is a functional unit that acquires a document and an identification card as an image.
The face image identification unit 302 is a functional unit that identifies a face image from the identification card image acquired by the image acquisition unit 301.

The storage unit 303 is a functional unit that stores the document image acquired by the image acquisition unit 301 and the identification image acquired by the image acquisition unit 301 or the face image specified by the face image specifying unit 302 in association with each other. Is.

The screen display unit 304 displays a screen in which the face image identified by the face image identifying unit 302 is associated with the document image stored in association with the face image in the storage unit 303 or a link for displaying the document image. It is a functional unit to display.

The screen display unit 304 displays the link for displaying the identification card image acquired by the image acquisition unit 301 and the document image or the document image stored in the storage unit 303 in association with the identification image. It is a functional unit that displays a screen that is associated with the links.

The screen display unit 304 identifies that the document image has been displayed or has been confirmed when the document image or the ID image is displayed, or when an input indicating that the user has confirmed the document image or the ID image is received. It is a functional part that is displayed by.
The characteristic information storage unit 305 is a functional unit that stores identification information for identifying the type of identification card and the characteristic information of the identification card in association with each other.
The identification card identification unit 306 is a functional unit that identifies the type of identification card using the characteristic information stored in the characteristic information storage unit 305.
The face image identification unit 302 is a functional unit that identifies a face image according to the type of identification card identified by the identification card identification unit 306.
The applicant identification unit 307 is a functional unit that identifies the applicant of the document from the document image acquired by the image acquisition unit 301.
The owner identification unit 308 is a functional unit that identifies the owner of the identification card from the identification card image acquired by the image acquisition unit 301.

When the applicant identified by the applicant identifying unit 307 and the owner identified by the owner identifying unit 308 match, the storage unit 303 is identified by the document image and the identification card image or the face image identifying unit 302. It is a functional unit that stores the face image associated with it.
The document definition information storage unit 309 is a functional unit that stores the document definition information including the recognition area of the entry content of the document and the recognition condition.

The deficiency check unit 310 is a functional unit that checks the deficiency for each recognition region of the document image acquired by the image acquisition unit 301 according to the recognition region and the recognition conditions stored in the document definition information storage unit 309.

The deficiency area display unit 311 is a functional unit that displays the deficiency of the document checked by the deficiency check unit 310 according to the recognition area stored in the document definition information storage unit 309 so that the user can identify it.
The deficiency number display hand unit 312 is a functional unit that tallies and displays the number of deficiencies checked by the deficiency check unit 310 for each document form.

When the applicant identified by the applicant identifying unit 307 and the owner identified by the owner identifying unit 308 match, the storage unit 303 uses the document image and the identification card image or the face image identifying unit 302. The functional unit stores the identified face image in association with each other.

The screen display unit 304 is a functional unit that displays an error when the applicant specified by the applicant specifying unit 307 and the owner specified by the owner specifying unit 308 do not match.
The image capturing unit 313 is a functional unit that captures the face of the applicant of the document.

The same person determination unit 314 is a functional unit that determines whether or not the face image identified by the face image identification unit 302 and the face image captured by the image capturing unit 313 are images of the same person.

When the same person determination unit 314 determines that the images are images of the same person, the display control 315 controls to display the document image data captured by the image acquisition unit 301, and the same person determination unit 314 controls the same person. Is a functional unit that controls not to display the document image data captured by the image acquisition unit 301 when it is determined that the image is not the image.

When the same person determination unit 314 determines that the images are not images of the same person, the storage unit 303 associates the document image data captured by the image acquisition unit 301 with the face image identified by the face image identification unit. It is a functional unit that is attached and stored.

FIG. 4 is a flowchart showing an example of a process of displaying the check result of the entry defect in the information processing apparatus 101 (hereinafter referred to as the entry defect check result display process). It should be noted that all of the following processing is performed by the information processing apparatus 101 or the CPU 201 of the server 102.

In step S401, the information processing apparatus 101 receives a login to the document system performed by a user of the document system. Note that the document application that serves as a user interface in the entry defect check result acquisition process may be an application installed in the information processing apparatus 101 or an application such as a Web application displayed on a browser.

In step S<b>402, the information processing apparatus 101 uses the camera 215 to shoot an application form for which the entry defect is to be checked, and stores the taken image (hereinafter, taken image) in the external memory 212. In this embodiment, the camera 215 of the information processing apparatus 101 is used to acquire a captured image, but an image captured using a digital camera or an image scanned using a scanner is used as the information processing apparatus. It may be taken into 101.

The "application form" in this embodiment is an example of a "document" handled by the information processing system 100 of the present invention. Examples of "documents" include documents such as "forms", "approval form", "notification form", "questionnaire" in addition to "application form".

In step S403, the information processing apparatus 101 uses the camera 215 to shoot the identity card of the applicant of the application taken in step S402, and stores the taken image in the external memory 212. That is, steps S402 and S403 are steps showing an example of a process of fetching a document and the identification document as image data. In this step as well, similarly to step S402, an image captured by using a digital camera or the like or an image scanned by using a scanner or the like may be loaded into the information processing apparatus 101.

In addition, in this embodiment, a driver's license and a passport are given as examples of the identification card, but the identification is not limited to these, and other than those, the “health insurance card”, “resident resident card”, “(my It may be a document or card for certifying the person such as "number) personal number card".
Steps S402 and S403 are repeated for the number of application forms.

In step S404, the information processing apparatus 101 transmits the captured images of the application form and the identification card stored in the external memory 212 in steps S402 and S403 to the server 102. Here, it is assumed that a plurality of files such as the captured image list 1100 (FIG. 11) are sequentially transmitted.

In step S405, the server 102 receives the captured image and stores the captured image in the external memory 212. Further, the file name of the received captured image file is stored in the “captured image file name” column of the captured image for analysis list 1200 on the external memory 212 in the order of the received captured images.

In step S<b>406, the server 102 acquires one captured image stored in the external memory 212 and uses the file path stored in the “form image” column in each record of the application form table 800 to store the captured image in the external memory 212. The form ID of the captured image is specified by sequentially comparing the images with the extracted form image. When the form ID can be specified, the specified form ID is stored in the “form ID/identification ID” column of the photographed image list for analysis 1200 on the external memory 212, and the process proceeds to step S407. If the form ID cannot be specified, the process proceeds to step S408.

Specifically, the photographed image “20141201_001.jpg” is compared with each form image of the application form table 800, and when the images match at a ratio of a predetermined value or more, the “form of the photographed image list for analysis 1200” is displayed. The value of the form ID of the form image that matches the “ID/ID card ID” column is stored. In the case of this photographed image “20141201 — 001.jpg”, it is determined that it is the form with the form ID “FORM01”, because it matches the form image “D:/form/01.jpg”.

In step S407, the server 102 performs character recognition by OCR, identifies the applicant name described in the application form, and identifies the applicant name in the analysis applicant image list 1200 on the external memory 212. Remember the applicant's name. That is, step S407 is a step showing an example of a process of identifying a document applicant from a document image. In this embodiment, step S407 is the process performed by the server 102, but the information processing apparatus 101 may perform character recognition by OCR to specify the applicant name described in the application form.

In step S408, the server 102 performs an entry deficiency check process (FIG. 5). That is, step S408 is a step showing an example of the process of checking the deficiency for each recognition area of the document image according to the recognition area and the recognition condition.

In this embodiment, step S408 is the process performed by the server 102. However, all the tables and lists (FIGS. 8 to 13) are stored in the information processing apparatus 101, and the information processing apparatus 101 executes step S408. You may do it. In that case, the information processing apparatus 101 performs all the processes in FIG. In that case, transmission/reception of information (for example, steps S404 to S405 and steps S413 to S414) is unnecessary processing.
Here, the entry defect checking process of FIG. 5 will be described.

FIG. 5 is a flowchart showing an example of the entry deficiency check process performed by the server 102.
In step S501, the server 102 acquires one captured image of the application form from the captured image list 1100 on the external memory 212 and stores it in the RAM 202. Specifically, the photographed image record in which the form ID is stored in the “form ID/identification ID” column of the photographed image list for analysis 1200 such as “FORMxx” is acquired, and the “photographed image file” column is further acquired. The captured image files are obtained using the captured image file names stored in the RAM 202 and stored in the RAM 202.

In step S502, the server 102 searches the entry defect check table 900 (FIG. 9) using the form ID of the captured image record stored in the RAM 202 in step S501 as a search key, and the entry defect check record related to the document form to be checked. List is acquired and stored in the RAM 202.

Specifically, since the form ID of the first record in the photographed image list for analysis 1200 (FIG. 12) is “FORM01”, the form ID=FORM01 is used as the search key, and the form ID is checked from the entry defect check table 900. =A list of entry defect check records related to the "shopping credit application form" of FORM01 is acquired and stored in the RAM 202.

The entry deficiency check table 900 in FIG. 9 is an example of a table for managing entry deficiency check recognition areas and recognition conditions for each form ID. In the entry defect check table 900, the form ID, the check ID that is the serial number of the check target item, the recognition condition that is the check content, and the starting point coordinates (X, Y) and the end point coordinates (X , Y) are stored.

The entry defect check table 900 is a table created and maintained by the system administrator as definition information (document form identification information, recognition area and recognition condition of the entered content) of a document used in the information processing system 100. That is, the entry deficiency check table 900 is a functional unit showing an example of a means for storing document definition information including a recognition area of the entry content of a document and a recognition condition.

Although the entry defect check table 900 is stored in the server 102 in this embodiment, it may be stored in the information processing apparatus 101. In this case, the information processing apparatus 101 performs all the entry defect check processing of FIG.

In step S503, the server 102 substitutes 0 for the variable error and stores the variable error in the RAM 202. This variable error is a variable that counts up each time there is a deficiency in the photographed image to be checked for deficiencies.

In step S504, the server 102 substitutes 1 into the variable i and stores the variable i in the RAM 202. This variable i is a variable that becomes a check ID for the entry defect check to be performed on the captured image, and is a variable that counts up to check the next check ID after the check is completed.

In step S505, the server 102 determines the start point (X coordinate, Y coordinate) and end point (X coordinate) of the recognition region of the entry defect check record with check ID=i in the entry defect check record list stored in the RAM 202 in step S504. (Coordinate, Y coordinate) and the recognition condition are acquired. Specifically, when the check ID is 1 in the entry defect check record list of the form ID=1 of the check target form, the recognition area of the entry defect check record=starting point (100, 100) and end point (300, 150) , Recognition condition=“required” is acquired.
In step S506, the server 102 determines the acquired recognition condition.

If the recognition condition is "essential", the recognition area is an input required item, and therefore, in step S507, it is determined whether or not a character or symbol is written in the rectangle indicated by the recognition area of the captured image. That is, step S507 is a step showing an example of the process of checking the deficiency for each recognition area of the document image according to the recognition area and the recognition condition. Note that this determination processing uses an analysis technique such as OCR, but since it is a known technique, description thereof will be omitted.
If it is determined that characters or symbols are entered, the process proceeds to step S511, and if it is determined that characters or symbols are not entered, the process proceeds to step S509.

If the recognition condition is “encircled”, the recognition area is a multiple-choice item, so in step S508, it is determined whether or not a circle is entered in the rectangle indicated by the recognition area of the captured image. That is, step S508 is a step showing an example of the process of checking the deficiency for each recognition area of the document image according to the recognition area and the recognition condition. This determination processing uses an analysis technique such as OMR or OCR, but since it is a known technique, its explanation is omitted. In the present embodiment, the presence/absence of a circle mark is determined by setting the entry deficiency check recognition condition as “circled”, but the present invention is not limited to the circle mark, and may be a check mark or other figure determination. Good.

Further, in the above description, the number of circles is “1”, but the entry defect check table 900 is provided with “upper limit value” and “check number” columns, and when the upper limit value is exceeded, “NG”, the number of marks May be determined to be “NG” if the number is other than “the number of checks”. In addition, even if a “digit number” column is provided in the entry defect check table 900 and it is determined that the entered character is “OK” if it is the permitted digit number and “NG” if it is not the authorized digit number. Good. In addition, a “character attribute” column is provided in the entry defect check table 900, and “OK” or “NG” is determined by the attribute of the entered character, for example, “number”, “hiragana”, “katakana”, “alphanumeric character”, or the like. May be.

Further, when the written character cannot be recognized as a result of analysis by OCR or the like, it may be determined as “NG”. The value of the variable result[i] is not limited to the two values of “OK” and “NG”, and may have values such as “unclear”, “unrecognizable”, and “contradiction”.

If it is determined in step S508 that the circle is filled, the process proceeds to step S511 if it is determined, and to step S509 if it is determined that the circle is not written.
In step S509, the server 102 counts up the variable error. In other words, it means that the deficiency was found.

In step S510, the server 102 inserts a record indicating the error content into the entry deficiency check result list 1300. The entry defect check result record to be inserted in the entry defect check result list 1300 is “captured image file” “check ID” “error content” “starting point coordinates (X, Y)” “end point coordinates (X, Y)”. This entry defect check result record is transmitted to the information processing apparatus 101 after completion of the entry defect check process, and is used to display the entry defect check result list screen.

In step S511, the server 102 counts up the variable i, determines whether or not there is an entry defect check record with the check ID=i in the RAM 202, and if there is, the entry defect check of the document form remains. Since it means that the document form has been entered, the process returns to step S505 to continue the check for incompleteness in the document form. If it does not exist, it means that all the checking for incompleteness of the document form has been completed. Therefore, the process returns to step S501, and the document form is filled in for the next captured image record in the captured image list for analysis 1200 on the external memory 212. Check for defects.

If the next photographed image record does not exist in the RAM 202, this means that all of the photographed image records in the analysis photographed image list 1200 have been checked for entry defects, and thus the entry defect checking process is terminated.
At this point, in the present embodiment, an entry defect check result record such as the entry defect check result list 1300 (FIG. 13) is stored.

This is the end of the description of the entry deficiency check process in FIG. By this processing, the entry defects are checked for all the captured images. Returning to the description of the entry deficiency check result display process in FIG.

In step S409, the captured image acquired in step S406 and the file path stored in the “sample image file” column in each record of the identification certificate master table 1000 (FIG. 10) are used to acquire the image from the external memory 212. The ID card is specified by sequentially comparing the images with the ID card sample images.

That is, step S409 is a step showing an example of a process of identifying the type of identification card using the stored characteristic information. The identification certificate master table 1000 is a functional unit showing an example of a unit that stores identification information for identifying the type of identification certificate and characteristic information of the identification certificate in association with each other.

If the identification ID can be specified, the specified identification ID is stored in the “form ID/identification ID” column of the captured image list for analysis 1200 on the external memory 212, and the process proceeds to step S410. If the identification ID could not be specified, the captured image is an image that could not be specified from the “form image” of the application form table 800 or the “sample image file” of the identification master table 1000. If so, the process returns to step S406.

Specifically, the photographed image “20141201_002.jpg” is compared with each sample image file of the identification card master table 1000, and when the sample image file “D:/cert/01.jpg” matches, the photographed image Since “20141201_002.jpg” can be identified as the “driver's license”, the value of “CERT01” is stored in the “form ID/identification ID” column of the analysis photographed image list 1200.

In step S410, the server 102 acquires a face image from the captured image acquired in step S406. That is, step S410 is a step showing an example of a process of identifying a face image from the acquired ID card image data. Further, step S410 is a step showing an example of processing for identifying a face image according to the identified type of identification card.

Specifically, the identification card master table 1000 is searched by the identification card ID specified in step S408, and the record (face image start point X, face image start point Y), (face image end point X, face) of the hit record is searched. A face image is cut out from the captured image acquired in step S406 by using a rectangle having two diagonal points of the image end point Y). The cut face image is stored in the external memory 212 with a file name such as “captured image file name_trim”.

In this embodiment, the face image is cut out using the coordinates stored in the identification certificate master table 1000, but the present invention is not limited to this method, and the coordinates defined in the identification card master table 1000 are used. Instead of using, the face image of which the face image portion is determined may be cut out from the identification card image using a face recognition technique or the like. Since the face recognition technology is a known technology, the description thereof is omitted here.

In step S411, the server 102 stores the file name of the face image cut out in step S410 in the “ID file/face image file” column of the analysis photographed image list 1200 on the external memory 212. This makes it possible to associate the captured image of the ID card with the face image, so if you want to keep a copy of the ID card on the system, you can apply the image data of the ID card obtained by scanning the ID card and the application. It is possible to solve the problem that the user must perform the work of associating with the data of the book.

In addition, for the purpose of analysis, it is assumed that the identification certificate whose identification ID has been specified in step S409 is an image taken to verify the identity of the application form whose form ID was specified immediately before in step S406. In the photographed image list 1200, the ID card file name specified in step S409 is stored in the “ID card file/face image file” column of the immediately preceding application record.

This allows the photographed image of the application form and the photographed image of the identification card to be associated with each other. At the same time, the photographed image of the application form and the face image can be associated with each other. That is, step S411 is a step showing an example of a process of storing the captured document image data and the identification document image data in association with each other.

In this embodiment, after the application form is taken, the ID card for confirming the identity of the applicant is taken, but the present invention is not limited to this, and the order of shooting may be in any order. .. In that case, the association between the photographed image of the application form and the photographed image of the ID card in step S411 cannot be determined in the order of the photographed images, and therefore the applicant name of the application form and the name of the ID card match. You may use a method such as associating the application form with the ID card.

Specifically, when the “ID card ID” of the identification card is specified in step S409, the OCR of the “name” column of the ID card is performed, and the “Applicant” column of the photographed image list for analysis 1200 is displayed. The name obtained by OCR is stored, and an application record having the same name as the applicant's name stored in the “Applicant” column of the photographed image list for analysis 1200 in step S407 is used as a related application record. judge.

That is, step S409 is a step showing an example of a process of identifying the owner of the identification card from the identification card image. That is, step S411 is a step showing an example of a process of storing the document image and the identification image or the face image in association with each other when the applicant of the document and the owner of the identification document match.

The method of associating the photographed image of the application form with the photographed image of the identification card in step S411 is not limited to these methods, and the related application form and the identification card may be captured in the same image, or It is also possible to take a method such as taking a photograph together with a mark (bar code etc.) which is the same or related to the application form and the identification card.
The above steps S406 to S411 are repeated for the number of images received in step S405.

In step S412, the server 102 totals the number of deficiencies for each application form regarding the entry deficiency check result list 1300 (FIG. 13) stored in the entry deficiency check process (FIG. 5). Specifically, the records of check ID≠NULL in the entry defect check result list 1300 are grouped and counted for each captured image file name.

In step S<b>413, the server 102 transmits the analysis captured image list 1200, the entry defect check result list 1300, and the face image file acquired in step S<b>410 to the information processing apparatus 101.

In step S<b>414, the information processing apparatus 101 receives the analysis shot image list 1200, the entry defect check result list 1300, and the face image file and stores them in the external memory 212.

In step S415, the information processing apparatus 101 uses the received photographed image list for analysis 1200, the entry deficiency check result list 1300, and the face image file to receive the entry deficiency check result list screen 610 (FIG. 6). ) Is displayed on the display 211. Specifically, the face image 611, the applicant name, the number of incompleteness in the application form, the application form button 613, and the identification card button 614 are displayed for each application form.

That is, step S415 is a step showing an example of a process of displaying a screen in which a face image and a link for displaying a document image stored in association with the face image are associated with each other. In addition, in step S415, a screen in which the document image data or the link for displaying the document image data and the face image identified from the identification certificate image data stored in association with the document image data are associated with each other. It is a step showing an example of a process of displaying.

Further, step S415 is an example of a process of displaying a screen in which a link for displaying the identification image and a link for displaying the document image stored in association with the identification image are associated with each other. These are the steps shown. In addition, a step S415 is a screen in which the link for displaying the document image data or the document image data is associated with the link for displaying the identification image data stored in association with the document image data. It is a step showing an example of a process of displaying. Further, step S415 is a step showing an example of processing for totaling and displaying the number of checked defects for each document form.

As for the number of incompleteness in the application form, the records of the received captured image list for analysis 1200 are grouped by "captured image file", and the number of records for each "captured image file" is related to the "captured image file". It is displayed as the number of deficiencies filled in. As a result, the number of deficiencies in each application form can be ascertained, and if there are deficiencies, it is possible to take measures such as calling the applicant to request correction of the deficiencies.

Note that the application file button 613 and the identification card button 614 are associated with the image file names stored in the “photographed image file” column of the received photographed image list for analysis 1200. Accordingly, when the application form button 613 and the identification card button 614 are pressed, it is possible to know which image file should be displayed among the image files captured in step S402 and stored in the external memory 212 on the next screen. ..

In addition, in this embodiment, the application is displayed for each application, but the present invention is not limited to this. If the same applicant submits a plurality of applications, the applications are collected for each applicant. It may be displayed. In that case, when the application form button 613 is pressed, a plurality of application forms related to the applicant are displayed in order or collectively, or a plurality of application form buttons 613 are displayed in the line of the applicant. To take.

If you do not have an identification card related to the application form, you can display a message such as "No ID", "NO IMAGE", or "X" like 612 in Fig. 6 or leave it blank. It is displayed by identifying that the ID card was not photographed or that the type of the photographed ID card could not be specified. As a result, it becomes possible to easily recognize the presence or absence of the identification card related to the application form. Further, when the entire image data of the identification card is displayed on the screen for the identification of the application form, it is possible to solve the problem of taking up space on the screen.

Further, as a result, when a call is made to the applicant to request correction of a deficiency, the face of the applicant is compared with the face image 611 displayed on the display 211 to confirm the identity of the applicant of the document. It is possible to provide a mechanism that facilitates. Therefore, the defect check result can be presented to the applicant without fail, and the risk of accidentally leaking personal information can be reduced.

In step S416, the information processing apparatus 101 accepts an operation by the user and determines the accepted operation. When the application form button 613 is pressed, the image file name of the application form associated with the pressed application form button 613 is acquired and the process proceeds to step S417. When the identification card button 614 is pressed, the image file name of the identification card associated with the pressed identification card button 614 is acquired, and the process proceeds to step S419. If the record defect check result record row is dragged and dropped, the process proceeds to step S420. If the end button 618 is pressed, the entry defect check result acquisition process of FIG. 4 ends.

In step S417, the information processing apparatus 101 uses the application form image captured in step S402 and stored in the external memory 212 and the face image file received in step S415 to complete the entry defect check result screen 710 (FIG. 7). Is displayed.

Specifically, from the application form image captured in step S402 and stored in the external memory 212, an image file having the image file name of the application form associated with the application form button 613 pressed in step S416 is acquired and applied. It is displayed as a calligraphy image 711 (FIG. 7). Further, from the face image file received in step S415, the face image file of the face image file name associated with the application form button 613 pressed in step S416 in the analysis captured image list 1200 also received in step S415. Is acquired and displayed as a face image 712 (FIG. 7).

That is, step S417 is a step showing an example of a process for displaying a screen in which the face image and the document image stored in association with the face image are associated with each other. Further, step S417 is a step showing an example of a process of displaying a screen in which the document image data and the face image identified from the identification image data stored in association with the document image data are associated with each other. .

Further, an identification card button 715 for displaying the identification image is displayed beside the face image 712 or over the face image 712. The identification card image may be displayed by pressing the face image 712 without displaying the identification card button 715.

That is, step S417 is a step showing an example of a process of displaying a screen in which the link for displaying the identification card image and the document image stored in association with the identification card image are associated with each other. In addition, step S417 is a step showing an example of a process of displaying a screen in which the document image data and the link for displaying the identification image data stored in association with the document image data are associated with each other. .

In step S418, the information processing apparatus 101 superimposes and displays the deficient entry portion on the application form image 711 displayed in step S417. That is, step S418 is a step showing an example of a process for displaying the deficiency of the checked document in a distinguishable manner by the user according to the recognition area.

Specifically, the entry defect check result list 1300 is searched using the image file name of the application form corresponding to the application form button 613 pressed in step S416 as a search key, and the starting point coordinates (X, Y) and the ending point coordinates ( (X, Y) is acquired, a rectangle having these two points as a diagonal is superimposed and drawn on the application form image 711, and it is filled with a distinguishable color (for example, red) (filling defect portion 713 in FIG. 7). In this way, it is possible to display the incompletely filled area in an identifiable manner. Further, by this, it is possible to carry out the deficiency check and the person confirmation together on the system.

In this embodiment, the defective portion 713 is painted in red. However, the invention is not limited to this. Filling in with a color or pattern other than red, enclosing with a frame, blinking, enlarging, and reversing display of color. Any display that can be identified by the user may be used. Further, the display method is not limited to the display that can be identified, and an identification method using a tactile sensation such as reading out an incomplete portion by voice or vibration when a screen is touched may be used.

In step S419, the information processing apparatus 101 displays the identification image captured in step S403 and stored in the external memory 212. Specifically, from the ID image captured in step S403 and stored in the external memory 212, an image file having the image file name of the ID associated with the ID button 614 pressed in step S416 is acquired. Then, an identification confirmation screen 720 is displayed. Thereby, the image data of the identification document obtained by scanning the identification document and the image data of the application form can be displayed in association with each other.

In step S420, the information processing apparatus 101 replaces the display row by moving the dragged row to the dropped location, and proceeds to step S415 to redisplay the entry defect check result list screen. By this drag & drop, the order of applicants notified by the person in charge can be changed. For example, when the called applicant is absent, the notification can be postponed by dragging and dropping the entry check result record of this applicant.

Specifically, when the entry defect check result record 616 on the entry defect check result list screen 610 is dragged and dropped below the entry defect check result record 617, the entry defect check result record 621 on the entry defect check result list screen 620. As described above, the order of the deficiency check result record can be changed.

When the information processing apparatus 101 accepts the pressing of the return button by the user in step S421, the processing returns to step S415, and the entry defect check result list screen 620 (FIG. 6) is displayed on the display 211. At this time, like the application form button 622 and the identification card button 623, the displayed application form and identification card identify and display the “completed” mark.

That is, step S415 is a step showing an example of a process of displaying a document image or an identification card image by identifying whether the document image or the identification image has been displayed or confirmed. Further, in step S415, the fact that the document image data or the ID card image data is displayed, or the input indicating that the user has confirmed the document image data or the ID card image data is accepted is distinguished from other images. It is a step showing an example of a process of displaying as.

Specifically, for the captured image file displaying the check result, the “completed” flag is set in the “notified” column of the analysis captured image list 1200 on the external memory 212, and the entry defect check result list screen is displayed in step S415. When “620” (FIG. 6) is displayed, “Completed” mark is displayed on each button when “Completed” is stored in the “Notified” column of the entry defect check result record. As a result, it is possible to easily identify how far the person in charge notifies the applicant, how far the person in charge confirms, which applicant has completed the identification, and so on.

In addition, this “done” mark may be displayed only when the notified check box 714 of the entry deficiency check result screen 710 and the confirmed check box 721 of the identification confirmation screen 720 are checked. That is, step S415 is a step showing an example of a process of identifying and displaying the document image or the identification document image when the input indicating that the user has confirmed the document image or the identification document image is received.
This is the end of the description of the entry deficiency check result acquisition process in FIG.

As described above, it is possible to provide a mechanism that facilitates the identification of the applicant of a document. In addition, the deficiency check and the identity verification can be performed together on the system. Further, the image data of the identification card and the data of the application form can be associated with each other. In addition, the person in charge can correctly understand that the identification of the application form has been completed. Further, when the entire image data of the identification card is displayed on the screen for the identification of the application form, it is possible to solve the problem of taking up a space on the screen.

[Second Embodiment]
The first embodiment displays a screen in which a document image or a document link is associated with a face image and a screen in which a document image or a document link is associated with an identification card link to confirm the identity of a document applicant. The second embodiment is a mechanism for facilitating the process, and in the second embodiment, the person in charge (the user of the information processing system 100) makes the same person determination of the face image of the applicant and the face image of the identification card, thereby making the applicant of the document. It is a mechanism that makes it easy to identify yourself.

FIG. 14 is a flowchart showing an example of a process of displaying a check result of entry defects in the information processing apparatus 101 according to the second embodiment. FIG. 15 is a flowchart showing an example of processing for making a face match determination in the information processing apparatus 101 according to the second embodiment. Note that in the second embodiment, the flowchart of FIG. 4 in the first embodiment is replaced with FIG. 14, and the same steps as those in the first embodiment are given the same step numbers and the description thereof is omitted. It should be noted that all of the following processing is performed by the information processing apparatus 101 or the CPU 201 of the server 102.

In addition, FIG. 16 is a diagram illustrating an example of a screen that displays the entry defect check result list (identification certificate identification error) in the information processing apparatus 101 according to the second embodiment. FIG. 17 is a diagram showing an example of a screen for displaying a list of entry deficiency check results (face-to-face shooting) in the information processing apparatus 101 according to the second embodiment. FIG. 18 is a diagram showing an example of a screen displaying a list of entry defect check results (face match determination error) in the information processing apparatus 101 according to the second embodiment.

The second embodiment will be described with reference to FIGS. 14 and 15.
In step S1401, the server 102 sets “identification certificate identification error” for the immediately previous application form. The specific case of the "identification certificate identification error" is that the identification ID could not be specified from the captured image of the identification acquired in step S406, and the identification that matches the applicant name of the application form. Is not present in the captured image.
In step S1402, the information processing apparatus 101 performs face match determination processing (FIG. 15). The process of FIG. 15 will be described.

In step S1501, the information processing apparatus 101 determines whether or not the application form associated with the application form button 1601 (FIG. 16) pressed in step S416 is set to “identification certificate identification error”. In the case of "identification certificate identification error", the flow proceeds to step S1502, and if not "identification certificate identification error", the flow proceeds to step S1504.

In step S1502, the information processing apparatus 101 displays a message indicating “identification certificate identification error” (1602 in FIG. 16). That is, step S1502 is a step showing an example of a process of displaying as an error when the specified applicant and the specified owner do not match.

In step S1503, the information processing apparatus 101 receives the operation of the user (the person in charge who handles the applicant) and makes a determination. In the case of the cancel button 1603, the process proceeds to END in FIG. 14 and this process ends. In the case of the face-to-face shooting button 1604, the process proceeds to step S1504. In the case of the ID card photographing button 1605, the process proceeds to step S403 in FIG. 14, the ID card is photographed (step S403), and the photographed ID card is transmitted to the server 102 (step S404).

In step S1504, the information processing apparatus 101 uses the camera 215 to capture an image of the applicant (hereinafter, face-to-face person) whom the person in charge faces (FIG. 17). That is, step S1504 is a step showing an example of a process of capturing the face of the applicant of the document. As shown in FIG. 17, an in-person image capturing window 1703 displays an in-person image 1704 captured using the camera 215, a cancel button 1705, and an OK button 1706. To do.

In step S1505, the information processing apparatus 101 compares the face image 1702 of the identification card cut in step S410 of FIG. 14 with the face-to-face image 1704, determines whether the person is the same person, and is the same person. If it is determined that the person is not the same person, the process in FIG. 15 is terminated, and the process proceeds to step S417 in FIG. That is, step S1505 is a step showing an example of a process of determining whether or not the face image identified from the identification card and the captured face image are images of the same person.

Note that the determination of whether or not they are the same person is a known image comparison technique, and thus a description thereof will be omitted. Further, in the present embodiment, the image comparison is used to determine whether or not the same person is determined. However, the present invention is not limited to this, and a feature part is extracted from each image and the extracted person is compared to determine whether the same person is the same person. It may be determined whether or not. In addition to the face image, biometric authentication such as a fingerprint, palm print, and voice print may be used to determine whether or not the same person is present.

In step S1506, the information processing apparatus 101 displays a message indicating “same person determination error” (1801 in FIG. 18). As shown in FIG. 18, a face-to-face image 1802 captured using the camera 215, a cancel button 1803, and a continue button 1804 are displayed. That is, in step S1506, if it is determined that the image is the image of the same person, control is performed to display the captured document image data, and if it is determined that the image is not the image of the same person, the captured document image is acquired. It is a step which shows an example of processing which controls not to display data.

As a result, when it is determined that the person who applied for the application form is not the person who applied, an error message is displayed, so that it is possible to prevent the risk that the contents of the application form are viewed by anyone other than the applicant. In other words, in the first embodiment, the person in charge is solely responsible for confirming the identity of the person who has applied for the application form, but in the second embodiment, the information processing system 100 is also the principal. Since the confirmation is performed, the above risks can be further prevented. In particular, my number, which has been required for administrative procedures since January 2016, must not be shown to anyone other than the person himself or herself, and in a system that includes my number in the application form, it is necessary to reduce the above risks as much as possible. It is valid.

In step S1507, the information processing apparatus 101 accepts a user (person in charge who handles the applicant) operation and makes a determination. In the case of the cancel button 1803, the process proceeds to END in FIG. 14 and this process ends. In the case of the continue button 1804, the process proceeds to step S1508.

In step S1508, the information processing apparatus 101 stores the face-to-face image 1802 captured in step S1504 in the external memory 212 in association with the application form associated with the application form button 1701 (FIG. 17), and in step S417 of FIG. Proceed to. That is, step S1508 is a step showing an example of a process of storing the captured document image data and the specified face image in association with each other when it is determined that the images are not images of the same person.

Note that in the present embodiment, the face-to-face image 1802 is stored in the external memory 212 of the information processing apparatus 101, but the present invention is not limited to this. It may be stored in the memory 212.

As a result, even if the information processing system 100 determines that the person who applied for the application is not the same person, if the person in charge determines that the person is the same person, the content of the application can be viewed by the other person. By storing the face image of the face-to-face person in association with the application form, the image of the person who browsed the content of the application form can be stored.

As described above, the recording medium recording the program that realizes the functions of the above-described embodiments is supplied to the system or apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by reading and executing.

In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium recording the program constitutes the present invention.

As a recording medium for supplying the program, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, non-volatile memory card, ROM, EEPROM, silicon. A disk or the like can be used.

Further, not only the functions of the above-described embodiments are realized by executing the program read by the computer, but also the OS (operating system) or the like running on the computer is actually executed based on the instructions of the program. It goes without saying that a case where a part or all of the processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Furthermore, after the program read from the recording medium is written in the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the function expansion board is instructed based on the instruction of the program code. Needless to say, this also includes the case where the CPU or the like included in the function expansion unit performs some or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

Further, the present invention may be applied to a system including a plurality of devices or an apparatus including a single device. Further, it goes without saying that the present invention can be applied to the case where it is achieved by supplying a program to a system or an apparatus. In this case, by reading the recording medium storing the program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

The form of the program may be an object code, a program code executed by an interpreter, script data supplied to an OS (operating system), or the like.

Furthermore, by downloading and reading a program for achieving the present invention from a server, a database or the like on a network using a communication program, the system or apparatus can enjoy the effects of the present invention. It should be noted that the present invention also includes all configurations that combine the above-described embodiments and modifications thereof.

100 Information Processing System 101 Information Processing Device 102 Server 103 Network

Claims (12)

An information processing system for managing the image data of the captured document using an information processing apparatus, and image data identification that includes the applicant's face image of the document,
Storage means for storing collected incorporated document image data and the identification image data and the association by using a pre-Symbol information processing apparatus,
Face image specifying means for specifying a face image from the identification card image data stored in the storage means;
When to verify the document image data, the document image data stored in the storage unit, the a person the document image data and the face image identified from rather with corresponding identification image data by the face image identification unit both the information processing system characterized in that it comprises a Viewing control means for display control on the screen of the information processing apparatus. The Display control unit,
A link for displaying the document image data stored in the storage means or the document image data, and a link for displaying the identification image data stored in the storage means in association with the document image data The information processing system according to claim 1, wherein a screen associated with is displayed. The Display control unit,
Displaying that the document image data or the ID image data has been displayed, or that an input indicating that the user has confirmed the document image data or ID image data has been accepted is displayed separately from other images. The information processing system according to claim 1, wherein: Identification information for identifying the type of the identification card, and characteristic information storage means for storing the characteristic information of the identification card in association with each other,
Further comprising: an identification certificate specifying unit for specifying the type of the identification certificate using the characteristic information stored by the characteristic information storage unit,
The face image identification means is
The information processing system according to any one of claims 1 to 3, wherein the face image is specified according to the type of the identification card specified by the identification card specifying unit. An applicant identifying unit that identifies an applicant of the document from the document image data stored in the storage unit;
From the identification certificate image data stored in the storage means, further comprising an owner identification means for identifying the owner of the identification card,
The storage means is
When the applicant specified by the applicant specifying unit and the owner specified by the owner specifying unit match, the document image and the identification card image or the face image specified by the face image specifying unit The information processing system according to any one of claims 1 to 4, further comprising: The Display control unit,
The information processing system according to claim 5, wherein if the applicant specified by the applicant specifying unit does not match the owner specified by the owner specifying unit, an error is displayed. .. Imaging means for imaging the face of the applicant of the document,
The same person determination unit that determines whether the face image identified by the face image identification unit and the face image captured by the image capturing unit are images of the same person;
When it is determined by the same person determination unit that the image is the same person, the information processing apparatus is controlled to display the captured document image data, and the same person determination unit determines that the same person image is displayed. 7. If it is determined that there is no document image, the display control means for controlling not to display the document image data taken in by using the information processing apparatus is further provided. Information processing system described. The storage means is
When it is determined by the same person determination unit that the images are not images of the same person, the document image data captured by using the information processing apparatus and the face image identified by the face image identification unit are stored in association with each other. The information processing system according to claim 7, wherein: Document definition information storage means for storing document definition information including a recognition area of the entered content of the document and a recognition condition;
Insufficiency checking means for checking insufficiency for each recognition area of the document image captured by using the information processing device according to the recognition area and recognition conditions stored in the document definition information storage means,
A deficiency area display means for displaying a deficiency of the document checked by the deficiency checking means in a identifiable manner by a user in accordance with a recognition area stored in the document definition information storage means. The information processing system according to any one of 1 to 8. 10. The information processing system according to claim 9, further comprising defect number display means for totaling and displaying the number of defects checked by the defect check means for each document form. And the image data of the captured document using an information processing apparatus, a processing method in an information processing system for managing the image data identification that includes the applicant's face image of the document,
The information processing system,
A registration step of registering in association with the previous SL document image data and the identification image data captured using an information processing apparatus,
A face image identification step of identifying a face image from the identification card image data registered in the registration step;
When to verify the document image data, the document image data registered in the registration step, and a face image specified from this the document image data and rather with corresponding identification image data by the face image specifying step processing method characterized in that together run the Viewing control step of controlling display on the screen of the information processing apparatus. And the image data of the captured document using an information processing apparatus, an executable program in an information processing system which manages the image data identification that includes the applicant's face image of the document,
The information processing system,
Storage means for storing collected incorporated document image data and the identification image data and the association by using a pre-Symbol information processing apparatus,
Face image specifying means for specifying a face image from the identification card image data stored in the storage means;
When to verify the document image data, the document image data stored in the storage unit, the a person the document image data and the face image identified from rather with corresponding identification image data by the face image identification unit a program characterized by causing both function as Viewing control means for display control on the screen of the information processing apparatus.

Images