JP2017120565A - Portable terminal and processing method and program thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a mechanism for facilitating photography at a portable terminal, and for achieving efficiency of an entry check using an image.SOLUTION: A portable terminal 101 including an imaging part comprises: a photography determination part 301 for determining whether or not a photography condition matches from an image corresponding to a document obtained by an imaging part; an automatic photography execution part 302 for, when it is determined that the photography condition matches by the photography determination part 301, executing automatic photography; a switching part 303 for, when it is determined that the photography condition does not match by the photography determination part 301, switching the automatic photography to manual photography; a check part 304 for performing an entry check by the portable terminal by using an image obtained by photography executed by the automatic photography execution part 302; and a transmission part 305 for transmitting the image such that the image obtained by the manual photography switched by the switching part 303 is checked by a server.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a portable terminal that performs entry check using a photographed image, a processing method thereof, and a program.

  Conventionally, there is a mechanism in which form data is transmitted from a user terminal to a form server, the form server checks whether there are any deficiencies in the entered contents, and returns a check result to the user terminal.

  Patent Document 1 discloses an electronic processing system that performs all processing such as determination of contents of form data transmitted from a user terminal, error correction by an operator's operation, and creation of a processing result with a correction result message on the form server side. The technology is described.

JP-A-6-28275

  However, in the system configuration of Patent Document 1, in order to perform a deficiency check, it is necessary to always send form data to the form server, and a deficiency check cannot be performed in an environment where communication with the form server is difficult. In addition, there is a problem that even a minor deficiency check such as omission of information causes communication between the user terminal and the form server, and it takes time to complete a deficient form.

Therefore, it is conceivable to check for deficiencies in the user terminal (for example, a portable terminal such as a tablet terminal), but when a form is photographed with a portable terminal, unlike a scanner or the like, Alternatively, there is a problem that it is difficult to take an image with high accuracy (photographing with less camera shake and less distortion).
For this reason, it is required to perform shooting when an appropriate state is obtained, for example, by displaying a guide for shooting.

  An appropriate state is, for example, the case where an image is obtained according to a guide. However, changing the angle of the mobile device or moving the mobile device away from or close to the form makes it an appropriate state. In some cases, it was necessary to get used to the operation. In particular, in order to simplify the operation, since it is desired to automatically shoot when the conditions are met, if the appropriate state is not obtained, shooting is not performed, and usability may be deteriorated.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a mechanism that facilitates photographing with a mobile terminal and makes an entry check using an image efficient.

  In order to achieve the object of the present invention, there is provided a mobile terminal including an imaging unit, and a shooting determination unit that determines whether a shooting condition is met from an image corresponding to a document obtained from the imaging unit, An automatic shooting execution unit that executes automatic shooting when the shooting determination unit matches the shooting conditions, a switching unit that switches to manual shooting when the shooting determination unit does not match the shooting conditions, and the automatic shooting execution In order to check the image obtained by the photographing of the manual photographing switched by the switching means, the checking means for performing the entry check on the portable terminal using the image obtained by the photographing performed by the means, A portable terminal comprising: a transmission unit that transmits the image.

  In addition, the portable terminal includes a first imaging unit that captures a document and a second imaging unit on the display unit side that displays an image obtained by the first imaging unit. A shooting determination unit that determines whether or not the shooting condition is matched from an image corresponding to the obtained document, and an automatic shooting execution unit that executes automatic shooting when the shooting determination unit matches the shooting condition; Face recognition means for performing face recognition using an image obtained from the second imaging unit, and the automatic photographing execution means is a case where the photographing determination means does not match photographing conditions, and the face recognition means When the face is recognized by the above, automatic shooting is executed.

  ADVANTAGE OF THE INVENTION According to this invention, imaging | photography with a portable terminal can be made easy and the entry check using an image can be made efficient.

The figure which shows an example of a structure of the form system 100 of this invention. The figure which shows an example of the hardware constitutions of the information processing apparatus 101 Functional block diagram showing an example of the information processing apparatus 101 and the server 102 The flowchart which shows an example of the whole process which acquires the check result of an incomplete entry in the information processing apparatus 101, and transmits to the server 102 The flowchart which shows an example of imaging | photography process with the information processing apparatus 101 The flowchart which shows an example of the deficiency check process performed with the information processing apparatus 101 The flowchart which shows an example of the process which receives the picked-up image of an application form, an accompanying image, an incompleteness check flag, a login user name, and a form name in the server 102 The flowchart which shows an example of the process which displays the application form list received with the server 102 The flowchart which shows an example of the process which adds the image incidental to the picked-up image of an application form in the information processing apparatus 101. The figure which shows an example of the deficiency confirmation cover sheet | seat which covers an application form in order to confirm the check of the deficiency of an application form in the information processing apparatus 101 clearly visually. The figure which shows an example of the screen which puts and displays a deficiency confirmation cover sheet on an application form photography image in order to make an entry deficiency check easy visually by the information processing apparatus 101 The figure which shows an example of the screen which displays a deficiency check result in the information processing apparatus 101 The figure which shows an example of the table which manages the form form and deficiency confirmation cover sheet in the server 102 and the information processing apparatus 101 The figure which shows an example of the table which manages the transmission time of the application form which the server 102 received, a sender, form ID, and a deficiency check flag The figure which shows an example of the screen which displays the table managed in FIG. The figure which shows an example of the screen which displays the application received by the server 102 The figure which shows an example of the table which manages the deficiency check recognition area and deficiency check method for every form The figure which shows an example of the screen which performs the confirmation display of the image of the application form transmitted with the information processing apparatus 101 The figure which shows an example of the screen which performs the confirmation display of the incidental image of the application form transmitted with the information processing apparatus 101 The figure which shows an example of the screen which performs the information setting preset in the information processing apparatus 101 The figure which shows an example of the hardware constitutions of the server 102 The image figure which shows an example of the state at the time of imaging | photography with the information processing apparatus 101 An image diagram showing an example of a manual photographing switching state during photographing by the information processing apparatus 101 The figure which shows an example of the setting (mark image etc.) used for photography conditions

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
Note that the characteristic processing of the present embodiment will be described with reference to the following flowchart, and among them, the particularly characteristic processing is FIG. 4B.

  As shown in FIG. 1, a form system 100 (information processing system) according to this embodiment has a configuration in which an information processing apparatus 101 (mobile terminal) and a server 102 are communicably connected via a network 103. The information processing apparatus 101 captures an image of a form using a camera or the like and performs a check for incomplete entry of the form. The server 102 manages the recognition area and the recognition condition of the entry contents necessary for checking the incompleteness of the form for each form form identification information. The information processing apparatus 101 is a portable terminal such as a mobile phone, a smart phone, a tablet terminal, and a PDA terminal. A browser or a form system application is installed on the information processing apparatus 101. Communication is possible. Needless to say, the server 102 may be a general-purpose computer.

Hereinafter, an example of a hardware configuration applicable to the information processing apparatus 101 illustrated in FIG. 1 will be described with reference to FIG.
In FIG. 2, reference numeral 201 denotes a CPU that comprehensively controls each device and controller connected to the system bus 204. Further, the ROM 203 or the external memory 212 is necessary to realize a BIOS (Basic Input / Output System) or an operating system program (hereinafter referred to as an OS), which is a control program of the CPU 201, or a function executed by each server or each PC. Various programs are stored.

  A RAM 202 functions as a main memory, work area, and the like for the CPU 201. The CPU 201 implements various operations by loading a program or the like necessary for execution of processing from the ROM 203 or the external memory 212 to the RAM 202 and executing the loaded program.

  An input controller 205 controls input from the touch sensor 210. A video controller 206 controls display on a display device such as the display 211. In FIG. 2, the display 211 is described, but the display may be not only a CRT but also other display such as a liquid crystal display. These are used by the user of the information processing apparatus 101 as necessary.

  A memory controller 207 is provided in an external storage device (hard disk (HD)), a flexible disk (FD), or a PCMCIA card slot for storing a boot program, various applications, font data, user files, editing files, various data, and the like. Controls access to an external memory 212 such as a compact flash (registered trademark) memory connected via an adapter.

  A communication I / F controller 208 is connected to and communicates with an external device via the network 103, and executes communication control processing on the network. For example, communication using TCP / IP is possible.

  Note that the CPU 201 enables display on the display 211 by, for example, executing outline font rasterization processing on a display information area in the RAM 202. In addition, the CPU 201 can accept a user's touch operation instruction on the touch sensor 210 on the display 211.

  Various programs to be described later for realizing the present invention are recorded in the external memory 212, and are executed by the CPU 201 by being loaded into the RAM 202 as necessary. Further, definition files, map data, and the like used when executing the program are also stored in the external memory 212, and detailed descriptions thereof will be described later.

A captured image input controller 209 controls input from the camera 215 via the captured image input processing unit 214 (corresponding to a capturing unit). A captured image input from the camera 215 can be recognized. One camera 215 is provided on the display side that is the front surface and one on the opposite side of the display that is the back surface. The back camera 215 shoots a form such as an application form, and the front camera 215 recognizes the face of the photographer. Also, there is a captured image input processing unit 214 corresponding to each camera 215, and receives an input of an image of each camera. In addition, when performing shooting automatically or manually, the entry deficiency check application according to the present embodiment requests the operating system to issue an instruction (shutter ON) to perform shooting of the rear camera. Specifically, since it has identification information of the camera 1 (rear surface) and the camera 2 (front surface), a request including the identification information of the camera that releases the shutter is performed, so that the shutter can be given to any camera. Can be cut.
Next, the hardware configuration of the information processing apparatus applicable to the server 102 shown in FIG. 1 will be described with reference to FIG.

  In FIG. 20, reference numeral 2001 denotes a CPU that comprehensively controls each device and controller connected to the system bus 2004. Further, the ROM 2002 or the external memory 2011 is necessary for realizing a BIOS (Basic Input / Output System) or an operating system program (hereinafter referred to as an OS), which is a control program of the CPU 2001, or a function executed by each server or each PC. Various programs to be described later are stored.

  Reference numeral 2003 denotes a RAM that functions as a main memory, work area, and the like of the CPU 2001. The CPU 2001 implements various operations by loading a program or the like necessary for execution of processing from the ROM 2002 or the external memory 2011 into the RAM 2003 and executing the loaded program.

  An input controller 2005 controls input from a keyboard (KB) 2009 or a pointing device such as a mouse (not shown). A video controller 2006 controls display on a display device such as a CRT display (CRT) 2010. In FIG. 20, although described as CRT2010, the display device may be not only the CRT but also other display devices such as a liquid crystal display. These are used by the administrator as needed.

  A memory controller 2007 is connected to the hard disk (HD), flexible disk (FD), or PCMCIA card slot for storing a boot program, various applications, font data, user files, editing files, various data, etc. via an adapter. Controls access to an external memory 2011 such as a compact flash (registered trademark) memory and database.

  A communication I / F controller 2008 is connected to and communicates with an external device via a network (for example, 103 shown in FIG. 1), and executes communication control processing in the network. For example, communication using TCP / IP is possible.

  Note that the CPU 2001 enables display on the CRT 2010 by executing outline font rasterization processing on a display information area in the RAM 2003, for example. Further, the CPU 2001 enables a user instruction with a mouse cursor (not shown) on the CRT 2010.

  Various programs to be described later for realizing the present invention are recorded in the external memory 2011, and are executed by the CPU 2001 by being loaded into the RAM 2003 as necessary. Furthermore, definition files and various information tables used when executing the program are also stored in the external memory 2011.

  FIG. 3 is a block diagram illustrating an example of a functional configuration of the information processing apparatus 101 (mobile terminal). The information processing apparatus 101 will be described as a mobile terminal including an imaging unit. Or it demonstrates as a portable terminal provided with the 1st imaging part which image | photographs a document, and the 2nd imaging part by the side of the display part which displays the image obtained by a 1st imaging part.

  The shooting determination unit 301 is a functional unit that determines whether or not the shooting conditions are matched from an image corresponding to the document obtained from the imaging unit. The shooting condition is a shooting condition for determining whether or not a plurality of predetermined locations in the document image match a pre-registered image. The photographing condition is a condition as to whether or not the locations related to the four corners of the document image match. The photographing determination unit 301 is a functional unit that determines whether the image matches a previously registered image.

The automatic shooting execution unit 302 is a functional unit that executes automatic shooting when the shooting conditions match. In addition, it is a functional unit that executes automatic shooting when it matches an image registered in advance. In addition, the function unit performs automatic shooting when a location related to the four corners matches a pre-registered image, or when a location related to the three corners matches a pre-registered image. Furthermore, this is a functional unit that performs automatic shooting when a portion related to the three corners matches a pre-registered image and a face is recognized by the face recognition unit 308.
The switching unit 303 is a functional unit that switches to manual shooting when the shooting conditions do not match.
The check unit 304 is a functional unit that performs an entry check on the portable terminal using an image obtained by the executed automatic photographing.
The transmission unit 305 is a functional unit that transmits the image so that the server can check the image obtained by the switched manual shooting.
The specifying unit 306 is a functional unit that specifies the point of the location related to the remaining corner when the location related to the three corners of the document image matches a pre-registered image.

The correction unit 307 is obtained by automatic shooting using points obtained according to the locations related to the four corners when automatic shooting is performed with the locations related to the four corners matching the pre-registered images. When the image is corrected and automatic shooting is performed with the location related to the three corners matching the pre-registered image, the points obtained according to the locations related to the three corners and the points specified by the specifying unit 306 are It is a functional unit that uses and corrects an image obtained by automatic photographing.
The face recognition unit 308 is a functional unit that performs face recognition using an image obtained from the second imaging unit.

  FIG. 4A is a flowchart illustrating an example of an overall process (hereinafter referred to as an entry deficiency check result acquisition / transmission process) in which the information processing apparatus 101 acquires an entry deficiency check result and transmits it to the server 102.

  FIG. 19 is an example of a screen for performing information setting set in advance in the information processing apparatus 101. In the initial setting screen 1901, when a store name, a person in charge, a telephone number are input and an OK button is pressed, information is stored in the information processing apparatus 101. As in the initial setting screen 1902, when the form of the application form to be photographed is determined in advance, the form form definition information for character recognition and the form ID for specifying the cover sheet are input. May be. This initial setting is set by the deficiency check app.

  In step S401, the information processing apparatus 101 starts an entry deficiency check application and logs in. When the deficiency check application is activated, an initial setting button (not shown) is pressed to display the above initial setting screen, and the initial setting is appropriately executed by a user operation.

  In step S402, when it is detected that an application form photographing button (not shown) is pressed, the camera 215 is activated to perform photographing of the application form for checking the incompleteness of entry. Here, four corners of a form such as an application form are detected, and photographing is performed when the form is in focus. With this execution, a captured image corresponding to the application form is acquired. The photographing by the four corner detection will be described with reference to FIG. 4B. This 4B is a characteristic process in this embodiment.

  Here, FIG. 4B is a flowchart illustrating an example of the photographing process. Note that each step is executed by the CPU 201.

  In step S430, the front side (display side) camera 215 (first imaging unit) and the back side (opposite side of the display) camera 215 (second imaging unit) are activated. The CPU 201 displays the received image on the display 211 via the captured image input controller 209 for the image of the rear camera 215. Alternatively, the CPU 201 accumulates the image of the front camera 215 in the RAM 202 via the captured image input controller 209. The image from the front camera 215 is deleted from the memory when the next image is received. Face recognition is performed using the accumulated image.

Here, it is displayed on the display 211 as 2100 in FIG. A guide is displayed so that a photographer can easily photograph an application form as a subject, and a mark registered in FIG. By aligning the mark of the subject with this mark, it becomes easier to perform automatic shooting. Reference numeral 2102 denotes an image acquired from the rear camera 215. This state is a state where a photographed image has not been recorded yet. This state is also referred to as a live view, and is a technique for displaying an image to be captured using an image obtained by an image sensor (CCD).
In step S 431, the counter value for specifying the number of detected marks is set to 0 and stored in the RAM 202.

  In step S432, it is determined whether or not the subject (in this embodiment, a form such as an application form) is in focus using the autofocus technique. If the focus is achieved, the process proceeds to step S433, and the process is repeated until the focus is achieved. Note that the detection of the focus is made by receiving an entry deficiency check application from the operating system.

In step S433, image recognition for the corner A is performed, and it is determined whether or not there is a predetermined mark in the corner A. Specifically, using the recognition area (predetermined range of coordinates) registered in the setting information 2300 in FIG. 23, the image of the corner A (upper left) of the image captured from the rear camera 215, and the setting information 2300 The image 2301 (mark: black circle) registered as the value of is compared. The image recognition technique is realized by using a known technique. For example, it is realized by using pattern matching between images.
If a mark can be detected at the corner A, the process proceeds to step S434. If no mark is detected at the corner A, the process proceeds to step S435.
In step S434, since a mark is detected at the corner A, the counter is incremented by 1 and stored in the RAM 202.

In step S435, image recognition for the corner B is performed, and it is determined whether or not there is a predetermined mark in the corner B. Specifically, using the recognition area (predetermined range of coordinates) registered in the setting information 2300 in FIG. 23, the image at the corner B (upper right) of the image captured from the rear camera 215, and the setting information 2300 Is compared with the image 2302 (mark: black triangle) registered as the value of. The image recognition technique is realized by using a known technique. For example, it is realized by using pattern matching between images.
If a mark can be detected at corner B, the process proceeds to step S436. If no mark is detected at the corner B, the process proceeds to step S437.
In step S436, since a mark is detected at the corner B, the counter is incremented by 1 and stored in the RAM 202.

In step S437, image recognition for the corner C is performed, and it is determined whether or not there is a predetermined mark in the corner C. Specifically, using the recognition area (predetermined range of coordinates) registered in the setting information 2300 in FIG. 23, the image at the corner C (lower right) of the image captured from the rear camera 215, and the setting information The image 2303 registered as the value 2300 (mark: black square) is compared. The image recognition technique is realized by using a known technique. For example, it is realized by using pattern matching between images.
If a mark can be detected at corner C, the process proceeds to step S438. If no mark can be detected at corner C, the process proceeds to step S439.
In step S438, since a mark is detected at the corner C, the counter is incremented by 1 and stored in the RAM 202.

In step S439, image recognition for the corner D is performed, and it is determined whether or not there is a predetermined mark in the corner D. Specifically, using the recognition area (predetermined range of coordinates) registered in the setting information 2300 in FIG. 23, the image of the corner D (lower left) of the image captured from the rear camera 215, and the setting information 2300 Is compared with the image 2303 (mark: black star) registered as the value of. The image recognition technique is realized by using a known technique. For example, it is realized by using pattern matching between images.
If a mark can be detected at the corner D, the process proceeds to step S440. If no mark is detected at the corner D, the process proceeds to step S441.
In step S440, since a mark is detected at the corner D, the counter is incremented by 1 and stored in the RAM 202.
Note that the mark is an example and can be recognized as long as it is different at the four corners.

  In step S441, the value of Counter stored in the RAM 202 is acquired, and it is determined whether or not the value is “4”. That is, it is determined whether or not four points (four corners) have been detected. If the value is “4”, the process proceeds to step S422. If the value is not “4”, the process proceeds to step S433.

  In step S442, automatic shooting is executed. This automatic photographing is executed using an auto shutter technique. A photographed image obtained by this automatic photographing is stored in an internal memory. 2110 in FIG. 21 is an image when the automatic shooting is performed.

  Note that a flag indicating that the image is a captured image obtained by automatic shooting is assigned and managed. Further, the captured image is corrected based on the coordinates of the four points obtained at the time of shooting. For example, keystone correction is performed. In addition, the correction of the application form itself is corrected using a known correction technique. The corrected image is used for subsequent deficiency check processing. In addition, when a later-described three corner can be recognized and the fourth point cannot be obtained, the image is corrected using the estimated fourth point and the three points by the three corners. That is, when automatic shooting is performed with the locations related to the four corners matching the pre-registered images, the points obtained according to the locations related to the front four corners are used to obtain the images obtained by automatic shooting. When correction is performed and a point relating to the three corners is automatically photographed in accordance with a pre-registered image, a point obtained according to the point relating to the three corners and a point identified (estimated fourth point) are used. This is a step showing an example of a process for correcting an image obtained by automatic photographing.

  In step S443, it is determined whether it has been repeated for a predetermined time or a predetermined number of times. If step S431 to step S441 are repeated for a predetermined time or a predetermined number of times, the image recognition of four points is not successful, and the process is moved to step S444. If the predetermined time or the predetermined number of times has not been reached, the process proceeds to step S431. Note that the process of step S443 may be executed when NO in step S444. That is, the determination as to whether the predetermined time or the predetermined number of times has been reached may be performed at any timing.

  In step S444, it is determined whether or not the acquired counter value is “3”. That is, it is determined whether or not the fourth point (four corners) can be detected, but the third to fourth points can be estimated. When using a tablet terminal to shoot a form such as an application placed on a table with the rear camera, the guide may not fit well depending on the shooting direction (angle of the tablet with respect to the form) and the distance between the form and the tablet terminal. Occur. This is a problem that can actually occur. If the photographer is not familiar with photographing, the image portion of the form is distorted, for example, trapezoidal, and does not match the guide well. In that case, since it is necessary to correct the image, it is necessary to specify whether the four points can be estimated. In particular, in order to automatically check entry deficiencies or use the deficiency confirmation cover sheet shown in FIG. 9, image correction is essential, and it is important whether four points can be estimated. If four points can be estimated, the automatic entry defect check or the defect check using the defect confirmation cover sheet can be performed, so the processing shifts to automatic photographing.

  That is, if the value of Counter is “3”, the process proceeds to step S425. If the value of Counter is not “3”, ie, “0”, “1”, or “2”, the process proceeds to step S447.

  Step S432 to step S441, step S443, and step S444 are steps showing an example of processing for determining whether or not the shooting condition is matched from the image corresponding to the document obtained from the imaging unit. The imaging conditions are imaging conditions for determining whether or not a plurality of predetermined locations (for example, four corners) in the document image match with previously registered images (2301 to 2304 in FIG. 23).

  Step S442 is a step showing an example of processing for executing automatic shooting when the shooting conditions are met. Moreover, it is a step which shows an example of the process which performs automatic imaging | photography when it corresponds with the image registered beforehand. Further, it is a step showing an example of processing for executing automatic photographing when a location related to the four corners matches an image registered in advance or when a location related to the three corners matches an image registered in advance.

  In step S445, it is determined whether or not there is face recognition in the image acquired from the front camera 215. That is, it is a step showing an example of processing for performing face recognition using an image obtained from the second imaging unit.

  Since the face recognition technique is a known technique, a description thereof will be omitted, but the area for detecting face recognition is set at the center of the video according to the setting of 2305. As shown in FIG. 23, the image 2305 is an area inside the imaging area. Here, face recognition is detected by an actual photographing operation, and when the automatic photographing is not performed because the guide does not fit well (automatic photographing is used for basic photographing), the photographer photographs while viewing the display from the front. By recognizing the photographer's face, it can be determined that the photographer is unable to photograph well. If you ca n’t recognize your face (it may be a side profile), you ’re not looking at the display and you ’re not looking at it. Move to manual shooting.

  If it is determined that face recognition has been performed in the set area, the process proceeds to step S445. If it is determined that there is no face recognition in the set area, the process proceeds to step S447. That is, this is a step showing an example of a process for shifting to a process of executing automatic shooting (step S442) when a location related to the three corners matches a pre-registered image and a face is recognized.

  In step S446, the fourth point is specified based on the recognized three points (three corners). For example, when the coordinates of the corner A, corner B, and corner C points can be specified, a vector 1 that connects the corner A and the corner B is generated. Also, a vector 2 connecting the corner B and the corner C is generated. Vector 1 is applied to corner C (vector 1C), vector 2 is applied to corner A (vector 2A), and the point where vector 1C and vector 2A intersect is the fourth point.

  In other words, this is a step showing an example of processing for specifying the points of the remaining corners when the positions of the three corners of the document image coincide with the pre-registered images in step S444.

  In step S447, the automatic shooting is switched to the manual shooting. This is achieved by switching the camera setting from auto to manual. When switching to this manual setting, notification is made as to whether or not to switch to manual shooting as indicated by 2200 in FIG. When the manual photographing switching button is pressed, manual switching is performed. When the automatic photographing continuation button is pressed, the process proceeds to step S431, and the four corner detection is performed again. When switching to manual shooting, the process proceeds to step S448. That is, it is a step showing an example of a process for switching to manual shooting when the shooting conditions are not matched.

  In step S448, it is determined whether or not a photographing instruction has been given by the photographer. Specifically, when switching to manual shooting, a shooting button is displayed as indicated by 2210 in FIG. 22, so that it is detected that the shooting button has been pressed. If a shooting instruction has been given, the process proceeds to step S449. If no shooting instruction is given, the instruction is waited.

In step S449, shooting is performed in accordance with the shooting instruction. Save the shot image in the internal memory. Since this photographed image is not photographed along the guide, it is not possible to perform an automatic entry defect check or a defect check using a defect confirmation cover sheet. Therefore, unlike an automatically shot image, the shot image is sent to the server 102 and the process proceeds to a visual check.
As a result, it is possible to facilitate shooting with a mobile terminal, and to make the entry check using images more efficient.

Returning to the flowchart of FIG.
In step S402-1, it is determined whether automatic shooting has been performed. If automatic shooting has been performed, the process proceeds to step S403. If manual shooting is performed, the process proceeds to step S402-2. In the case of automatic shooting, since shooting is performed along the guide, it is possible to automatically check for deficiencies in entry or deficiency checks using a deficiency confirmation cover sheet. Therefore, when automatic shooting has been performed, the process proceeds to step S403. Further, since it is difficult to automatically check for deficiencies in the entry or deficiency checks using the deficiency confirmation cover sheet, the process proceeds to step S402-2.

  In step S402-2, the manually captured image is transmitted to the server 102. That is, this is a step showing an example of a transmission process for transmitting an image obtained by manual photographing so as to be checked by the server.

  At this time, a flag indicating that manual shooting has been performed is also sent to the server 102 to make the checker of the server 102 recognize manual shooting and facilitate checking. On the server side, manual shooting is performed and a client terminal connected to the server side displays that the information processing apparatus 101, which is a portable terminal, is not checked for deficiencies.

  In step S403, it is determined whether there is a form that matches the captured image. Whether there is a matching form is determined by comparing the captured image and the form image using the form recognition technique in the order of data in the local form table 1200 (FIG. 12). As described above, when the form ID is set in advance, the determination may be made based on whether or not the form ID is in the form table. Further, the form may be specified by extracting the form ID from a predetermined area of the photographed image. If it is determined in step S403 that there is no matching form, the process proceeds to step S404. If it is determined that there is a matching form, the process proceeds to step S406.

  If there is no matching form table in step S404, an error message indicating that the application form photographed in step S405 is not a deficiency check correspondence form is displayed, and the application is terminated. If the form is not a corresponding form, the process may be shifted to a re-photographing process after displaying an error message. If there is a matching form table, the cover sheet (image) corresponding to the photographed application form is obtained from the external memory 212 with reference to the form and the form table (FIG. 12) in step S406. The form table 1200 and corresponding images (form image, cover sheet) are stored in the external memory 212. That is, it shows an example of a process for acquiring a second image that is transmitted so that the entry check portion that is a cover sheet corresponding to the first image obtained by photographing is visible.

  In step S407, a photographed image of the photographed application form is overlaid with a deficiency confirmation cover sheet (FIG. 9) corresponding to the photographed application form obtained in step S404, thereby generating a check image. That is, an example of processing for generating a third image for entry check using the first image and the second image is shown.

The generated image is displayed on the screen for checking the incomplete entry (FIG. 10-1). That is, an example of a display process for displaying the generated third image in order to make the user check the entry is shown.
The cover sheet may be a cover sheet that includes a description of the check location so that the user can easily check the cover sheet.

  In the cover sheet of FIG. 9, the check part is transparently processed, and when the photographed image is superimposed, it is possible to easily confirm the entry of the place where the check of the entry defect is to be performed. In addition, although parts other than the check part may be transmitted, the transmittance is made lower than the check part or the color is changed (for example, set to gray) so that the user can easily see the check part. It is desirable to make it an image.

  In step S408, in order to initialize the deficiency check process, the entry deficiency check completion flag is turned OFF, and the OK button flag / cancel button flag is turned OFF.

  In step S409, an incomplete entry check processing flow (described later in FIG. 5) is performed. This incomplete entry check process is executed in the background displaying the incomplete entry check screen shown in FIG. That is, an example of a check process for performing an entry check on the portable terminal using an image obtained by the executed photographing is shown. In addition, an example of executing recognition processing based on the first image and the definition information in the background of the display of the third image is shown.

  In step S410, an incomplete entry check flag, an OK button flag, and a cancel button flag, which are recognition results, are acquired. Whether or not the OK button flag is ON is checked in the entry defect check processing flow (FIG. 5), and if it is ON, the flow proceeds to an accompanying image addition processing flow (described later in FIG. 8) in step S422. The case where the OK button flag is ON is a case where the deficiency check is completed by performing a visual deficiency check using the overlaid image on the entry deficiency check in progress screen of FIG. 10-1. . When the process of step S422 is completed, in step S423, the captured image of the application form, the accompanying image, the form name, the login user name, and the deficiency check completion flag OFF are transmitted from the information processing apparatus 101 to the server 102, and the process ends.

  In step S410, it is confirmed whether or not the OK button flag is turned on in the incomplete entry check processing flow (FIG. 5), and if it is turned off, it is confirmed in step S411 whether the cancel button flag is turned on. If the cancel button is ON, the application is terminated.

  If the cancel button flag is OFF in step S411, a confirmation button indicating that the entry defect check has been completed and a confirmation button are displayed (recognition result information display) in step S412, and an entry defect check completion screen (FIG. 10) is displayed. -Change to 2). When the cancel button flag is OFF, the entry defect check completion flag is ON. That is, this is a case where the entry deficiency check is completed by recognition by the system instead of visual observation in the background.

  In other words, when the result recognized based on the first image and the definition information is acquired, and the result is recognized and acquired, information indicating that the result recognized together with the third image (for example, An example of processing for displaying a (confirmation button) is shown.

  In step S413, it is determined which one of the OK button, the cancel button, and the confirmation button has been pressed on the entry defect check completion screen (FIG. 10-2). When the cancel button is pressed, the application is terminated. If the OK button is pressed, the process proceeds to an accompanying image addition process flow (described later in FIG. 8) in step S422. When the process of step S422 is completed, in step S424, the information processing apparatus 101 transmits the captured image of the application form, the accompanying image, the form name, the login user name, and the deficiency check completion flag ON to the server 102, and the process ends. If there is a defect, the confirmation button is pressed to confirm, and control is performed to execute transmission. If the confirmation button is pressed, the process proceeds to step S414.

  In step S414, the information processing apparatus 101 searches the entry defect check table 1610 using the form ID of the photographed image specified in step S403 as a search key, and stores the RAM 202 in the result column 1611 of check ID = i of all the acquired records. The value of the variable result [i] stored in is registered respectively. In this embodiment, the variable result [i] = [OK, NG, OK, NG, OK] indicating the check result of form ID = 1000 in the result column 1611 of 5 records of form ID = 1000 of the entry defect check table 1610. The value of is registered.

In step S415, the information processing apparatus 101 closes the entry defect check completion screen (FIG. 10-2) from the display 211, and displays the photographed image display photographed image on the entry defect check result screen (FIG. 11). That is, when there is an instruction to display the recognized result by pressing the confirmation button, a process of switching from the third image to the recognition result image generated according to the result recognized as the first image is displayed. An example is shown.
In step S416, the information processing apparatus 101 determines whether NG exists in the variable result [i].

  If NG does not exist, in step S417, the information processing apparatus 101 displays an entry defect check result screen (no defect) 1110 on which a message “There was no entry defect” is written.

  If NG exists, in step S418, the information processing apparatus 101 displays an incomplete entry check result screen (incomplete) 1120 in which a message “There is incomplete entry” is written.

  In step S419, the information processing apparatus 101 searches the entry defect check table 1610 using the form ID = form ID of the photographed image and the result column 1611 = NG as a search key, and starts the coordinates (X, Y) of the entry defect check recognition area. ) And end point coordinates (X, Y).

  In step S420, the information processing apparatus 101 draws a rectangle with the starting point coordinates (X, Y) and the ending point coordinates (X, Y) of the acquired entry defect check recognition area as diagonal lines, and fills it with red ( 1121). At this time, the rectangle is set to be transparent. Thereby, it is possible to display an incompletely filled area in an identifiable manner.

  In step S421, it is determined which button is pressed on the entry defect check result screen. If the cancel button is pressed, the application is terminated, and if the OK button is pressed

  The incidental image addition process (described later in FIG. 8) in step S422 is performed. After step S422, in step S424, the information processing apparatus 101 transmits a captured image of the application form, an accompanying image, a form name, a login user name, and a defect check completion flag ON to the server 102, and the process ends. That is, an example of processing for outputting information related to the entry check and the first image when there is an output instruction after displaying the third image is shown. In addition, when there is an accompanying image, an example of processing for outputting the first image and the accompanying image together is shown. Needless to say, when there is no accompanying image, the accompanying image is not transmitted.

  The transmission may be performed for each application form. If communication with the server 102 is not possible, the information is stored in the information processing apparatus 101 and data relating to a plurality of application forms is obtained when communication is possible. May be configured to transmit.

FIG. 5 is a flowchart illustrating an example of an incomplete entry check process performed by the information processing apparatus 101.
FIG. 16 is an example of a table for managing a deficiency check recognition area and a deficiency check method for each form.

  The entry deficiency check table 1610 stores a form ID, a check ID assigned to the check location, recognition conditions for the entry deficiency check, start coordinates (X, Y) and end coordinates (X, Y) of the entry deficiency check recognition area. Yes. In other words, this is an example of storing definition information including a recognition area and recognition conditions for recognizing entry contents from the first image.

  The “entry deficiency check table 1610” has a result column 1611 for storing the entry deficiency check processing result, but the result column 1611 is not yet used in the entry deficiency check processing of FIG. null.

In step S501, the entry defect check table 1610 is searched using the form ID of the photographed image specified in step S404 as a search key, and an entry defect check record is acquired.
In step S <b> 502, 1 is substituted into the variable i and stored in the RAM 202.

In step S503, the starting point (X coordinate, Y coordinate) and end point (X coordinate, Y coordinate) and the filling defect check recognition condition of the filling defect check recognition area are acquired from the i-th of the filling defect check record.
In step S504, the value of the entry defect check recognition condition is determined.

If the entry deficiency check recognition condition is “essential”, it is an entry indispensable item. Therefore, in step S505, it is determined whether characters or symbols are entered in the check recognition area of the photographed image. This determination process uses an analysis technique such as OCR, but since it is a known technique, a description thereof will be omitted.
If a character or symbol is entered, “OK” is substituted into variable result [i] for storing the check result in the RAM 202 in step S506.
If no character or symbol is entered, “NG” is substituted into variable result [i] in step S507.

  If the entry defect check recognition condition is “circle”, since it is an item that circles one of a plurality of options, whether or not a circle is entered in the check recognition area of the photographed image in step S508. Determine whether. This determination process uses an analysis technique such as OCR, but since it is a known technique, a description thereof will be omitted. In the present embodiment, the presence / absence of a circle is determined. However, the present invention is not limited to the circle, and a check of a check point or other figure may be used.

  In the above description, the number of circles is set to “1”, but an “upper limit value” or “check number” column is provided in the incomplete entry check table 1610, and “NG” is exceeded when the upper limit value is exceeded. May be determined as “NG” when the number is not “the number of checks”. In addition, a “digits” column is provided in the incomplete entry check table 1610, and “OK” is determined if the entered characters are permitted digits, and “NG” is determined if they are not permitted. Good. In addition, a “character attribute” column may be provided in the entry defect check table 1610, and OK / NG determination may be made based on the attribute of the entered character, for example, “number”, “hiragana”, “katakana”, “alphanumeric”. .

Further, when the entered character cannot be recognized as a result of analysis by OCR or the like, it may be determined as “NG”. Further, the value of the variable result [i] is not limited to two, “OK” and “NG”, and may have values such as “unclear”, “unrecognizable”, and “contradiction”.
If a circle is entered, “OK” is substituted into variable result [i] in step S509.
If no circle is entered, “NG” is substituted into variable result [i] in step S510.

  In step S511, the variable i is incremented to check for deficiencies in the next area. In step S512, it is determined whether or not an incomplete check record with check ID = i exists. If it does not exist, the entry deficiency check completion flag is turned ON in step S517, and this entry deficiency check process is terminated. If it exists, in step S513, it is determined whether or not the OK button has been pressed on the screen for checking the entry defect (FIG. 10-1). If the OK button has been pressed, the OK button flag is turned ON in step S515, Returning to (1) of FIG. 4A. If the OK button has not been pressed, the process proceeds to step S514. Since the user presses the OK button or the cancel button at an arbitrary timing, the determination in step S513 and step S514 is performed at the timing when the user presses the OK button or the cancel button.

In step S514, it is determined whether or not the cancel button has been pressed on the incomplete entry check screen (FIG. 10-1). If the cancel button has been pressed, the cancel button flag is turned ON in step S516, and ( Return to 1). If the cancel button has not been pressed, the process returns to step S503 to continue checking for deficiencies. The case where the cancel button is pressed is a case where the deficiency check is terminated.

  The description of the entry defect check process in FIG. By this processing, the check result for the incomplete check record corresponding to the form ID of the photographed image is stored in the variable result [i]. Specifically, in this embodiment, the value “OK, NG, OK, NG, OK” is stored in the variable result [i].

FIG. 6 is an example of a flow in which the server 102 receives the image and attribute information transmitted from the information processing apparatus 101 in step S424.
In step S601, the server 102 receives an application form image, an accompanying image, an incompleteness check completion flag (OFF, ON), a login user name, and a form name.

  In step S602, the received images (images of application forms that have been checked for incompleteness) and incidental images are stored in a predetermined folder, and the reception date / time, form name, login user name, Add a record of check entry completion flag (OFF / ON). As a result, information that has been checked for deficiencies in the information processing apparatus 101 is registered. After that, an operator on the server 102 side performs a detailed check, and if there is any deficiency, calls the person in charge corresponding to the login user name. For example, the telephone number used in the header of the received image is used for the telephone.

  FIG. 7 shows an example of a flow in which the server 102 displays the image and attribute information received from the information processing apparatus 101, and the operator confirms that there are no omissions.

  In step S701, the reception application list table (FIG. 13) is read and the reception application list screen (FIG. 14) is displayed. At this time, when the server 102 receives more than a predetermined number of application forms within a predetermined time (for example, when 20 or more application images are received within 5 minutes), the application system lacks. It is decided to prioritize and display the application forms that are not checked, and the reception application list screen 1420 is displayed. If a predetermined number of application forms are received within a predetermined time, a reception application list screen 1410 is displayed. In the mode in which application forms without defect checks are preferentially displayed, the server 102 generates a list sorted by defect check “incomplete” and displays it on the received application list screen.

  In step S702, it is determined whether the end button has been clicked on the received application form list screen (FIG. 14), and if the end button is pressed, the application confirmation process is terminated. If the end button has not been clicked, it is determined in step S703 whether the form name has been clicked.

  In step S703, it is determined whether the form name is clicked. If not clicked, the process returns to step S702. If the form name is clicked, the application form received in step S704 is displayed on the reception application form confirmation screen (FIG. 15), and the operator on the server side visually checks for incompleteness.

At this time, if an application for deficiency check completion is clicked on the received application form list screen (Fig. 14), a reception application confirmation screen (completed deficiency checked) 1510 is displayed, and an application for which deficiency check has not been completed is clicked If so, a reception application confirmation screen (incomplete check check not completed) 1511 is displayed. Therefore, the server-side operator can easily confirm whether the application form has already been checked for deficiencies in the system.
In step S705, the operator on the server side determines whether there is an incomplete place in the application form.

If there is any incomplete entry, the sender (login user) indicated in the header of the image of the application form is notified of the incomplete entry place, and the application form is photographed and retransmitted after entry. To do. If there is no incomplete entry, the sender (login user) described in the header of the application form is notified of no incomplete entry, and the process proceeds to Step 708.

  In addition, since the telephone number is described as an example in the header, in this embodiment, it is assumed that the sender is contacted by telephone. However, the header includes an email address, a URL of the system, etc. The communication means may be described. In addition, when a deficiency contact button (not shown) is pressed on the reception application confirmation screen shown in FIG. 15, a window (a screen including a text box or the like) in which deficiency contents can be entered is displayed. It is also possible to adopt a configuration for transmitting to the information processing apparatus 101 of the person in charge).

  When the completion button is pressed on the reception application confirmation screen (FIG. 15) in step S708 and the screen is closed, the record of the application displayed in step 704 is deleted from the reception application list table (FIG. 13) in step S709. In addition, you may manage by setting a completion flag as confirmed, without deleting.

FIG. 8 is an example of a processing flow for adding an incidental image to the captured image of the application form in the information processing apparatus 101.
In step S801, a transmission image confirmation screen (FIG. 17) is displayed, and the incidental image number and the number of incidental images are set to zero.

In step S802, the dealer name, person in charge name, person in charge telephone number, number of incidental images set on the initial setting screen (FIG. 19), entry defect check completion status based on the result column of the entry defect check table 1610, etc. (Referred to as information). Based on this acquired information, it is displayed as a header image added to the photographed image of the application form. Specifically, based on the acquired dealer name, person-in-charge name, person-in-charge telephone number, number of incidental images, and completion status of entry deficiency check, it is imaged in a predetermined size and inserted at the top of the application form image. That is, an example of a process for displaying the number of auxiliary images of the first image obtained by photographing together with the first image is shown. In addition, an example of processing for acquiring header information including checker information related to entry check and generating a first image including header information is shown.
An image including this header information is transmitted to the server 102 as an image for which the incompleteness check has been performed.

  In addition, the information added to a header can add arbitrary items, and is not limited to said information. Further, although a telephone number is described as an example of means for contacting the sender from the operator of the server 102, another means for contacting the sender such as an e-mail address or a system URL is described in the header. It doesn't matter.

  In step S803, it is determined which button is pressed on the transmission image confirmation screen (FIG. 17). If the cancel button is pressed, the process returns to (2) in FIG. 4A, and the application form is re-photographed in step S402. If the transmission button is pressed and there is an accompanying image, the accompanying image number / number of accompanying image number information is added to the header of each accompanying image, and this processing flow ends.

If the accompanying document image photographing button is pressed, an accompanying image such as a license necessary for the application form is photographed in step S804. The captured incidental image is stored in the memory. That is, an example of a process for storing an incidental image obtained by photographing when an instruction for photographing an incidental image is given by the user.
In step S805, the incidental image captured in step S804 is displayed on the incidental image confirmation screen (FIG. 18), and the number of incidental images is incremented by one.
In step S806, the number of accompanying images is entered in the accompanying image number of the photographed accompanying image.
In step S807, it is determined which button has been pressed on the accompanying image confirmation screen (FIG. 18). If the OK button is pressed, the screen returns to the transmission image confirmation screen (FIG. 17).

  If the cancel button is pressed, the number of incidental images is decremented by 1 in step S808, &#8232; the incidental image number of the captured incidental image is cleared, and the screen returns to the transmission image confirmation screen (FIG. 17).

If the reshoot button is pressed, the number of incidental images is decremented by 1 in step S809, &#8232; the incidental image number of the captured incidental image is cleared, and the process returns to step S804 to recapture the incidental image.
The above is an example of a processing flow in which an image attached to the captured image of the application form is captured and added.

  As described above, according to the present embodiment, it is possible to facilitate photographing with a mobile terminal and to make the entry check using an image efficient.

  In addition, the entry check can be simply performed by the time when the entry check process by image recognition is completed, and the convenience can be improved. By enabling a simple entry check using a cover sheet, a large number of application forms can be efficiently processed.

  In addition, if you are uneasy with a simple entry check (visual check) using a cover sheet, you can wait for the system to check for entry errors, and the checker (user) will choose which to do. Therefore, it is more convenient and the efficiency of business can be improved.

  In addition, by attaching the contact information of the end user as header information to the application form and the accompanying image, it is possible for the operator on the server side to find out the incomplete entry and to contact the checker easily.

  Furthermore, since it is possible to know whether the application form sent by the server-side operator has already been checked for deficiencies in the system, confirmation of deficiencies in entry is performed preferentially from those not checked for deficiencies in the system. be able to.

  As described above, a recording medium that records a program that implements the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by reading and executing.

  In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium recording the program constitutes the present invention.

  As a recording medium for supplying the program, for example, flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, nonvolatile memory card, ROM, EEPROM, silicon A disk or the like can be used.

  Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on an instruction of the program is actually It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the processing and the processing is included.

  Furthermore, after the program read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is based on the instructions of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  The present invention may be applied to a system constituted by a plurality of devices or an apparatus constituted by a single device. Needless to say, the present invention can be applied to a case where the present invention is achieved by supplying a program to a system or apparatus. In this case, by reading a recording medium storing a program for achieving the present invention into the system or apparatus, the system or apparatus can enjoy the effects of the present invention.

  The form of the program may be in the form of object code, program code executed by an interpreter, script data supplied to an OS (operating system), and the like.

  Furthermore, by downloading and reading a program for achieving the present invention from a server 102, a database, etc. on the network using a communication program, the system or apparatus can enjoy the effects of the present invention. In addition, all the structures which combined each embodiment mentioned above and its modification are also included in this invention.

100 form system 101 information processing apparatus 102 server 103 network

Claims (9)

A portable terminal including an imaging unit,
Shooting determination means for determining whether or not a shooting condition is matched from an image corresponding to a document obtained from the imaging unit;
Automatic shooting execution means for executing automatic shooting when the shooting determination means matches the shooting conditions;
Switching means for switching to manual shooting when the shooting determination means does not match the shooting conditions;
Check means for performing entry check on the portable terminal using an image obtained by photographing performed by the automatic photographing execution means;
A portable terminal comprising: a transmission unit configured to transmit an image obtained by manual imaging switched by the switching unit so that the image is checked by a server. The shooting condition is a shooting condition for determining whether or not a plurality of predetermined locations in the image of the document match a pre-registered image,
The shooting determination unit determines whether or not the image matches the previously registered image,
The portable terminal according to claim 1, wherein the automatic photographing execution unit executes automatic photographing when the image coincides with the previously registered image. The shooting condition is a condition as to whether or not the locations related to the four corners of the image of the document match,
A specifying means for specifying a point of a portion relating to the remaining corner when a portion relating to the three corners of the image of the document matches the pre-registered image;
The automatic photographing execution means executes automatic photographing when a location related to the four corners matches the pre-registered image or when a location related to the three corners matches a pre-registered image,
When automatic shooting is executed with the locations related to the four corners matching the pre-registered images, the points obtained according to the locations related to the four corners are used to obtain the automatic shooting execution means. When the automatic shooting is executed with the three corners corresponding to the pre-registered image, the points obtained according to the three corners and the specifying means are specified. The mobile terminal according to claim 2, further comprising a correction unit that corrects an image obtained by the automatic photographing execution unit using the determined point. The imaging unit includes a first imaging unit that captures the document and a second imaging unit on the display unit side that displays an image obtained by the first imaging unit, and is obtained from the second imaging unit. Face recognition means for performing face recognition using an image;
4. The automatic photographing execution unit executes automatic photographing when a location related to the three corners matches a pre-registered image and a face is recognized by the face recognition unit. Mobile devices. A portable terminal including a first imaging unit that captures a document, and a second imaging unit on a display unit side that displays an image obtained by the first imaging unit,
Photographing determination means for determining whether or not a photographing condition is matched from an image corresponding to the document obtained from the first imaging unit;
Automatic shooting execution means for executing automatic shooting when the shooting determination means matches the shooting conditions;
Check means for performing entry check on the portable terminal using an image obtained by photographing performed by the automatic photographing execution means;
Face recognition means for performing face recognition using an image obtained from the second imaging unit,
The mobile terminal according to claim 1, wherein the automatic photographing execution unit executes automatic photographing when the photographing determination unit does not match photographing conditions and the face is recognized by the face recognition unit. A processing method for a portable terminal including an imaging unit,
A shooting determination step for determining whether or not a shooting condition is met from an image corresponding to the document obtained from the imaging unit;
An automatic shooting execution step for performing automatic shooting when the shooting determination step matches the shooting conditions;
When the shooting determination step does not match the shooting conditions, a switching step for switching to manual shooting,
A check step for performing an entry check on the mobile terminal using the image obtained by the photographing performed in the automatic photographing execution step;
And a transmission step of transmitting the image so as to check the image obtained by the manual photographing switched by the switching step with a server. A processing method of a portable terminal including a first imaging unit that captures a document and a second imaging unit on a display unit side that displays an image obtained by the first imaging unit,
A shooting determination step for determining whether or not a shooting condition is met from an image corresponding to the document obtained from the first imaging unit;
An automatic shooting execution step for performing automatic shooting when the shooting determination step matches the shooting conditions;
A check step for performing an entry check on the mobile terminal using the image obtained by the photographing performed in the automatic photographing execution step;
A face recognition step of performing face recognition using an image obtained from the second imaging unit,
The automatic photographing execution step performs automatic photographing when the photographing determination step does not coincide with photographing conditions and when the face is recognized by the face recognizing step. A mobile terminal program including an imaging unit,
The portable terminal;
Shooting determination means for determining whether or not a shooting condition is matched from an image corresponding to a document obtained from the imaging unit;
Automatic shooting execution means for executing automatic shooting when the shooting determination means matches the shooting conditions;
Switching means for switching to manual shooting when the shooting determination means does not match the shooting conditions;
Check means for performing entry check on the portable terminal using an image obtained by photographing performed by the automatic photographing execution means;
A program that causes a server to check an image obtained by manual photographing switched by the switching unit, and that functions as a transmission unit that transmits the image. A program for a portable terminal including a first imaging unit that captures a document and a second imaging unit on a display unit side that displays an image obtained by the first imaging unit,
The portable terminal;
Photographing determination means for determining whether or not a photographing condition is matched from an image corresponding to the document obtained from the first imaging unit;
Automatic shooting execution means for executing automatic shooting when the shooting determination means matches the shooting conditions;
Check means for performing entry check on the portable terminal using an image obtained by photographing performed by the automatic photographing execution means;
Function as face recognition means for performing face recognition using an image obtained from the second imaging unit;
The automatic photographing execution means executes automatic photographing when the photographing determination means does not match photographing conditions and the face recognition means recognizes a face.

Images