JP6738008B2 - Mobile terminal, processing method and program therefor - Google Patents

Description

The present invention relates to a mobile terminal that performs entry check using a captured image, a processing method thereof, and a program.

Conventionally, there is a mechanism in which form data is transmitted from a user terminal to a form server, the form server side checks whether or not there are any deficiencies in the entered contents, and returns the check result to the user terminal.

Patent Document 1 discloses an electronic processing system in which all the processes such as content determination of form data transmitted from a user terminal, error correction by an operator's operation, and creation of a process result with a correction result message are performed on the form server side. Technology is described.

JP-A-6-28275

However, in the system configuration of Patent Document 1, in order to perform the entry deficiency check, it is necessary to send the form data to the form server without fail, and the entry deficiency check cannot be performed in an environment where it is difficult to communicate with the form server. In addition, even if a small deficiency check such as omission is made, communication between the user terminal and the form server occurs, and it takes time to complete a form without any deficiency.

Therefore, it is conceivable that a user terminal (for example, a mobile terminal such as a tablet terminal) is checked for incompleteness. However, when a form is photographed by the mobile terminal, unlike a scanner or the like, an image of a predetermined size, Alternatively, there is a problem that it is difficult to capture an image with high accuracy (image with less camera shake and less distortion).
Therefore, it is necessary to display a guide at the time of shooting and to execute shooting when an appropriate state is reached.

An appropriate state is, for example, when the condition that an image can be obtained along a guide is met, but by changing the angle of the mobile terminal, moving the mobile terminal away from or near the form, and setting it in an appropriate state. It took a while to get used to the operation, and there were cases where the shooting state did not go well. In particular, in order to simplify the operation, it is desired to automatically shoot the image if the conditions are met. Therefore, if the appropriate state is not obtained, the image is not executed and the usability may be deteriorated.

An object of the present invention switches the imaging method by the face recognition is that it provides a mechanism you facilitate shooting on mobile devices.

To achieve the object of the present invention, there is provided a mobile terminal having an image pickup unit, and image pickup determination means for determining whether or not image pickup conditions match from an image corresponding to a document obtained from the image pickup unit, When the shooting determination means matches the shooting conditions, the automatic shooting execution means executes the automatic shooting, and when the shooting determination means does not match the shooting conditions, the switching means for switching to the manual shooting and the automatic shooting execution Using the image obtained by the photographing performed by the means, the checking means for performing the entry check on the mobile terminal, and the image obtained by the photographing of the manual photographing switched by the switching means to check on the server, A mobile terminal, comprising: a transmission unit that transmits the image.

The mobile terminal includes a first image capturing unit that captures a document, and a second image capturing unit on the display unit side that displays an image obtained by the first image capturing unit. Shooting determination means that determines whether or not the shooting conditions match from the image corresponding to the obtained document, and automatic shooting execution means that executes automatic shooting when the shooting conditions are matched by the shooting determination means, A face recognizing unit for recognizing a face using an image obtained from the second image capturing unit, wherein the automatic image capturing executing unit is a case where the image capturing determining unit does not match the image capturing condition. When face recognition is performed by, automatic shooting is performed.

According to the present invention, to switch the imaging method by the face recognition, easily it can be Rukoto shooting on mobile devices.

The figure which shows an example of a structure of the form system 100 of this invention. The figure which shows an example of the hardware constitutions of the information processing apparatus 101. Functional block diagram showing an example of the information processing apparatus 101 and the server 102. The flowchart which shows an example of the whole process which acquires the check result of deficiency in the information processing apparatus 101, and transmits to the server 102. Flowchart showing an example of shooting processing in the information processing apparatus 101 Flowchart showing an example of the entry deficiency check processing performed by the information processing apparatus 101 Flowchart showing an example of a process in which the server 102 receives the photographed image of the application form, the accompanying image, the entry check flag, the login user name, and the form name. The flowchart which shows an example of the process which displays the application list received by the server 102. The flowchart which shows an example of the process which adds the image which accompanies the picked-up image of an application in the information processing apparatus 101. The figure which shows an example of the deficiency confirmation cover sheet put on an application form in order to confirm visually the check of the deficiency of the application form entry in the information processing apparatus 101. The figure which shows an example of the screen which covers the deficiency confirmation cover sheet on the application form photographed image in order to make it easy to visually check the entry deficiency in the information processing apparatus 101. The figure which shows an example of the screen which displays the entry deficiency check result in the information processing apparatus 101. The figure which shows an example of the table which manages the form form and the defect confirmation cover sheet in the server 102 and the information processing apparatus 101. The figure which shows an example of the table which manages the transmission time of the application received by the server 102, the sender, the form ID, and the entry defect check flag. The figure which shows an example of the screen which displays the table managed by FIG. 14 on the server 102. The figure which shows an example of the screen which displays the application received by the server 102. A diagram showing an example of a table for managing the deficiency check recognition area and the deficiency check method for each form The figure which shows an example of the screen which confirms the image of the application form transmitted with the information processing apparatus 101. The figure which shows an example of the screen which confirms and displays the incidental image of the application form transmitted by the information processing apparatus 101. The figure which shows an example of the screen which performs the information setting preset with the information processing apparatus 101. The figure which shows an example of the hardware constitutions of the server 102. Image diagram showing an example of a state at the time of shooting by the information processing apparatus 101 Image diagram showing an example of a manual shooting switching state at the time of shooting by the information processing apparatus 101 FIG. 3 is a diagram showing an example of settings (mark images, etc.) used for shooting conditions.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
The characteristic processing of the present embodiment will be described with reference to the following flowchart, and the particularly characteristic processing thereof is FIG. 4B.

As shown in FIG. 1, the form system 100 (information processing system) according to the present embodiment has a configuration in which an information processing apparatus 101 (mobile terminal) and a server 102 are communicably connected via a network 103. The information processing apparatus 101 uses a camera or the like to capture an image of the form and checks the form incompleteness. The server 102 manages, for each form form identification information, a recognition area and a recognition condition of the entry content required for checking the entry deficiency of the form. The information processing apparatus 101 is a portable terminal such as a mobile phone, a smart phone, a tablet terminal, and a PDA terminal, on which an application of a browser or a form system is installed, and wirelessly communicates with the server 102 via the network 103. Can communicate. It goes without saying that the server 102 may be a general-purpose computer.

Hereinafter, an example of a hardware configuration applicable to the information processing apparatus 101 shown in FIG. 1 will be described with reference to FIG.
In FIG. 2, reference numeral 201 denotes a CPU, which centrally controls each device and controller connected to the system bus 204. Further, the ROM 203 or the external memory 212 is necessary to realize a control program of the CPU 201, such as a BIOS (Basic Input/Output System), an operating system program (hereinafter, OS), and a function executed by each server or each PC. Various programs and the like are stored.

A RAM 202 functions as a main memory and a work area of the CPU 201. The CPU 201 implements various operations by loading a program or the like required for executing the processing from the ROM 203 or the external memory 212 into the RAM 202 and executing the loaded program.

An input controller 205 controls the input from the touch sensor 210. A video controller 206 controls display on a display device such as the display 211. Although the display 211 is shown in FIG. 2, the display is not limited to the CRT and may be another display such as a liquid crystal display. These are used by the user of the information processing apparatus 101 as needed.

A memory controller 207 is installed in an external storage device (hard disk (HD)) for storing a boot program, various applications, font data, user files, edit files, various data, a flexible disk (FD), or a PCMCIA card slot. It controls access to an external memory 212 such as a CompactFlash (registered trademark) memory connected via an adapter.

A communication I/F controller 208 connects and communicates with an external device via the network 103, and executes communication control processing on the network. For example, communication using TCP/IP is possible.

The CPU 201 enables display on the display 211, for example, by executing outline font rasterization processing in the display information area in the RAM 202. Further, the CPU 201 can accept a touch operation instruction from the user on the touch sensor 210 on the display 211.

Various programs to be described later for implementing the present invention are recorded in the external memory 212 and are executed by the CPU 201 by being loaded into the RAM 202 as necessary. Further, the definition file, map data, etc. used when the above program is executed are also stored in the external memory 212, and a detailed description thereof will be described later.

A photographed image input controller 209 controls an input from the camera 215 via the photographed image input processing unit 214 (corresponding to photographing means). It is possible to recognize a captured image input from the camera 215. One camera 215 is provided on the front side of the display side, and one camera 215 is provided on the back side opposite to the display side. The camera 215 on the back side photographs a form such as an application form, and the camera 215 on the front side recognizes the face of the photographer. Further, there is a photographed image input processing unit 214 corresponding to each camera 215, and receives an image of each camera. Further, at the time of automatic or manual photographing execution, the entry defect checking application in the present embodiment is realized by requesting an instruction (shutter ON) to execute photographing of the camera on the rear side to the operating system. Specifically, since the camera 1 (rear surface) and the camera 2 (front surface) have identification information, a request including the identification information of the camera that releases the shutter is performed, and the shutter is released to any camera. Can be cut.
Next, the hardware configuration of the information processing apparatus applicable to the server 102 shown in FIG. 1 will be described with reference to FIG.

In FIG. 20, reference numeral 2001 denotes a CPU, which comprehensively controls each device and controller connected to the system bus 2004. Further, the ROM 2002 or the external memory 2011 is necessary to realize a control program of the CPU 2001 such as a BIOS (Basic Input/Output System), an operating system program (hereinafter, OS), and a function executed by each server or each PC. Various programs to be described later are stored.

Reference numeral 2003 denotes a RAM, which functions as a main memory and a work area of the CPU 2001. The CPU 2001 implements various operations by loading a program or the like necessary for executing processing from the ROM 2002 or the external memory 2011 into the RAM 2003 and executing the loaded program.

An input controller 2005 controls inputs from a keyboard (KB) 2009 and a pointing device such as a mouse (not shown). A video controller 2006 controls display on a display such as a CRT display (CRT) 2010. In addition, although it is described as a CRT 2010 in FIG. 20, the display device is not limited to the CRT, and may be another display device such as a liquid crystal display. These are used by the administrator as needed.

A memory controller 2007 is connected to a hard disk (HD) for storing a boot program, various applications, font data, user files, edit files, various data, etc., a flexible disk (FD), or a PCMCIA card slot via an adapter. It controls access to the external memory 2011 such as a CompactFlash (registered trademark) memory and a database.

A communication I/F controller 2008 connects and communicates with an external device via a network (for example, 103 shown in FIG. 1), and executes communication control processing on the network. For example, communication using TCP/IP is possible.

Note that the CPU 2001 enables display on the CRT 2010 by executing outline font rasterization processing in a display information area in the RAM 2003, for example. Further, the CPU 2001 enables a user instruction with a mouse cursor or the like (not shown) on the CRT 2010.

Various programs to be described later for implementing the present invention are recorded in the external memory 2011, and are executed by the CPU 2001 by being loaded into the RAM 2003 as needed. Furthermore, the definition file and various information tables used when the above program is executed are also stored in the external memory 2011.

FIG. 3 is a block diagram showing an example of the functional configuration of the information processing apparatus 101 (mobile terminal). The information processing apparatus 101 will be described as a mobile terminal including an imaging unit. Alternatively, a mobile terminal including a first imaging unit that captures a document and a second imaging unit on the display unit side that displays an image obtained by the first imaging unit will be described.

The shooting determination unit 301 is a functional unit that determines whether or not the shooting conditions match from the image corresponding to the document obtained from the imaging unit. The shooting condition is a shooting condition for determining whether or not a plurality of predetermined portions in the image of the document match the pre-registered image. Further, the shooting condition is a condition of whether or not the parts related to the four corners of the image of the document match. The shooting determination unit 301 is a functional unit that determines whether or not it matches an image registered in advance.

The automatic shooting execution unit 302 is a functional unit that executes automatic shooting when the shooting conditions are met. It is also a functional unit that executes automatic shooting when it matches an image registered in advance. In addition, it is a functional unit that executes automatic shooting when the parts related to the four corners match the image registered in advance or the parts related to the three corners match the image registered in advance. Further, it is a functional unit that executes automatic photographing when the face recognition unit 308 recognizes a face in which the positions related to the three corners match the image registered in advance.
The switching unit 303 is a functional unit that switches to manual shooting when the shooting conditions do not match.
The check unit 304 is a functional unit that performs an entry check on the mobile terminal using an image obtained by the executed automatic shooting.
The transmitting unit 305 is a functional unit that transmits the image so that the server can check the image obtained by the shooting of the switched manual shooting.
The identifying unit 306 is a functional unit that identifies the points of the remaining corners when the three corners of the image of the document match the images registered in advance.

The correction unit 307 is obtained by the automatic photographing using the points obtained according to the portions related to the four corners when the automatic photographing is performed with the portions related to the four corners matching the images registered in advance. When the image is corrected and the automatic photographing is performed in such a manner that the parts related to the three corners match the image registered in advance, the points obtained according to the parts related to the three corners and the points specified by the specifying unit 306 are determined. It is a functional unit that corrects an image obtained by automatic shooting by using it.
The face recognition unit 308 is a functional unit that performs face recognition using an image obtained from the second imaging unit.

FIG. 4A is a flowchart showing an example of an overall process in which the information processing device 101 obtains a check result of entry defects and transmits it to the server 102 (hereinafter, entry defect check result acquisition/transmission process).

In addition, FIG. 19 is an example of a screen for performing information setting that is preset in the information processing apparatus 101. On the initial setting screen 1901, when the store name, the person in charge, and the telephone number are input and the OK button is pressed, the information is stored in the information processing apparatus 101. When the form of the application form to be photographed is determined in advance like the initial setting screen 1902, the form form definition information for character recognition and the form ID for specifying the cover sheet are input. May be. This initial setting is set by the filling-in deficiency check application.

In step S401, the information processing device 101 activates the entry defect checking application and logs in. By pressing an initial setting button (not shown) when the entry defect checking application is activated, the above initial setting screen is displayed and the initial setting is appropriately executed by the user's operation.

In step S402, when it is detected that the application form photographing button (not shown) is pressed, the camera 215 is activated to photograph the application form for checking the entry defects. Here, the four corners of the form such as the application form are detected, and the photographing is executed when the focus is obtained. By this execution, the captured image corresponding to the application form is acquired. Imaging by the four corner detection will be described with reference to FIG. 4B. This 4B is a characteristic process in this embodiment.

Here, FIG. 4B is a flowchart showing an example of the photographing process. Note that each step is executed by the CPU 201.

In step S430, the front (display side) camera 215 (first imaging unit) and the rear (display side) camera 215 (second imaging unit) are activated. The CPU 201 displays the image of the camera 215 on the back surface, which is accepted via the captured image input controller 209, on the display 211. Alternatively, the CPU 201 stores the image of the front camera 215 in the RAM 202 via the captured image input controller 209. The image of the front camera 215 is deleted from the memory when the next image is received. Face recognition is performed using this accumulated image.

Here, the display 211 is displayed as 2100 in FIG. A guide is displayed so that the photographer can easily photograph the application form as a subject, and the mark registered in FIG. 23 is displayed in 2301. By aligning the mark of the subject with this mark, automatic shooting becomes easy. In addition, 2102 is displaying the image acquired from the camera 215 on the back surface. In this state, the captured image has not been recorded yet. This state is also referred to as live view, and is a technology for displaying a video to be shot using an image obtained by an image sensor (CCD).
In step S431, the value of Counter for specifying the number of detected marks is set to 0 and stored in the RAM 202.

In step S432, it is determined whether or not the subject (the form such as the application form in the present embodiment) is in focus by using the autofocus technique. If the image is in focus, the process proceeds to step S433, and the process is repeated until the image is in focus. It is to be noted that the detection of the in-focus state is determined by receiving the entry defect check application from the operating system.

In step S433, image recognition is performed on the corner A, and it is determined whether the corner A has a predetermined mark. Specifically, using the recognition area (coordinates in a predetermined range) registered in the setting information 2300 of FIG. 23, the image of the corner A (upper left) of the image captured from the camera 215 on the rear surface and the setting information 2300. The image 2301 (mark: black circle) registered as the value of is compared. The image recognition technique is realized by using a known technique, but is realized by using pattern matching between images, for example.
If the mark can be detected at the corner A, the process proceeds to step S434. If no mark is detected at the corner A, the process proceeds to step S435.
In step S434, since the mark is detected at the corner A, the counter is incremented by 1 and stored in the RAM 202.

In step S435, image recognition is performed on the corner B, and it is determined whether or not there is a predetermined mark in the corner B. Specifically, using the recognition area (coordinates in a predetermined range) registered in the setting information 2300 of FIG. 23, the image of the corner B (upper right) of the image captured from the camera 215 on the rear surface and the setting information 2300. The image 2302 (mark: black triangle) registered as the value of is compared. The image recognition technique is realized by using a known technique, but is realized by using pattern matching between images, for example.
If the mark can be detected at the corner B, the process proceeds to step S436. If no mark is detected at the corner B, the process proceeds to step S437.
In step S436, since the mark is detected at the corner B, the counter is incremented by 1 and stored in the RAM 202.

In step S437, image recognition is performed on the corner C, and it is determined whether or not the corner C has a predetermined mark. Specifically, by using the recognition area (coordinates in a predetermined range) registered in the setting information 2300 of FIG. 23, the image of the corner C (lower right) of the image captured from the camera 215 on the rear surface and the setting information. The image 2303 (mark: black square) registered as the value of 2300 is compared. The image recognition technique is realized by using a known technique, but is realized by using pattern matching between images, for example.
If the mark can be detected at the corner C, the process proceeds to step S438. If no mark is detected at the corner C, the process proceeds to step S439.
In step S438, since the mark is detected at the corner C, the counter is incremented by 1 and stored in the RAM 202.

In step S439, image recognition is performed on the corner D, and it is determined whether the corner D has a predetermined mark. Specifically, using the recognition area (coordinates in a predetermined range) registered in the setting information 2300 of FIG. 23, the image of the corner D (lower left) of the image captured from the camera 215 on the rear surface and the setting information 2300. The image 2303 (mark: black star) registered as the value of is compared. The image recognition technique is realized by using a known technique, but is realized by using pattern matching between images, for example.
If the mark can be detected at the corner D, the process proceeds to step S440. If no mark is detected at the corner D, the process proceeds to step S441.
In step S440, since the mark is detected at the corner D, Counter is incremented by 1 and stored in the RAM 202.
Note that the mark is an example, and any mark that can be recognized and is different at the four corners may be used.

In step S441, the value of Counter stored in the RAM 202 is acquired and it is determined whether the value is "4". That is, it is determined whether or not four points (four corners) have been detected. If the value is "4", the process proceeds to step S422. If the value is not "4", the process proceeds to step S433.

In step S442, automatic shooting is performed. This automatic shooting is performed using the automatic shutter technology. The photographed image obtained by this automatic photographing is stored in the internal memory. An image when automatically photographed is 2110 in FIG.

Note that a flag indicating that the image is a photographed image obtained by automatic photographing is added and managed. Further, the captured image is corrected based on the coordinates of the four points obtained at the time of capturing. For example, keystone correction is performed. In addition, the distortion of the application itself is corrected using a known correction technique. It is assumed that this corrected image is used for subsequent defect check processing. Further, when the three corners described later can be recognized and the fourth point cannot be obtained, the image is corrected using the estimated fourth point and the three points based on the three corners. That is, when the automatic shooting is performed by matching the parts related to the four corners with the pre-registered image, the image obtained by the automatic shooting is used by using the points obtained according to the parts related to the front four corners. If the correction is performed and the portions related to the three corners are automatically photographed in conformity with the pre-registered image, the points specified as the points obtained according to the portions related to the three corners (the estimated fourth point) are used. Is a step showing an example of a process for correcting an image obtained by automatic photographing.

In step S443, it is determined whether or not the process has been repeated for a predetermined time or a predetermined number of times. When steps S431 to S441 are repeated for a predetermined time or a predetermined number of times, the image recognition of four points is not successful, so the process proceeds to step S444. If it has not reached the predetermined time or the predetermined number of times, the process proceeds to step S431. The process of step S443 may be executed in the case of NO in step S444. That is, the determination as to whether the predetermined time or the predetermined number of times has been reached may be executed at any timing.

In step S444, it is determined whether the acquired value of Counter is "3". That is, it is determined whether or not four points (four corners) cannot be detected, but the third to fourth points can be estimated. When you use a tablet device to shoot a form such as an application form placed on a table with the camera on the back, it may not fit well in the guide depending on the shooting direction (the angle of the tablet with respect to the form) and the distance between the form and the tablet device. appear. This is a problem that can actually occur, and if the photographer is not accustomed to shooting, the image portion of the form will be distorted such as a trapezoid, and it will not match the guide well. In that case, since it is necessary to correct the image, it is necessary to specify whether the four points can be estimated. In particular, in order to automatically check the incompleteness or to use the incompleteness confirmation cover sheet in FIG. 9, image correction is indispensable, and it is important that four points can be estimated. If four points can be estimated, the process can be automatically switched to automatic shooting because it is possible to automatically check for incompleteness or incompleteness using a defect confirmation cover sheet.

That is, if the value of Counter is “3”, the process proceeds to step S425. If the value of Counter is not "3", that is, if it is "0", "1", or "2", the process proceeds to step S447.

Steps S432 to S441, step S443, and step S444 are steps showing an example of a process of determining whether or not the image capturing conditions match from the image corresponding to the document obtained from the image capturing unit. The shooting condition is a shooting condition for determining whether or not a plurality of predetermined portions (for example, four corners) in the image of the document match the images (2301 to 2304 in FIG. 23) registered in advance.

Further, step S442 is a step showing an example of a process of executing automatic shooting when the shooting conditions are matched. In addition, it is a step showing an example of a process of executing automatic shooting when the image matches a pre-registered image. In addition, it is a step showing an example of a process of executing the automatic photographing when the parts related to the four corners match the image registered in advance or the parts related to the three corners match the image registered in advance.

In step S445, it is determined whether or not there is face recognition in the image acquired from the front camera 215. That is, it is a step showing an example of processing for face recognition using an image obtained from the second imaging unit.

Although the face recognition technology is a known technology, description thereof will be omitted, but the area for detecting the face recognition is set to the central portion of the image according to the setting of 2305. As shown in FIG. 23, the image 2305 is an area inside the shooting area. The face recognition is detected here by an actual shooting operation, and when automatic shooting is not performed because it does not fit the guide well (basic shooting is automatic shooting), the photographer shoots while looking at the display from the front. By recognizing the face of the photographer, it is possible to determine that the photographer cannot capture well. In addition, when the face cannot be recognized (it may be a side face), it means that you are not operating by looking at the display, and it is possible that you are looking away from yourself, so you do not shift to automatic shooting, Move to manual shooting.

If it is determined that the face is recognized in the set area, the process proceeds to step S445. If it is determined that there is no face recognition in the set area, the process moves to step S447. That is, this is a step showing an example of a process of shifting to a process (step S442) of executing automatic shooting when a face recognition is carried out when the parts related to the three corners match the image registered in advance.

In step S446, the fourth point is specified based on the recognized three points (three corners). As a method of estimating the fourth point, for example, when the coordinates of the points of the corner A, the corner B, and the corner C can be specified, the vector 1 connecting the points of the corner A and the corner B is generated. Further, a vector 2 connecting the points of the corner B and the corner C is generated. The vector 1 is applied to the corner C (vector 1C), the vector 2 is applied to the corner A (vector 2A), and the point where the vector 1C and the vector 2A intersect is the fourth point.

That is, this is a step showing an example of the process of identifying the points of the remaining corners when the three corners of the image of the document match the images registered in advance in step S444.

In step S447, automatic shooting is switched to manual shooting. It is realized by switching the setting of the camera from auto to manual. When switching to this manual setting, whether or not to switch to manual shooting is notified as in 2200 of FIG. When the manual shooting switching button is pressed, the manual switching is performed. When the automatic shooting continuation button is pressed, the process proceeds to step S431, and the four corners are detected again. When the mode is switched to the manual shooting, the process proceeds to step S448. That is, this is a step showing an example of a process of switching to manual shooting when the shooting conditions do not match.

In step S448, it is determined whether or not the photographer has given a photographing instruction. Specifically, when switching to the manual shooting, the shooting button is displayed as in 2210 of FIG. 22, and thus it is detected that the shooting button is pressed. If the shooting instruction is given, the process proceeds to step S449. If no shooting instruction is given, the instruction is awaited.

In step S449, shooting is performed according to the shooting instruction. Save the captured image in the internal memory. Since this photographed image is not photographed in accordance with the guide, it is not possible to automatically check for incompleteness or to perform incompleteness check using the incompleteness confirmation cover sheet. Therefore, unlike the image captured automatically, the captured image is sent to the server 102 and the visual check is performed.
As a result, it is possible to facilitate shooting with a mobile terminal and improve the efficiency of entry check using images.

Returning to the flowchart of FIG. 4A, the description of the processing is continued.
In step S402-1, it is determined whether or not automatic shooting has been performed. If the automatic shooting has been performed, the process proceeds to step S403. If the image is manually photographed, the process proceeds to step S402-2. In the case of automatic photographing, since the photographing is performed along the guide, it is possible to automatically perform a check for incompleteness or an incompleteness check using an incompleteness confirmation cover sheet. Therefore, when the automatic shooting is performed, the process proceeds to step S403. In addition, since it is difficult to automatically check the deficiency in the entry or the deficiency check using the deficiency confirmation cover sheet, the process proceeds to step S402-2.

In step S402-2, the manually photographed image is transmitted to the server 102. That is, this is a step showing an example of a transmission process for transmitting the image so that the server can check the image obtained by the manual photographing.

At this time, a flag indicating that the manual photographing has been performed is also sent to the server 102 so that the checker of the server 102 recognizes the manual photographing and facilitates the check. On the server side, it is a manual shooting, and it is displayed on the client terminal connected to the server side that the information processing apparatus 101, which is a mobile terminal, is not checked for defects.

In step S403, it is determined whether there is a form that matches the captured image. If there is a matching form, the photographed image and the form image are compared in the data order of the local form table 1200 (FIG. 12) by using a form recognition technique or the like, and the matching form is specified. As described above, when the form ID is set in advance, it may be determined whether or not the form ID is in the form table. Further, the form may be specified by extracting the form ID from a predetermined area of the captured image. If it is determined in step S403 that there is no matching form, the process proceeds to step S404, and if it is determined that there is a matching form, the process proceeds to step S406.

If there is no matching form table in step S404, an error message indicating that the application form photographed in step S405 is not a defect check compatible form is displayed, and the application is terminated. If it is not the corresponding form, an error message may be displayed and then the process of re-shooting may be performed. If there is a matching form table, the cover sheet (image) corresponding to the photographed application form is acquired from the external memory 212 with reference to the form and the form table (FIG. 12) in step S406. The form table 1200 and the corresponding images (form image, cover sheet) are stored in the external memory 212. That is, it shows an example of a process of acquiring a second image corresponding to the first image obtained by photographing, which is a cover sheet and in which the entry check location is visibly transmitted.

In step S407, the captured image of the captured application form and the defect confirmation cover sheet (FIG. 9) corresponding to the captured application form acquired in step S404 are overlaid to generate a check image. That is, it shows an example of a process of generating a third image for entry check by using the first image and the second image.

The generated image is displayed on the entry defect checking screen (FIG. 10-1). That is, it shows an example of processing for displaying the generated third image so that the user can check the entry.
This cover sheet may be a cover sheet that includes a description of the check location so that the user can easily check it.

The cover sheet of FIG. 9 is transparently processed at the check points, and when it is superimposed on the photographed image, it is possible to easily check the entry at the point where the entry defect check is desired. It should be noted that it is also possible to make the parts other than the check part transparent, but in order to make the check part easier for the user to visually recognize, the transmittance is made lower than the check part or the color is changed (for example, set to gray). It is desirable to make the image so.

In step S408, in order to initialize the deficiency check process, the entry deficiency check completion flag is turned off, and the OK button flag/cancel button flag is turned off.

In step S409, an entry deficiency check processing flow (described later in FIG. 5) is performed. This entry deficiency check processing is executed in the background displaying the entry deficiency checking screen of FIG. That is, it shows an example of a check process in which an entry check is performed on the mobile terminal by using an image obtained by the executed shooting. In addition, it shows an example of execution of a recognition process based on the first image and the definition information in the background of the display of the third image.

In step S410, the entry defect check flag, the OK button flag, and the cancel button flag, which are the recognition results, are acquired. It is confirmed in the entry defect check processing flow (FIG. 5) whether the OK button flag is ON, and if it is ON, the flow goes to the incidental image addition processing flow of step S422 (described later in FIG. 8). Note that the case where the OK button flag is ON is the case where a visual defect check is performed using the overlaid image on the entry defect check screen of FIG. 10A and the defect check is completed. .. When the process of step S422 is completed, in step S423, the information processing apparatus 101 transmits the captured image of the application form, the accompanying image, the form name, the login user name, and the defect check completion flag OFF to the server 102, and the process ends.

In step S410, it is checked whether the OK button flag is ON in the entry defect check processing flow (FIG. 5), and if it is OFF, it is checked in step S411 whether the cancel button flag is ON. If the cancel button is ON, the application is terminated.

If the cancel button flag is OFF in step S411, in step S412 a confirmation button (display of recognition result information) indicating that the entry defect check is completed is displayed in FIG. -Change to 2). When the cancel button flag is OFF, the entry defect check completion flag is ON. In other words, it is the case where the deficiency check is completed in the background, not by visual inspection but by recognition by the system.

In other words, when the result recognized based on the first image and the definition information is acquired, and when the result is recognized and the result is acquired, information indicating that the result recognized with the third image is obtained (for example, It shows an example of a process of displaying a confirmation button).

In step S413, it is determined which of the OK button, the cancel button, and the confirmation button has been pressed on the entry defect check completion screen (FIG. 10-2). When the cancel button is pressed, the application is closed. When the OK button is pressed, the process moves to the auxiliary image addition processing flow (described later in FIG. 8) in step S422. When the process of step S422 is completed, in step S424, the information processing apparatus 101 transmits the captured image of the application form, the accompanying image, the form name, the login user name, and the defect check completion flag ON to the server 102, and ends the process. If there is any deficiency, the confirmation button is pressed to confirm and then the transmission is controlled. If the confirmation button is pressed, the process proceeds to step S414.

In step S414, the information processing apparatus 101 searches the entry defect check table 1610 using the form ID of the captured image identified in step S403 as a search key, and stores the RAM 202 in the result column 1611 of check ID=i of all acquired records. Register the value of the variable result[i] stored in. In the present embodiment, the variable result[i]=[OK, NG, OK, NG, OK] indicating the check result of the form ID=1000 in the result column 1611 of the five records of the form ID=1000 of the entry defect check table 1610. The value of is registered.

In step S415, the information processing apparatus 101 closes the entry defect check completion screen (FIG. 10-2) from the display 211 and displays the captured image display captured image on the entry defect check result screen (FIG. 11). That is, when there is an instruction to display the recognized result by pressing the confirmation button, the process of switching from the third image to the recognition result image generated according to the result recognized as the first image and displaying it FIG.
In step S416, the information processing apparatus 101 determines whether NG exists in the variable result[i].

If NG does not exist, in step S417, the information processing apparatus 101 displays the entry defect check result screen (no defect) 1110 in which the message "There was no entry defect" was written.

If NG exists, in step S418, the information processing apparatus 101 displays the entry deficiency check result screen (deficiency) 1120 in which the message "There is entry deficiency."

In step S419, the information processing apparatus 101 searches the entry defect check table 1610 using the form ID=form ID of the photographed image and the result column 1611=NG as a search key, and the starting point coordinates (X, Y) of the entry defect check recognition area. ) And the end point coordinates (X, Y).

In step S420, the information processing apparatus 101 draws a rectangle whose diagonal is the start point coordinates (X, Y) and the end point coordinates (X, Y) of the acquired entry defect check recognition area on the captured image and fills it with red ( 1121). At this time, the rectangle is set to be transparent. In this way, it is possible to display the incompletely filled area in an identifiable manner.

In step S421, it is determined which button has been pressed on the entry defect check result screen. If the cancel button is pressed, exit the application, if the OK button is pressed

The incidental image addition process of step S422 (described later in FIG. 8) is performed. After step S422, in step S424, the captured image of the application form, the accompanying image, the form name, the login user name, and the defect check completion flag ON are transmitted from the information processing apparatus 101 to the server 102, and the process ends. That is, it shows an example of a process of outputting the information related to the entry check and the first image when an output instruction is given after the display of the third image. Also, when there is an additional image, an example of a process of outputting the first image and the additional image together is shown. It goes without saying that the supplementary image is not transmitted if there is no supplementary image.

The transmission may be performed for each application form, or if the communication with the server 102 is not possible, the information is stored in the information processing apparatus 101, and the data relating to the plurality of application forms when the communication becomes possible. May be transmitted.

FIG. 5 is a flowchart showing an example of the entry deficiency check process performed by the information processing apparatus 101.
In addition, FIG. 16 is an example of a table that manages the entry deficiency check recognition area and the deficiency check method for each form.

The entry deficiency check table 1610 stores the form ID, the check ID attached to the check location, the recognition condition of the entry deficiency check, and the starting point coordinates (X, Y) and the end point coordinates (X, Y) of the entry deficiency check recognition area. There is. That is, this is an example of storing definition information including a recognition area and recognition conditions for recognizing the entry content from the first image.

Although there is a result column 1611 for storing the entry defect check processing result in the "entry defect check table 1610", the result column 1611 is not yet used in the entry defect check process of FIG. is null.

In step S501, the entry defect check table 1610 is searched using the form ID of the captured image identified in step S404 as a search key, and an entry defect check record is acquired.
In step S502, 1 is substituted for the variable i and stored in the RAM 202.

In step S503, the starting point (X coordinate, Y coordinate) and end point (X coordinate, Y coordinate) of the entry deficiency check recognition area and the entry deficiency check recognition condition are acquired from the i-th entry deficiency check record.
In step S504, the value of the entry deficiency check recognition condition is determined.

If the entry deficiency check recognition condition is "required", it is an input required item. Therefore, in step S505, it is determined whether or not characters or symbols are entered in the check recognition area of the captured image. Note that this determination processing uses an analysis technique such as OCR, but since it is a known technique, its description is omitted.
If characters or symbols are entered, then in step S506, "OK" is assigned to the variable result[i] that stores the check result in the RAM 202.
If no character or symbol is entered, "NG" is substituted for the variable result[i] in step S507.

If the entry deficiency check recognition condition is "circle encircling", it means that one of the plurality of options is encircled by a circle, and therefore, in step S508, it is determined whether or not a circle is entered in the check recognition area of the captured image. Determine whether. This determination process uses an analysis technique such as OCR, but since it is a known technique, its explanation is omitted. Although the presence/absence of a circle mark is determined in this embodiment, the present invention is not limited to the circle mark, and may be a check mark or another figure.

Further, in the above description, the number of circles is set to "1", but the entry defect check table 1610 is provided with "upper limit value" and "check number" columns, and when the upper limit value is exceeded, "NG", the number of marks May be determined to be “NG” if the value is other than “the number of checks”. Further, even if the “digit number” column is provided in the entry defect check table 1610 and it is determined that the entered character is “OK” if the number of digits is permitted and “NG” if the number of digits is not permitted. Good. In addition, the entry defect check table 1610 may be provided with a “character attribute” column, and OK/NG determination may be made based on the attribute of the entered character, for example, “number”, “hiragana”, “katakana”, “alphanumeric character”, or the like. ..

Further, when the written character cannot be recognized by the analysis result by OCR or the like, it may be determined as “NG”. The value of the variable result[i] is not limited to the two values of “OK” and “NG”, and may have values such as “unclear”, “unrecognizable”, and “contradiction”.
If the circle is filled in, "OK" is assigned to the variable result[i] in step S509.
If the circle is not entered, then in step S510, "NG" is substituted for the variable result[i].

In step S511, the variable i is incremented to check the deficiency of the next area. In step S512, it is determined whether there is an entry defect check record with check ID=i. If it does not exist, the entry deficiency check completion flag is turned ON in step S517, and the entry deficiency check process is terminated. If it exists, it is determined in step S513 whether or not the OK button is pressed on the entry defect checking screen (FIG. 10-1). If the OK button is pressed, the OK button flag is turned ON in step S515, It returns to (1) of FIG. 4A. If the OK button has not been pressed, the process proceeds to step S514. Since the user presses the OK button or the cancel button at an arbitrary timing, the determinations in steps S513 and S514 are performed at the timing when the user presses the OK button or the cancel button.

In step S514, it is determined whether or not the cancel button is pressed on the entry defect checking screen (FIG. 10A). If the cancel button is pressed, the cancel button flag is turned ON in step S516, and the ( Return to 1). If the cancel button has not been pressed, the process returns to step S503 to continue checking for incompleteness. The case where the cancel button is pressed is the case where the entry defect check is ended.

This is the end of the description of the entry deficiency check process in FIG. By this processing, the check result of the entry defect check record corresponding to the form ID of the captured image is stored in the variable result[i]. Specifically, in the present embodiment, the value of [OK, NG, OK, NG, OK] is stored in the variable result[i].

FIG. 6 is an example of a flow in which the server 102 receives the image and the attribute information transmitted from the information processing apparatus 101 in step S424.
In step S601, the server 102 receives the application image, the supplementary image, the entry defect check completion flag (OFF, ON), the login user name, and the form name.

In step S602, the received image (the image of the application form that has been checked for incompleteness) and the supplementary image are stored in a predetermined folder, and the reception date and time, the form name, the login user name, and the reception application list table 1300 (FIG. 13) are stored. Add a record with the entry deficiency check completion flag (OFF/ON). As a result, the information checked for incompleteness in the information processing apparatus 101 is registered. After that, the operator on the server 102 side makes a detailed check, and if there is any deficiency, the person in charge corresponding to the login user name is called. The telephone uses, for example, the telephone number entered in the header of the received image.

FIG. 7 is an example of a flow of displaying the image and the attribute information received from the information processing apparatus 101 on the server 102 and confirming by the operator whether there is any omission.

In step S701, the reception application list table (FIG. 13) is read and the reception application list screen (FIG. 14) is displayed. At this time, if the server 102 side receives more than a predetermined number of application forms within a predetermined time (for example, if it receives images of 20 or more application forms within 5 minutes), the application form system is defective. It is determined that the unchecked application form should be prioritized and displayed, and the reception application list screen 1420 is displayed. Further, when the predetermined number or less of application forms are received within the predetermined time, the reception application form list screen 1410 is displayed. In the case of the mode in which the application form having no defect check is preferentially displayed, the server 102 generates a list in which the defect check is sorted by “not completed” and displays the list on the reception application list screen.

In step S702, it is determined whether or not the end button has been clicked on the reception application list screen (FIG. 14), and if the end button is pressed, the application confirmation process ends. If the end button is not clicked, it is determined in step S703 whether the form name is clicked.

In step S703, it is determined whether the form name has been clicked. If not, the process returns to step S702. If the form name is clicked, the application form received in step S704 is displayed on the reception application form confirmation screen (FIG. 15), and the operator on the server side visually checks for incompleteness.

At this time, if the application for which the deficiency check is completed is clicked on the reception application list screen (Fig. 14), the reception application confirmation screen (completed deficiency check) 1510 is displayed, and the application for which the deficiency check is not completed is clicked. If yes, the reception application confirmation screen (completed check of incompleteness of the entry) 1511 is displayed. Therefore, the operator on the server side can easily confirm whether or not the application form has already been checked for incompleteness in the system.
In step S705, the operator on the server side determines whether or not there is a defective portion in the application form.

If there is any incompleteness, the sender (login user) described in the header of the image of the application form is notified of the incompleted place, and after the completion, the application form is photographed and resent, and the process proceeds to step 708. To do. If there is no entry error, the sender (login user) described in the header of the application form is notified of no entry error, and the process proceeds to step 708.

It should be noted that since the telephone number is described in the header as an example, it is assumed in the present embodiment that the sender is contacted by telephone. The contact means may be described. Further, in the reception application confirmation screen of FIG. 15, when the deficiency contact button (not shown) is pressed, a window (a screen including a text box etc.) in which the deficiency content can be entered is displayed, and when the deficiency content is entered, the sender ( It is also possible to adopt a configuration in which the information is transmitted to the information processing apparatus 101 of the person in charge.

When the completion button is pressed on the reception application confirmation screen (FIG. 15) to close the screen in step S708, the record of the application displayed in step 704 is deleted from the reception application list table (FIG. 13) in step S709. Instead of deleting, the completion flag may be set and managed as confirmed.

FIG. 8 is an example of a processing flow of adding an additional image to the captured image of the application form by the information processing apparatus 101.
In step S801, the transmission image confirmation screen (FIG. 17) is displayed, and the number of additional images and the number of additional images are set to 0.

In step S802, the name of the store, the person in charge, the telephone number of the person in charge, the number of attached images, the status of completion of the deficiency check based on the result column of the deficiency check table 1610, etc. Information). Based on the acquired information, the captured image of the application form is added and displayed as a header image. Specifically, based on the acquired store name, person-in-charge name, person-in-charge phone number, number of attached images, and completion status of check for incompleteness of the entry, the image is formed in a predetermined size and inserted at the top of the image of the application form. That is, it shows an example of a process of displaying the number of supplementary images of the first image obtained by shooting together with the first image. In addition, it illustrates an example of a process of acquiring header information including checker information related to entry check and generating a first image including header information.
The image including this header information is transmitted to the server 102 as an image for which the entry defect has been checked.

The information added to the header may be any item, and is not limited to the above information. Although the telephone number is described as an example of the means for the operator of the server 102 to contact the sender, another contact means for the sender such as the mail address and system URL should be described in the header. It doesn't matter.

In step S803, it is determined which button has been pressed on the transmission image confirmation screen (FIG. 17). If the cancel button is pressed, the process returns to (2) of FIG. 4A, and the re-imaging process of the application form is performed in step S402. If the send button is pressed and there are additional images, the additional image number/number of additional images is added to the header of each additional image, and this processing flow ends.

If the incidental document image photographing button is pressed, an incidental image such as a license required for the application form is photographed in step S804. The captured supplementary image is stored in the memory. That is, it shows an example of a process of storing the incidental image obtained by photographing when the user gives an instruction to photograph the incidental image.
In step S805, the supplementary image captured in step S804 is displayed on the supplementary image confirmation screen (FIG. 18), and the number of supplementary images is incremented by one.
In step S806, the number of additional images is added to the additional image number of the captured additional image.
In step S807, it is determined which button has been pressed on the accompanying image confirmation screen (FIG. 18). If the OK button is pressed, the screen returns to the transmission image confirmation screen (FIG. 17).

If the cancel button is pressed, the number of additional images is incremented by 1 in step S808, 
 the additional image number of the captured additional image is cleared, and the screen returns to the transmission image confirmation screen (FIG. 17).

If the re-shooting button is pressed, the number of supplementary images is decremented by 1 in step S809, 
 the supplementary image number of the captured supplementary image is cleared, and the process returns to step S804 to re-capture the supplementary image.
The above is an example of the processing flow of capturing and adding an image incidental to the captured image of the application form.

As described above, according to the present embodiment, it is possible to facilitate shooting with a mobile terminal and improve the efficiency of entry check using images.

In addition, the entry check can be simply performed by the time when the entry check process by image recognition is completed, and convenience can be improved. By enabling a simple entry check using a cover sheet, it becomes possible to efficiently proceed with the work of many application forms.

Furthermore, if you are worried about a simple entry check (visual check) using the cover sheet, you can wait for the entry check of the system before you check it, and the checker (user) will choose which to choose. Therefore, it is more convenient and the efficiency of work can be improved.

Also, by attaching the contact information of the end user as header information to the application form and the accompanying images, it is possible for the operator on the server side to find a deficiency in the entry and easily contact the checker.

In addition, the operator on the server side can grasp whether the application form sent has already been checked for defects in the system, so the items that have not been checked for defects in the system will be given priority to check for defects. be able to.

As described above, the recording medium recording the program that realizes the functions of the above-described embodiments is supplied to the system or apparatus, and the computer (or CPU or MPU) of the system or apparatus executes the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by reading and executing.

In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium recording the program constitutes the present invention.

A recording medium for supplying the program includes, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a DVD-ROM, a magnetic tape, a non-volatile memory card, a ROM, an EEPROM, and a silicon. A disk or the like can be used.

Further, not only the functions of the above-described embodiments are realized by executing the program read by the computer, but also the OS (operating system) running on the computer is actually executed based on the instructions of the program. It goes without saying that a case where a part or all of the processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Furthermore, after the program read from the recording medium is written in the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the function expansion board is instructed based on the instruction of the program code. Needless to say, this also includes the case where the CPU or the like included in the function expansion unit performs some or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

Further, the present invention may be applied to a system including a plurality of devices or an apparatus including a single device. Further, it goes without saying that the present invention can be applied to the case where it is achieved by supplying a program to a system or an apparatus. In this case, by reading the recording medium storing the program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

The form of the program may be an object code, a program code executed by an interpreter, script data supplied to an OS (operating system), or the like.

Further, by downloading and reading the program for achieving the present invention from the server 102, the database, etc. on the network using the communication program, the system or apparatus can enjoy the effects of the present invention. It should be noted that the present invention also includes all configurations that combine the above-described embodiments and modifications thereof.

100 form system 101 information processing device 102 server 103 network

Claims (7)

A mobile terminal comprising: a first image pickup unit; and a second image pickup unit on a display unit side that displays an image obtained by the first image pickup unit,
Photographing execution means for executing photographing by the first imaging unit,
Face recognition means for performing face recognition using an image obtained from the second image pickup section,
Switching means for switching the photographing method by the photographing execution means based on the face recognition result by the face recognition means,
The photographing execution means, when the subject obtained by the first image pickup unit satisfies that the four corners of the subject which are the first photographing conditions are located along the preset guides, and a is not satisfied one of the photographing conditions satisfy that a position three corners of the object is a second imaging condition along a preset guides, one or, in the face recognition result of the face recognition unit A mobile terminal characterized in that, when shooting is possible based on the above, shooting is performed using the first shooting method. Said switching means, if the first object obtained by the imaging unit said first photographing conditions are satisfied, and, fill the second imaging condition the subject does not satisfy the first imaging condition, and when the face recognition unit can image based on the face recognition result of switching to the first imaging method, if the object does not satisfy the first imaging condition and the second imaging condition, and, the subject is not satisfied the first imaging condition satisfies the second imaging condition, and, if not capable of shooting based on the face recognition result of the face recognition unit, characterized in that switching to the second imaging method The mobile terminal according to claim 1. The first imaging method is automatic imaging using the first imaging unit, and the second imaging method is manual imaging using the first imaging unit. The mobile terminal described in. The mobile terminal according to claim 3, further comprising a check processing switching means for switching the imaging by the first imaging method, the checking process of the image captured by the imaging by the second imaging method. Before Kikao recognition means, wherein said in the case of displaying the image captured by the first imaging unit to the display unit, characterized by a face recognition using an image obtained by the second imaging unit The mobile terminal according to any one of items 1 to 4. A processing method for a mobile terminal, comprising: a first image capturing unit; and a second image capturing unit on a display unit side that displays an image obtained by the first image capturing unit,
An image capturing execution step of performing image capturing by the first image capturing unit;
A face recognition step of performing face recognition using an image obtained from the second imaging unit;
Based on the face recognition result by the face recognition step, including a switching step of switching the shooting method by the shooting execution step,
In the photographing execution step, when the subject obtained by the first imaging unit satisfies the first photographing condition that the four corners of the subject are at positions along the preset guides, and a is not satisfied one of the photographing conditions satisfy that a position three corners of the object is a second imaging condition along a preset guides, one or, in the face recognition result of the face recognition step A processing method characterized by a step of executing shooting using the first shooting method when shooting is possible based on the shooting method. A mobile terminal including a first image pickup unit and a second image pickup unit on a display unit side for displaying an image obtained by the first image pickup unit,
Photographing execution means for executing photographing by the first imaging unit,
Face recognition means for performing face recognition using an image obtained from the second image pickup section,
Based on the face recognition result by the face recognition means, function as a switching means for switching the photographing method by the photographing execution means,
If satisfied that the photographing execution unit, said first object obtained by the image pickup unit is located at a position four corners of the object is a first imaging condition along a preset guides, and the second a is not satisfied one of the photographing conditions satisfy that a position three corners of the object is a second imaging condition along a preset guides, one or, in the face recognition result of the face recognition unit A program for functioning as a means for performing shooting using the first shooting method when shooting is possible based on the above.

Images