JP6505937B1 - Matching system, matching method and matching program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve a matching success rate at the time of identity verification.
A verification system (100) includes a verification destination digital information storage unit (33) for storing verification destination digital information inputted in advance, and a character string of a verification source from verification source image data of a verification source object during photographing. A character information recognition unit 31 that recognizes and recognizes as collation source text data, and a collation unit 32 that collates collation destination digital information with collation source text data, and the character information recognition unit 31 and the collation unit 32 respectively The processing of recognition and the processing of collation are executed a plurality of times in the background, using a plurality of temporally consecutive collation source images.
[Selected figure] Figure 1

Description

  The present invention relates to a technique for collating digital information input in advance with information obtained from sensor information in a physical space.

  In the past, acts have been carried out to identify the person in various ways. For example, when a bank opens an account, identification is performed using identification documents such as a license. When a bank account is newly opened online, a new customer inputs his / her personal information (name, date of birth, address, etc.) into a web page input form and sends it to a bank-controlled collator. In addition, the new customer photographs the license on which the personal information is displayed with a camera such as a smartphone terminal and the like, in order to prove the authenticity of the input personal information based on the subsequent identification request from the bank. Is sent to the matching device.

  Thereafter, the verification device recognizes (OCR) the character string of the personal information from the received image of the license, and collates (OCV) the character string of the personal information with the character string of the personal information previously input. Then, when the character strings of the two pieces of personal information are identical to each other as a result of the comparison, it is determined that the person has been confirmed, and the account opening process is started.

  Note that OCR (Optical Character Recognition) is an optical character recognition technology that reads characters printed on a printed matter and outputs the read characters for processing of collation and control. Further, OCV (Optical Character Verification) is an optical character verification technique of comparing characters being printed and characters for verification, and determining whether or not they match. For example, they are described in Patent Document 1 and Non-Patent Document 1.

U.S. Patent No. 9396404

Nello Zuech, "Considerations in OCR / OCV Applications", [online], aia VISION ONLINE, GLOBAL ASSOCIATION FOR VISION INFORMATION, May 10, 2000 [Search on October 18, 2018], Internet, <https: / /www.visiononline.org/vision-resources-details.cfm/vision-resources/Considerations-in-OCR-OCV-Applications/content_id/1325> B. Lucas, one other person, "An image of image registration technique with an application to stereo vision", In Proc. Seventh International Conference on Artificial Intelligence, 1981, p. B. Horn, one other person, "Determining optical flow", Artificial Intelligence, vol. 17, 1981, p. 185-p. 203 Florian Schroff, 2 others, "A Unified Embedding for Face Recognition and Clustering", [online], arXiv org, GLOBAL ASSOCIATION FOR VISION INFORMATION, March 12, 2015, [October 18, 2018], Internet , <Https://arxiv.org/abs/1503.03832>

  However, in the conventional verification apparatus, emphasis is placed on enhancing responsiveness to a plurality of account opening requests, and OCR and OCV are executed once each, so that there is a first problem that the verification accuracy is low. For example, if the OCR fails in character recognition, the subsequent OCV may also fail.

  Further, in the conventional collation device, since the character is OCRed from the image based only on the character shape, the character recognition rate is significantly reduced when the character font in the image to be recognized is an unknown font for the OCR processing unit that performs OCR. There was also a second issue called.

  Furthermore, in the conventional verification apparatus, since the OCV is performed regardless of whether or not the verification document of verification source is authentic, even if the verification document of verification source is a fake, the authenticity of the person is determined There was also a third issue.

  The present invention has been made in view of the above circumstances, and it is a first object of the present invention to improve the verification success rate at the time of identity verification, and a second object to improve the character recognition rate. The third purpose is to make it possible to confirm the authenticity of the document.

  In order to solve the above problems, the verification system according to the present invention uses the verification destination digital information storage unit for storing the character string of the verification destination input in advance, and the verification source image of the verification source object during imaging. A character information recognition unit that recognizes a character string of a collation source, and a collation unit that collates the character string of the collation source with the character string of the collation destination, the character information recognition unit and the collation unit respectively The present invention is characterized in that the process of the recognition and the process of the matching are performed a plurality of times in the background, using a plurality of temporally consecutive reference source images.

  In the above collating system, when the collating unit continues to execute the collating process, only the part where the collating fails is to be processed.

  The collation system further includes a substitution detection unit that detects substitution of the collation source object based on a magnitude of a movement amount of the collation source object.

  The collation system further includes an image generation unit that generates an image including characters of the same font as the font of the character displayed on the collation source object.

  In the collation system, the image generation unit generates a collation image that generates a collation image including the same font as the font of the character displayed on the collation source object and the same character string as the collation destination character string A direct image matching unit that calculates an image similarity between the matching source image and the matching image.

  In the above matching system, the image generation unit generates a learning image including a plurality of learning character strings respectively displayed in a plurality of fonts including the same font as the font of the character displayed on the matching source object. An OCR learning unit for generating a learning OCR model that is an image generation unit for learning to recognize an image of a learning character string of each font as the learning character string using the learning image; Further, the character information recognition unit is characterized by recognizing a character string of a collation source from the collation source image using the learning OCR model.

  In the above-described matching system, a determination unit that determines the authenticity of the matching source object based on whether the feature of the matching source object included in the matching source image matches the feature of the matching source object. And the like.

  The collation method according to the present invention is, in the collation method performed by the collation system, a first step of storing a character string of a collation destination inputted in advance, and a collation source from a collation source image of a collation source object during photographing. And a third step of checking the character string of the collation source and the character string of the collation destination, and the second step and the third step The present invention is characterized in that the process of the recognition and the process of the matching are performed a plurality of times in the background using a plurality of the matching source images that are temporally continuous with each other.

  A verification program according to the present invention causes a computer to function as the verification system.

  According to the present invention, the verification success rate can be improved at the time of identity verification. In addition, at the time of identity verification, the character recognition rate can be improved. Further, at the time of identity verification, the authenticity of the identity verification document can be made confirmable.

It is a figure showing functional block composition of a collation system concerning a 1st embodiment. It is a figure which shows the processing flow of collation processing operation performed by a terminal device. It is a figure which shows the example of a collation origin object. It is a figure which shows the processing flow of collation processing operation performed with a collation apparatus. It is a figure which shows the example of an input form screen and collation destination digital information. It is a figure which shows the example of a display of a collation result. It is a figure which shows the processing flow of the change detection operation | movement performed by a terminal device. It is a figure which shows the functional block structure of the collation system (direct collation form) which concerns on 2nd Embodiment. It is a figure which shows the processing flow in the direct collation form performed by a collation apparatus. It is a figure which shows the functional block structure of the collation system (pre-learning form) which concerns on 2nd Embodiment. It is a figure which shows the processing flow in the prior learning form performed with a collation apparatus. It is a figure showing functional block composition (modification) of a collation system (direct collation form) concerning a 2nd embodiment. It is a figure which shows the functional block structure of the collation system which concerns on 3rd Embodiment. It is a figure which shows the basic processing flow of the authenticity determination performed by a collation apparatus. It is a figure which shows the processing flow of the modification of the authenticity determination performed by a collation apparatus.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

First Embodiment
In the first embodiment, in order to solve the first problem, as a first feature of the present invention, the OCR and the OCV are executed a plurality of times in the background. This improves the matching success rate. In addition, if all areas (name, date of birth, address, etc.) are targeted for matching every time, the matching success rate drops significantly, and therefore, as a feature related to the first feature, the area for which matching has once succeeded is skipped And focus on the areas that failed or unmatched. This collation is applied not only to the case where the areas to be collated differ but also to parts within the same area. Furthermore, as a feature related to the first feature, it is detected that the real object in the real space being captured is replaced with another object during matching.

(Configuration of collation system)
FIG. 1 is a diagram showing a functional block configuration of a verification system 100 according to the first embodiment. The collation system 100 is configured to include the terminal device 1 and the collation device 3. The terminal device 1 and the verification device 3 are physically and electrically connected to be communicable with each other via the communication network 5.

(Configuration of terminal device)
First, the configuration of the terminal device 1 will be described. As shown in FIG. 1, the terminal device 1 is configured to include an imaging unit 11, a display unit 12, a replacement detection unit 13, and a previous frame storage unit 14. The terminal device 1 is a client device on the local side, and is, for example, a smartphone terminal, a mobile phone terminal, or a tablet terminal to which a camera function and a communication function are attached. It may be a personal computer terminal to which a web camera is connected.

  The photographing unit 11 photographs the collation source object using an optical camera or the like, reads the collation source image data of the collation source object being photographed at a relatively short time interval (1 second, etc.), and collates the collation source image data. It has a function of outputting to the change detection unit 13 each time it is read (for example, one second) and outputting the collation source image data to the character information recognition unit 31 of the collation device 3 at relatively long time intervals (for example, six seconds). The imaging unit 11 can be realized, for example, using a camera attached to a smartphone terminal. The collation source target object is, for example, an identification document in which personal information (name, date of birth, address, etc.) such as a license, a health insurance card, etc. is recorded.

  The display unit 12 has a function of displaying the collation source image data of the collation source object photographed by the photographing unit 11 so as to be referable. For example, it is realizable using the display panel and display function with which a smart phone terminal is provided.

  The substitution detection unit 13 has a function of detecting substitution of the collation source object based on the magnitude of the movement amount of the collation source object during shooting. Specifically, the replacement detection unit 13 stores the collation source image data output immediately before from the imaging unit 11 as the previous frame in the previous frame storage unit 14, and the time span in which the replacement can not occur (1 second, etc.) At each reading of verification source image data), the optical flow of the verification source object is calculated using the verification source image data output from the photographing unit 11 and the previous frame of the previous frame storage unit 14, and the verification source object is calculated. Calculate the movement distance of the object. Then, when the moving distance of the verification source object exceeds the threshold value, it is determined that the replacement of the verification source object has occurred, and an alert signal is output to the display unit 12 to display alert information.

  The optical flow is a technology for expressing the amount of movement of an object as vector data between two images captured at different times. As a typical calculation method of optical flow, for example, the Lucas-Kanade method (Non-patent document 2) and the Horn-Shunck method (Non-patent document 3) exist.

  The previous frame storage unit 14 has a function of storing the comparison source image data of the previous frame in order to calculate the optical flow using the comparison source image data of the next frame by the replacement detection unit 13.

(Configuration of collation device)
Next, the function of the collation device 3 will be described. As shown in FIG. 1, the collation device 3 includes a character information recognition unit 31, a collation unit 32, a collation target digital information storage unit 33, and a collation result storage unit 34. The collation device 3 is, for example, a server device, and is disposed in the cloud.

  The character information recognition unit 31 repeatedly executes processing (OCR) for recognizing the character string of the collation source from the collation source image data every time the collation source image data is received from the photographing unit 11 of the terminal device 1. It has a function to output. Specifically, each time the character information recognition unit 31 receives collation source image data output from the photographing unit 11 of the terminal device 1 at relatively long time intervals (eg, 6 seconds), the character information recognition unit 31 receives the collation source image data. The received verification source image data is converted into text and output as verification source text data using the existing OCR technology.

  The character information recognition unit 31 also has a function of accepting the next collation source image data for each time and continuing the character recognition processing, when it takes time to recognize the character from the collation source image data. For example, in the case where it takes 2 seconds to recognize characters from the collation source image data, the next recognition processing is performed with an interval of 2 seconds or more.

  The character information recognition unit 31 may be disposed in the terminal device 1.

  The collation unit 32 performs processing of collating the character string of the collation source with the character string of the collation destination stored in the collation destination digital information storage unit 33 every time the character string of the collation source is received from the character information recognition unit 31 ( OCV is repeatedly executed, and a function of outputting the collation result to the display unit 12 of the terminal device 1 is provided. Specifically, the collation unit 32 includes the collation source text data converted into text by the character information recognition unit 31, and the collation destination text information (collation target character string, which will be described later) of the collation destination digital information storage unit 33. Are stored in the collation result storage unit 34. The matching result is stored in the matching result storage unit 34. The matching result is stored in the matching result storage unit 34. Do.

  In addition, the collating unit 32 has a function of processing only a portion that fails to be collated or an unprocessed portion when continuing the collating process. Specifically, when the collation unit 32 refers to the collation result up to this point stored in the collation result storage unit 34 and executes the collation processing next, the collation part 32 is already a collated part (collation OK). To skip without matching. This collation is applied not only to the case where the areas to be collated differ but also to parts within the same area. Then, when all the areas have been collated, a signal of “all collated” is output and the process is ended.

  The collation destination digital information storage unit 33 has a function of storing, as collation destination digital information, a collation destination character string (character code string) which has been input in advance to a web page input form using a keyboard or the like. The collation destination digital information is, for example, a name, a date of birth, an address, and the like.

  The collation result storage unit 34 has a function of storing the collation result of each area so as to be referable each time the collation unit 32 performs the collation processing. When the collation unit 32 performs the collation process on the parts in one area, the collation result storage unit 34 stores the collation result of each part.

(Operation of collation system)
(Verification processing operation)
Next, the collation processing operation performed by the collation system 100 will be described with reference to FIGS. This collation processing operation is performed in the background. FIG. 2 is a diagram showing a processing flow of the collation processing operation performed by the terminal device 1. FIG. 3 is a view showing an example of the reference object. FIG. 4 is a diagram showing a processing flow of the collation processing operation performed by the collation device 3. FIG. 5 is a diagram showing an example of an input form screen and collation destination digital information. FIG. 6 is a view showing a display example of the comparison result.

  In the processing operation to be described later, including the second embodiment and later, a new customer who wants to open a bank account online uses the license as the verification source object and uses the personal information entered in the input form for the authenticity of the personal information The case of proving will be described. The new customer inputs personal information such as name, date of birth, address and the like in an input form (upper view in FIG. 5) of the bank homepage, and sends it in advance to the verification device 3 managed by the bank. Then, the collation device 3 stores the name and the like in the collation destination digital information storage unit 33 as collation destination digital information (the lower diagram in FIG. 5). The reason for using “as the collation destination digital information” is that the input name or the like is a character code that can be controlled by the collation device 3 which is a computer.

(Verification processing operation; Operation of terminal device)
First, the operation of the terminal device 1 will be described with reference to FIG.

Step S101;
When the new customer starts photographing a license (FIG. 3) on the terminal device 1 in order to prove that the name etc. entered in the input form is correct, the photographing unit 11 of the terminal device 1 receives an image of the license The display unit 12 displays an image of the driver's license while reading as verification source image data every one second. In addition, 1 second is an example and 0.1 second etc. may be sufficient.

Step S102;
Next, the photographing unit 11 sequentially outputs the collation source image data to the character information recognition unit 31 of the collation device 3 every six seconds. At this time, the imaging unit 11 sequentially outputs each of the collation source image data (every one second) to the switching detection unit 13. The operation of the switching detection unit 13 will be described later. Note that six seconds is an example, and may be ten seconds or the like.

Step S103;
Finally, when the display unit 12 receives the verification result from the verification device 3, the display unit 12 superimposes and displays the verification result on the image of the driver's license during shooting (FIG. 6).

(Verification processing operation; Operation of verification device)
Next, the operation of the collation device 3 will be described with reference to FIG.

Step S201;
First, the character information recognition unit 31 of the collation device 3 receives collation source image data from the photographing unit 11 of the terminal device 1.

Step S202;
Next, the character information recognition unit 31 converts the image character contained in the received collation source image data into a text character using the existing OCR technology (character recognition processing; OCR), thereby the collation source image data. Generate the text collated source text data. Thereafter, the character information recognition unit 31 outputs the collation source text data to the collation unit 32.

  Note that each time the character information recognition unit 31 receives the collation source image data from the terminal device 1, the character information recognition unit 31 repeatedly executes step S201 and step S202. In this example, since the collation source image data is received from the terminal device 1 every six seconds, the character information recognition unit 31 sequentially outputs collation source text data to the collation unit 32 every six seconds.

Step S203;
Next, when the collation unit 32 receives the collation source text data from the character information recognition unit 31, the collation unit 32 reads the collation destination digital information (the lower diagram in FIG. 5) from the collation destination digital information storage unit 33 and collates with the collation source text data. The destination digital information is collated (character collation processing; OCV). At this time, the collating unit 32 can also match the entire one area to be matched for each area, and can also match each part in the area. For example, in the case where an address is taken as one area, all the character strings of the address (address 00 of ○ prefecture 丁 city 01) may be matched, or some character strings included in the address (〇 prefecture 〇 ○ City) may match.

Step S204;
Thereafter, the collation unit 32 sequentially transmits the collation result obtained by collating the respective areas to the terminal device 1 for each area, and stores the collation result in the collation result storage unit 34. As described in step S103, the terminal device 1 superimposes and displays the verification result from the verification device 3 on the image of the license under imaging (FIG. 6).

  The left side of FIG. 6 shows an example of the first screen transition displayed on the display unit 12 of the collation device 3. First, since the name is collated and collation is OK, “collation OK” is superimposed and displayed in the name column of the driver's license image. Next, the date of birth is collated, and it takes time to perform collation processing, and the collation result is not obtained, so “(not collated)” is superimposed and displayed in the birth date column. Next, the address is collated, and since the collation is NG, “collation NG” is superimposed on the address column. Note that, in the case of matching failure, for example, it may be considered that the text character can not be correctly recognized from the matching source image data in step S202.

Step S205;
Next, the collation unit 32 compares the current number of collations with a threshold indicating the upper limit of the number of collations, and when the current number of collations exceeds the threshold, ends the processing and the current number of collations exceeds the threshold If not, the process proceeds to step S206.

Steps S206, S207;
After that, the collation unit 32 refers to the collation result storage unit 34 and determines whether there is a collation NG or an uncollated area. If there is an area of collation NG or uncollation, the process returns to step S203 in order to collate the area, and the collation unit 32 matches the collation destination digital information with the second collation source text data. At this time, the collation unit 32 skips the collation OK area without collation, and collates only the collation NG or the uncollated area. On the other hand, if there is no collation NG or uncollated area, that is, if all the collations are OK, the processing is ended.

  The right side of FIG. 6 shows an example of the second screen transition. The processing of collation is not performed for the name for which the collation is OK, and the processing of collation is started from the date of birth, while the display of the first collation result is continuously displayed. Then, the date of birth is collated, and since the collation is OK, “collation OK” is superimposed and displayed in the date of birth column. Next, the address is collated, and since the collation is OK, “collation OK” is superimposed on the address column. The reason why the verification result is changed to verification OK in the second verification is considered to be that the text character was correctly recognized by using the comparison source text data different from the first verification. That is, since the process of collation is repeatedly performed using a plurality of collation source text data corresponding to the same license, the collation success rate is improved.

(Replacing detection operation)
Next, with reference to FIG. 7, the operation of detecting the replacement of the reference source object will be described. FIG. 7 is a diagram showing a processing flow of the switching detection operation performed by the terminal device 1. If another license is replaced after collation OK is displayed for the name, a combination of nonexistent name and address will be generated. If the customer has a bad faith, entering the name and address corresponding to the combination into the input form will result in collation, which makes it possible to misrepresent the person. Therefore, in this operation, the replacement of the license is prevented, and the spoofing of the collator is prevented. The switching detection operation is performed only by the terminal device 1.

Step S301;
First, when shooting of a license with the terminal device 1 is started, the imaging unit 11 of the terminal device 1 reads an image of the license as verification source image data every second and changes every time it is read (every second). It sequentially outputs to the detection unit 13. One second is an example of a time span assuming that license replacement can not occur.

Step S302;
The replacement detection unit 13 stores the collation source image data output immediately before from the photographing unit 11 in the previous frame storage unit 14 as the previous frame, and then outputs the collation source image data output next (after one second) And the previous frame are used to calculate the optical flow of the license, and the movement distance of the license for one second is calculated.

Step S303;
Next, the replacement detection unit 13 compares the moving distance of the license with the threshold value indicating the variation distance of the position of the reference source object due to the switching, and if the moving distance of the license exceeds the threshold, the process proceeds to step S304. move on. In consideration of camera shake at the time of photographing, it is preferable to set the threshold to about 2 cm. On the other hand, if the one-second travel distance of the license does not exceed the threshold value, the process ends.

Step S304;
Finally, when the moving distance of the license exceeds a threshold, the replacement detection unit 13 determines that replacement of the license has occurred, and outputs an alert signal to the display unit 12 to display alert information. At this time, the change detection unit 13 may output an alert signal to a speaker provided in the terminal device 1 in addition to the display unit 12 to warn by sound, or may be output to the verification device 3.

  After step S303 or step S304, the change detection unit 13 repeatedly executes step S301 to step S304 each time the imaging unit 11 reads the collation source image data.

(effect)
According to the first embodiment, the collation system 100 collates the collation destination digital information storage unit 33 storing the collation destination digital information input in advance with the collation source image data of the collation source object during photographing. A character information recognition unit 31 that recognizes an original character string and sets it as collation source text data, and a collation unit 32 that collates collation destination digital information with collation source text data, and the character information recognition unit 31 and the collation unit Since the process 32 performs the process of recognition and the process of collation a plurality of times in the background using a plurality of temporally consecutive collation source images, the collation success rate can be improved.

  Further, according to the first embodiment, when the collation unit 32 continuously executes the collation processing, only the portion where the collation fails or the uncollated portion is the processing target, so the collation is successful. The matching success rate can be further improved because no matching failure or unmatching is made for the part.

  Further, according to the first embodiment, the replacement detection unit 13 detects replacement of the verification source object based on the magnitude of the movement amount of the verification source object, so replacement of the verification source object Can be detected reliably.

Second Embodiment
In the second embodiment, in order to solve the second problem, as a second feature of the present invention, a character to be collated is generated with a character font (typeface) conforming to the real object in the real space. This improves the accuracy of the OCR and OCV. Specifically, two methods are used.

  First, image data is generated by converting the verification destination digital information into the same character font as the character font displayed on the verification source object, and whether the image data is similar to the verification source image data (Direct collation form).

  The second is generating image data in which learning character strings such as an address list stored in advance are imaged with a plurality of character fonts including the character font displayed in the collation source object, and the image data If this is the case, the OCR learning to recognize as the learning character string of the image data is executed in advance, and the OCR is performed using the learned OCR model generated thereby (pre-learning form).

  Thereby, even if the character displayed on the collation source object is a font unknown to the character information recognition unit 31, it is realized that the collation is performed robustly.

(Direct collation form)
(Configuration of collation system)
FIG. 8 is a diagram showing a functional block configuration of a verification system (direct verification form) according to the second embodiment. The terminal device 1 includes a photographing unit 11 and a display unit 12. The collation device 3 includes a collation target digital information storage unit 33, a font information storage unit 35, a collation image generation unit 36, and an image direct collation unit 37. Hereinafter, each function part (35-37) of the collation apparatus 3 is demonstrated. The functional units (11, 12) of the terminal device 1 have the same functions as in the first embodiment, and thus the description thereof is omitted.

  The font information storage unit 35 has a function of readably storing font information of characters. The font information is, for example, Mincho type, Gothic type, etc.

  The collation image generation unit 36 has a function of generating an image including characters of the same font as the font of the character displayed on the collation source object. Specifically, the collation image generation unit 36 generates collation image data including the same font as the font of the character displayed on the collation source object and the same character string as the collation target character string. For example, if the character string "Tokyo" displayed on the collation source object is "Mincho", the character string "Tokyo" entered as collation destination digital information is Output as an image of "Tokyo".

  The image direct collation unit 37 has a function of calculating the image similarity between the collation source image data and the collation image data (image data generated by collation destination digital information according to font information). Specifically, the image direct matching unit 37 estimates and calculates the image similarity between the matching source image data and the matching image data, determines similarity and dissimilarity, and displays the determination result on the display unit 12 of the terminal device 1. Output to In addition, the method of calculating | requiring an image similarity is arbitrary, for example, the similar image retrieval method using triplet loss (nonpatent literature 4) based on deep learning, the average hash method performed without using deep learning, etc. exist. .

(Operation of collation system)
FIG. 9 is a diagram showing a processing flow in the direct collation mode performed by the collation device 3. In the first embodiment, the method in which the collating unit 32 collates the character strings has been described, but here, the image direct collating unit 37 performs the collating process between the images.

Step S401;
First, the collation image generation unit 36 acquires collation destination digital information (such as a name input to an input form) from the collation destination digital information storage unit 33, and refers to the font information of the font information storage unit 35 to obtain a license. The matching image data is generated by converting it into the same font as the font of the character displayed on the certificate. For example, in the case where the character font of the license is Mincho, the name etc. input to the input form is converted into a Mincho and image data for collation is generated. As a method of determining the font of the character of the license, for example, a method of designating a character font of the license by an input form, a method of detecting a font type when the license is photographed, and the like can be considered.

Step S402;
Next, when receiving the collation source image data from the photographing unit 11 of the terminal device 1, the image direct collation unit 37 calculates the image similarity between the received collation source image data and the collation image data generated in step S401. And determine similarity and dissimilarity. For example, as a result of calculating the image similarity between two image data, if the similarity is equal to or higher than a threshold, it is determined that the two are similar and the matching is OK. On the other hand, when the similarity is less than the same threshold, it is determined as non-similar and it is determined as matching NG.

Step S403;
Finally, the image direct matching unit 37 transmits the matching result in step S402 to the terminal device 1. As described in the first embodiment, the terminal device 1 superimposes and displays the verification result from the verification device 3 on the image of the license under imaging.

(Form of prior learning)
(Configuration of collation system)
Next, the pre-learning form will be described. FIG. 10 is a diagram showing a functional block configuration of a collation system (pre-learning form) according to the second embodiment. The terminal device 1 includes a photographing unit 11 and a display unit 12. The collation device 3 has a font information storage unit 35, a learning character string digital information storage unit 38, a learning image generation unit 39, and an OCR learning unit 40, in addition to the configuration described in the first embodiment. And a trained OCR model storage unit 41. Hereinafter, each function part (35, 38-41) of the collation apparatus 3 is demonstrated. The functional units (11, 12) of the terminal device 1 have the same functions as in the first embodiment, and thus the description thereof is omitted.

  The font information storage unit 35 has a function of readably storing font information of characters. The font information is, for example, Mincho type, Gothic type, etc.

  The learning character string digital information storage unit 38 has a function of readably storing any learning character string collected for OCR learning as learning character string digital information. The character string digital information for learning is, for example, a name list such as Nippon Taro, an address list such as Chiyoda-ku, Tokyo, a date of birth list such as January 1, 2000, etc. It is known information that is likely to be used as etc.

  The learning image generation unit 39 has a function of generating a learning image including a plurality of learning character strings respectively displayed in a plurality of fonts including the same font as the font of the character displayed on the collation source object. Specifically, the learning image generation unit 39 generates learning image data in which the arbitrary learning character string is displayed in a plurality of fonts. For example, the character string of the address list included in the learning character string digital information is converted to the "Mincho form" and output as an image of the "Mincho form" address list.

  The OCR learning unit 40 has a function of generating a learning OCR model for learning to recognize an image of a learning character string of each font as the learning character string using the learning image data. For example, a learning OCR model is generated which learns to recognize the character string of the address list from the image of the address list of “Mincho”.

  The learned OCR model storage unit 41 has a function of readably storing the learning OCR model generated by the OCR learning unit 40.

(Operation of collation system)
FIG. 11 is a diagram showing a process flow in the pre-learning mode performed by the verification device 3. In the first embodiment, the method of performing the OCR based on only the character shape of the character displayed on the verification source object when performing the OCR has been described, but here, the font (typeface) of the character displayed on the verification source object Is used as key information in character recognition for OCR.

Step S501;
First, the learning image generation unit 39 acquires learning character string digital information from the learning character string digital information storage unit 38, and generates learning image data converted to the same font as the font information of the font information storage unit 35. Do. For example, when there is a Mincho font in font information, learning image data of a name list based on the Mincho font is generated for each name. An address list, a date of birth list, etc. are similarly generated. More specifically, for example, when "Tanaka" and "Sato" are included in the name list and the font information is "Mincho type" and "Block type", the image of Mincho type "Tanaka" and the block type "Tanaka" The image of "", the image of "Sato" of Mincho type, and the image of "Sato" of block type are generated and output. As font information, a method of designating a font of the same type as a font of a license in an input form, a method of designating an arbitrary font, or the like can be considered.

Step S502;
Next, the OCR learning unit 40 uses the learning image data generated in step S501 to perform learning to recognize (determine / convert) each learning character string as long as it is an image of a learning character string for each font. Then, the learning result is generated as a learning OCR model and stored in the learned OCR model storage unit 41. For example, if the image is "Tanaka" in the morning, the character string "Tanaka" is recognized, and an OCR model for learning that recognizes the character string "Tanaka" is also generated for the image "Tanaka" in the block type. .

  The collation device 3 executes (preliminary learning) steps S501 and S502 in advance for all the learning character strings and all the fonts.

Step S503;
Thereafter, when the character information recognition unit 31 receives the collation source image data from the photographing unit 11 of the terminal device 1, the character information recognition unit 31 has learned about each image such as the name, date of birth, and address of a predetermined font included in the collation source image data. Among the plurality of images included in the OCR model, images such as the same font name, date of birth, address, etc. are specified respectively, and the received matching source is specified by respectively specifying the learning character string corresponding to each specified image. Recognize character strings (OCR) from image data. For example, in the case where "Mooncho""Tanaka" is displayed in the verification source image data, the character string "Tanaka" is specified from the learned OCR model. Thereby, the character information recognition unit 31 converts the image character included in the collation source image data into a text character.

Step S504;
Thereafter, the process described in step S202 (the process of outputting the collation source text data to the collation unit 32) to the process described in step S207 is executed.

(effect)
According to the second embodiment, the collation image generation unit 36 generates collation image data including the same font as the font of the character displayed on the collation source object and the same character string as the collation destination digital information. Since the image direct collation unit 37 calculates the image similarity with the collation source image data using the collation image data, even if the character displayed in the collation source object is an unknown font, the robustness is The character recognition rate can be improved.

  Further, according to the second embodiment, the learning image generation unit 39 includes a plurality of learning character strings displayed respectively in a plurality of fonts including the same font as the font of the character displayed on the collation source object. A learning OCR model is generated in which learning image data is generated, and the OCR learning unit uses the learning image data to recognize an image of a learning character string of each font as the learning character string. Since the character information recognition unit 31 recognizes the character string of the collation source from the collation source image using the learning OCR model, even if the character displayed on the collation source object is an unknown font , It is possible to perform the collation robustly and improve the character recognition rate.

  In addition, it is also possible to combine the direct matching form and the prior learning form in the first embodiment. For example, as illustrated in FIG. 12, the change detection unit 13 and the previous frame storage unit 14 are added to the terminal device 1 to the direct verification form shown in FIG. You may do so.

Third Embodiment
In the third embodiment, in order to solve the third problem, as a third feature of the present invention, a real object is sensed three-dimensionally, and it is confirmed whether there is physical deviation from the real object which should be originally To determine the authenticity of the real object.

(Configuration of collation system)
FIG. 13 is a diagram showing a functional block configuration of the collation system according to the third embodiment. The configuration of the terminal device 1 is the same as that of the first embodiment. The verification device 3 includes the authenticity determination unit 42, the verification source physical information storage unit 43, the biometric information authenticity determination unit 44, and the verification source biometric information storage unit 45 in addition to the configuration described in the first embodiment. And is comprised. The third embodiment can be combined with the second embodiment.

  The authenticity determination unit 42 has a function of determining the authenticity of the verification source object based on whether the feature of the verification source object included in the verification source image data matches the feature of the verification source object. . Specifically, if the physical feature of the reference source object matches the physical feature of the example object, the true / false determination unit 42 indicates that the reference source object is a real object. The determination result is output, and when it does not match, a false determination result indicating that the matching source object is a fake is output.

  The collation source physical information storage unit 43 has a function of storing information related to the original physical feature of the collation source object as a collation source physical information so as to be referable. The verification source physical information is, for example, the shape, size, thickness, content of display information, color, etc. of a license.

  The biometric information authenticity determination unit 44 determines the authenticity of the verification source object based on whether the feature of the verification source object included in the verification source image data matches the feature of the verification source object. Equipped with Specifically, the biometric information authenticity determination unit 44 indicates that the verification source object is a real object when the biometric feature of the verification source object matches the biometric feature of the example object. A true determination result is output, and when it does not match, a false determination result indicating that the matching source object is a fake is output.

  The collation source biometric information storage unit 45 has a function of storing information relating to the inherent biometric feature of the collation source object as a collation source biometric information so as to be referable. The collation source biometric information is, for example, a face, an outline of a face related to a fingerprint or the like, a fingerprint shape or the like.

(Operation of collation system)
(basic action)
First, the basic operation of authenticity determination will be described with reference to FIG. FIG. 14 is a diagram showing a basic processing flow of authenticity determination performed by the verification device 3. Here, in addition to the information input to the input form, the authenticity is determined based on the physical features of the matching source object.

Step S601;
First, when the authenticity determination unit 42 receives the verification source image data from the imaging unit 11 of the terminal device 1, the authenticity determination unit 42 calculates a physical feature such as a license included in the verification source image data. For example, in the case of a license, the positional relationship such as the position of a black line frame displayed in the license, the arrangement of photographs, etc. is calculated with respect to the appearance of the license. When the verification source object is a three-dimensional shape, three-dimensional physical features are calculated using verification source image data continuously captured from a plurality of angles. For example, in the case of a card like a license, in addition to the physical features of the front and back surfaces, three-dimensional features (such as height) in the thickness direction are also calculated.

Step S602;
Next, the authenticity determination unit 42 acquires, from the verification source physical information storage unit 43, information on the original physical features of the license which is the verification source physical feature, and obtains the license etc. obtained from the verification source image data. It is determined whether the physical feature matches the match source physical feature.

  As a method of determining the coincidence / non-pairing of the physical features, there can be considered a method in which feature amounts which are manually determined in advance are obtained, and the coincidence with the photographed physical features is obtained. For example, in the case of a card-like object, a characteristic that a black straight line runs vertically and horizontally from a point 1 cm to the right and 0.5 cm downward from the upper left vertex is used as an inherent physical feature. Besides, it is possible to use both deep learning such as CNN (Convolutional Neural Network) to find and calculate the original physical feature automatically, and to use it as a means to determine the degree of similarity. Any method may be used.

Step S603;
Finally, the authenticity determination unit 42 transmits the authenticity determination result of step S602 to the terminal device 1. For example, the authenticity determination unit 42 transmits a false determination result indicating that the license being photographed is a fake, when the physical characteristics of the two do not match or the similarity is lower than a threshold.

(Modification of basic operation)
Next, the operation (modification) of determining the authenticity will be described with reference to FIG. FIG. 15 is a diagram showing a process flow of a modification of the authenticity determination performed by the verification device 3. Here, in addition to the information input to the input form, the authenticity is determined based on the biological feature of the matching source object.

Step S701;
First, when the biometric information authenticity determination unit 44 receives the verification source image data from the photographing unit 11 of the terminal device 1, the biometric information authenticity determination unit 44 calculates the feature of the person's face from the photograph of the license included in the verification source image data. For example, the biometric information authenticity determination unit 44 calculates the position of the eye, the position of the nose, the shape of the face, and the like.

Step S702;
Next, the biometric information authenticity determination unit 44 acquires, from the verification source biometric information storage unit 45, information related to the biometric feature of the person who is the verification source biometric information, and the biometric feature of the person obtained from the verification source image data is , It is determined whether or not the matching source biometric information is matched. As a method of determining whether or not the biological characteristics match, for example, the method described in Non-Patent Document 3 or the like is used. The verification source biometric information is calculated in advance using images continuously captured from a plurality of angles in consideration of the three-dimensional shape.

(effect)
According to the third embodiment, the authenticity determination unit 42 determines whether the physical feature of the matching source object included in the matching source image data matches the physical feature of the matching source object. Since the authenticity of the verification source object is determined, the authenticity of the verification source object can be determined reliably. In particular, since a three-dimensional physical feature is used, for example, when a copy of a license is used, it can be determined as false.

  Further, according to the third embodiment, whether the biometric information included in the verification source object included in the verification source image data matches the biometric characteristic of the person according to the third embodiment. Since the authenticity of the verification source object is determined on the basis of this, it is possible to reliably determine the authenticity of the verification source object, such as replacement of a photograph part, and further, verification biometric using alternative photographs instead of actual biometric information. It is possible to prevent the replacement of information.

<Use>
In each embodiment, although the case where a bank account was opened was explained to an example, it is possible to apply to various scenes which perform a personal check processing using a personal check document.

<Others>
The collation system 100 described in the present embodiment can be realized by a computer provided with a CPU, a memory, a hard disk and the like. It is also possible to create a verification program for causing a computer to function as the verification system 100 and a storage medium for the verification program.

100 verification system 1 terminal device 11 photographing unit 12 display unit 13 substitution detection unit 14 front frame storage unit 3 verification device 31 character information recognition unit 32 verification unit 33 verification target digital information storage unit 34: collation result storage unit 35: font information storage unit 36: collation image generation unit 37: image direct collation unit 38: learning character string digital information storage unit 39: learning image generation unit 40: OCR learning unit 41: learning OCR model storage unit 42 ... authenticity determination unit 43 ... verification source physical information storage unit 44 ... biometric information authenticity determination unit 45 ... verification source biometric information storage unit

Claims (9)

A collation destination digital information storage unit for storing a character string of a collation destination input in advance;
A character information recognition unit that recognizes a character string of a collation source from a collation source image of a collation source object being photographed;
And a collating unit that collates the character string of the collation source with the character string of the collation destination,
The character information recognition unit and the check unit are respectively
A verification system characterized by performing processing of said recognition and processing of said collation in multiple times in the background using a plurality of said collation original images which continue temporally. The collation unit
The collation system according to claim 1, wherein when the processing of the collation is continuously executed, only the part which fails in the collation is a processing target.   The collation system according to claim 1 or 2, further comprising a substitution detection unit that detects substitution of the collation source object based on the magnitude of the movement amount of the collation source object.   The collation system according to any one of claims 1 to 3, further comprising an image generation unit that generates an image including a character of the same font as the font of the character displayed on the collation source object. The image generation unit is a collation image generation unit that generates a collation image including the same font as the font of the character displayed on the collation source object and including the same character string as the collation destination character string,
5. The collation system according to claim 4, further comprising an image direct collation unit which calculates an image similarity between the collation source image and the collation image. The image generation unit is a learning image generation unit that generates a learning image including a plurality of learning character strings respectively displayed in a plurality of fonts including the same font as the font of the character displayed on the collation source object. There,
The image processing apparatus further comprises an OCR learning unit that generates a learning OCR model that uses an image for learning character string of each font to be recognized as the learning character string using the image for learning.
The character information recognition unit
5. The collation system according to claim 4, wherein the character string of the collation source is recognized from the collation source image using the learning OCR model.   It further comprises a determination unit that determines the authenticity of the verification source object based on whether the feature of the verification source object included in the verification source image matches the feature of the verification source object. The collation system according to any one of claims 1 to 6, characterized in that: In the collation method performed by the collation system,
A first step of storing a character string of a collation destination inputted in advance;
A second step of recognizing a character string of a collation source from a collation source image of a collation source object being photographed;
Performing a third step of collating the character string of the collation source with the character string of the collation destination;
In each of the second step and the third step,
A collation method comprising: executing a process of the recognition and the process of the collation a plurality of times in the background by using a plurality of temporally consecutive reference images.   A collation program which causes a computer to function as the collation system according to any one of claims 1 to 7.

Images