JP2020086922A - Collation system, collation method, and collation program - Google Patents

Abstract

To improve a collation success rate in a person confirmation.SOLUTION: A collation system 100 comprises: a collation destination digital information storage unit 33 that stores collation destination digital information to be input in advance; a character information recognition unit 31 that recognizes a character string of a collation source from collation source image data of a collation source object during a photographing to obtain collation source text data; and a collation unit 32 that collates the collation destination digital information with the collation source text data. The character information recognition unit 31 and the collation unit 32 execute a process of the recognition and the collation multiple times in a background by using a plurality of collation source images which is continued in time-sequence.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for collating digital information input in advance with information obtained from sensor information in a physical space.

BACKGROUND ART Conventionally, an act of confirming an individual has been performed by various methods. For example, when opening an account at a bank, identity verification is performed using identity verification documents such as a license. When opening a new bank account online, the new customer enters his/her personal information (name, date of birth, address, etc.) in the input form on the web page and sends it to the matching device under bank control. In addition, the new customer, in order to prove the authenticity of the personal information entered based on the subsequent identity verification request from the bank, photographs the license with the personal information displayed with a camera such as a smartphone terminal, and the image is displayed. To the collation device.

After that, the collation device recognizes (OCR) the character string of the personal information from the received image of the license, and collates (OCV) the character string of the personal information with the character string of the personal information previously input. Then, as a result of the collation, when the character strings of the two pieces of personal information are the same as each other, it is determined that the person has been confirmed, and the account opening process is started.

Note that OCR (Optical Character Recognition) is an optical character recognition technology for reading a character printed on a printed matter and outputting the character for collation and control processing. Further, OCV (Optical Character Verification) is an optical character matching technique that matches printed characters with matching characters to determine whether they match. For example, it is described in Patent Document 1 and Non-Patent Document 1.

US Patent 9396404

Nello Zuech, "Considerations in OCR/OCV Applications", [online], aia VISION ONLINE, GLOBAL ASSOCIATION FOR VISION INFORMATION, May 10, 2000, [Search October 18, 2018], Internet, <https:/ /www.visiononline.org/vision-resources-details.cfm/vision-resources/Considerations-in-OCR-OCV-Applications/content_id/1325> B. Lucas, 1 person from outside, "An iterative image registration technique with an application to stereo vision", In Proc. Seventh International Conference on Artificial Intelligence, 1981, p.674-p.479 B. Horn, 1 person outside, “Determining optical flow”, Artificial Intelligence, vol.17, 1981, p.185-p.203 Florian Schroff, 2 others, “A Unified Embedding for Face Recognition and Clustering”, [online], arXiv org, GLOBAL ASSOCIATION FOR VISION INFORMATION, March 12, 2015, [Search October 18, 2018], Internet , <https://arxiv.org/abs/1503.03832>

However, in the conventional collation device, the first problem is that the collation accuracy is low because the OCR and the OCV are executed once each time with an emphasis on improving the responsiveness to a plurality of account opening requests. For example, if the character recognition by the OCR fails, the OCV after that may also fail.

Further, in the conventional collating device, characters are OCR'd from the image based on only the glyphs. Therefore, when the character font in the image to be recognized by the OCR processing unit for OCR is an unknown font, the character recognition rate is significantly reduced. There was also the second issue.

Further, in the conventional collation device, since the OCV is performed regardless of whether or not the identification document of the collation source is authentic, the authenticity of the person is confirmed even when the identification document of the collation source is a fake. There was also a third issue.

The present invention has been made in view of the above circumstances, and a first object is to improve the collation success rate and a second object is to improve the character recognition rate at the time of identity verification. The third purpose is to make it possible to confirm the authenticity of a document.

In order to solve the above problems, the collation system according to the present invention includes a collation destination digital information storage unit that stores a character string of a collation destination that is input in advance, and a collation source image of a collation source target being photographed. A character information recognition unit that recognizes a character string of a collation source, and a collation unit that collates the character string of the collation source and the character string of the collation destination, the character information recognition unit and the collation unit, respectively. The recognition process and the verification process are performed multiple times in the background by using a plurality of the verification source images that are temporally continuous.

In the above collation system, when the collation unit continues to execute the collation process, only the part in which the collation fails is processed.

The collation system is characterized by further comprising a replacement detection unit that detects replacement of the verification source object based on the amount of movement of the verification source object.

The collation system is characterized by further comprising an image generation unit that generates an image including a character having the same font as the character font displayed on the collation source object.

In the above collation system, the image generation unit generates a collation image that generates a collation image including a font that is the same as the font of the characters displayed on the collation source object and that includes the same character string as the collation destination. The image direct collating unit that calculates the image similarity between the collation source image and the collation image is further included.

In the above collation system, the image generation unit is configured to perform learning for generating a learning image including a plurality of learning character strings displayed in a plurality of fonts including the same font as the character font displayed in the collation source object. And an OCR learning unit that generates a learning OCR model that learns by using the learning image to recognize an image of a learning character string of each font as the learning character string. Further, the character information recognition unit is characterized by recognizing a character string of a collation source from the collation source image using the learning OCR model.

In the above matching system, a determination unit that determines the authenticity of the matching source target object based on whether or not the characteristics of the matching source target object included in the matching source image match the features included in the matching source target object. Is further provided.

In the matching method according to the present invention, in the matching method performed by the matching system, a first step of storing a character string of a matching destination that is input in advance, and a matching source image from a matching source image of a matching source target being photographed. The second step of recognizing the character string of and the third step of matching the character string of the collation source with the character string of the collation destination are performed, and in the second step and the third step, , The recognition processing and the matching processing are executed a plurality of times in the background by using the plurality of matching source images that are temporally continuous.

A verification program according to the present invention causes a computer to function as the verification system.

According to the present invention, the verification success rate can be improved at the time of identity verification. In addition, the character recognition rate can be improved at the time of identity verification. Further, at the time of identity verification, it is possible to confirm the authenticity of the identity verification document.

It is a figure which shows the functional block structure of the collation system which concerns on 1st Embodiment. It is a figure which shows the process flow of the collation process operation performed in a terminal device. It is a figure which shows the example of a matching object. It is a figure which shows the processing flow of the collation processing operation performed with a collation apparatus. It is a figure which shows the example of an input form screen and collation destination digital information. It is a figure which shows the example of a display of a collation result. It is a figure which shows the processing flow of the substitution detection operation performed in a terminal device. It is a figure which shows the functional block structure of the collation system (direct collation form) which concerns on 2nd Embodiment. It is a figure which shows the process flow in the direct collation form performed by a collation apparatus. It is a figure which shows the functional block structure of the collation system (preliminary learning form) which concerns on 2nd Embodiment. It is a figure which shows the processing flow in the prior learning form performed by a collation apparatus. It is a figure which shows the functional block structure (modification) of the collation system (direct collation form) which concerns on 2nd Embodiment. It is a figure which shows the functional block structure of the collation system which concerns on 3rd Embodiment. It is a figure which shows the basic process flow of authenticity determination performed with a collation apparatus. It is a figure which shows the processing flow of the modification of the authenticity determination performed by a collation apparatus.

Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings.

<First Embodiment>
In the first embodiment, in order to solve the first problem, as a first feature of the present invention, OCR and OCV are executed multiple times in the background. This improves the verification success rate. In addition, if all areas (name, date of birth, address, etc.) are targeted for verification each time, the verification success rate will decrease significantly. Therefore, as a characteristic related to the first characteristic, the area where verification is successful once is skipped. Then, the matching is focused on the failed area or the unmatched area. This collation is applied not only when the regions to be collated are different, but also to a portion within the same region. Further, as a feature related to the first feature, it is detected that the real object in the real space being photographed is replaced with another object during the matching.

(Structure of collation system)
FIG. 1 is a diagram showing a functional block configuration of a matching system 100 according to the first embodiment. The matching system 100 includes a terminal device 1 and a matching device 3. The terminal device 1 and the matching device 3 are physically and electrically connected to each other via a communication network 5 so that they can communicate with each other.

(Terminal device configuration)
First, the configuration of the terminal device 1 will be described. As shown in FIG. 1, the terminal device 1 is configured to include a photographing unit 11, a display unit 12, a switching detection unit 13, and a previous frame storage unit 14. The terminal device 1 is a client device on the local side, and is, for example, a smartphone terminal, a mobile phone terminal, or a tablet terminal having a camera function and a communication function. It may be a personal computer terminal to which a webcam is connected.

The image capturing unit 11 captures an image of the matching source object using an optical camera or the like, reads the matching source image data of the matching source object being photographed at relatively short time intervals (1 second, etc.), and reads the matching source image data. It has a function of outputting to the substitution detection unit 13 every time it is read (for example, 1 second), and outputting the collation source image data to the character information recognition unit 31 of the collation device 3 at a relatively long time interval (6 seconds or the like). The imaging unit 11 can be realized using, for example, a camera attached to a smartphone terminal. The verification source object is, for example, a personal identification document in which personal information (name, date of birth, address, etc.) such as a license, health insurance card, etc. is recorded.

The display unit 12 has a function of displaying collation source image data of the collation source target imaged by the image capturing unit 11 in a referenceable manner. For example, it can be realized using the display panel and the display function of the smartphone terminal.

The replacement detection unit 13 has a function of detecting the replacement of the reference source object based on the amount of movement of the reference source object during image capturing. Specifically, the switching detection unit 13 stores the collation source image data output immediately before from the image capturing unit 11 as a previous frame in the previous frame storage unit 14, and a time span in which switching cannot occur (for example, 1 second, Every time the collation source image data is read), the optical flow of the collation source object is calculated using the collation source image data output next from the imaging unit 11 and the previous frame of the previous frame storage unit 14, and the collation source object is calculated. Calculate the distance traveled by an object. Then, when the movement distance of the verification source object exceeds the threshold value, it is determined that the verification source object has been replaced, and an alert signal is output to the display unit 12 to display the alert information.

The optical flow is a technique for expressing the amount of movement of an object as vector data between two images captured at different times. As typical optical flow calculation methods, there are, for example, the Lucas-Kanade method (Non-Patent Document 2) and the Horn-Shunck method (Non-Patent Document 3).

The previous frame storage unit 14 has a function of storing the comparison source image data of the previous frame in order for the substitution detection unit 13 to calculate the optical flow using the comparison source image data of the next frame.

(Structure of collating device)
Next, the function of the matching device 3 will be described. As shown in FIG. 1, the collation device 3 includes a character information recognition unit 31, a collation unit 32, a collation destination digital information storage unit 33, and a collation result storage unit 34. The matching device 3 is, for example, a server device and is arranged in the cloud.

The character information recognition unit 31 repeatedly executes the process (OCR) of recognizing the character string of the collation source from the collation source image data every time the collation source image data is received from the photographing unit 11 of the terminal device 1, and then the collation unit 32 is executed. It has a function to output. Specifically, the character information recognition unit 31 receives the collation source image data output from the image capturing unit 11 of the terminal device 1 at a relatively long time interval (6 seconds or the like), and each time the collation source image data is received. Using the existing OCR technology, the received collation source image data is converted into text and output as collation source text data.

Further, the character information recognition unit 31 has a function of accepting the next collation source image data and continuing the character recognition process at each time when the character recognition process from the collation source image data requires time. For example, when the character recognition process from the matching source image data takes 2 seconds, the next recognition process is performed at intervals of 2 seconds or more.

The character information recognition unit 31 may be arranged in the terminal device 1.

The collation unit 32 collates the collation source character string with the collation destination character string stored in the collation destination digital information storage unit 33 every time the collation source character string is received from the character information recognition unit 31 ( OCV) is repeatedly executed and the collation result is output to the display unit 12 of the terminal device 1. Specifically, the collation unit 32 includes the collation source text data converted into text by the character information recognition unit 31 and the collation destination text information (collation destination character string, which will be described later) in the collation destination digital information storage unit 33. , And the matching area is set as the matched location information (matching OK), and the unmatched area or the unprocessed area is set as the unmatched location information (matching NG or unmatched), and the matching result is stored in the matching result storage unit 34. To do.

In addition, the collation unit 32 has a function of targeting only a portion that has failed collation or an unprocessed portion when the collation processing is continuously executed. Specifically, the collation unit 32 refers to the past collation results stored in the collation result storage unit 34, and when the collation process is executed next, it is already a collated portion (collation OK). Existing areas are skipped without matching. This collation is applied not only when the regions to be collated are different, but also to a portion within the same region. Then, when all the areas have been verified, the signal of "all verified" is output and the processing is terminated.

The collation destination digital information storage unit 33 has a function of storing, as collation destination digital information, a collation destination character string (character code string) that has been input in advance on a web page input form using a keyboard or the like and transmitted in advance. The collation destination digital information is, for example, a name, a date of birth, an address, or the like.

The collation result storage unit 34 has a function of storing the collation result of each area so that it can be referred to each time the collation unit 32 performs collation processing. When the matching unit 32 performs the matching process on the portion within one area, the matching result storage unit 34 stores the matching result for each portion.

(Operation of collation system)
(Verification processing operation)
Next, the collation processing operation performed by the collation system 100 will be described with reference to FIGS. This collation processing operation is performed in the background. FIG. 2 is a diagram showing a processing flow of the collation processing operation performed by the terminal device 1. FIG. 3 is a diagram illustrating an example of the matching source target object. FIG. 4 is a diagram showing a processing flow of the matching processing operation performed by the matching device 3. FIG. 5 is a diagram showing an example of the input form screen and the collation destination digital information. FIG. 6 is a diagram showing a display example of the matching result.

In the processing operation described later, including the second and subsequent embodiments, a new customer who wants to open a bank account online confirms the authenticity of his/her personal information entered in the input form by using the license as the verification source object. The case of proof will be described. The new customer inputs personal information such as name, date of birth and address in the input form on the bank home page (upper part of FIG. 5) and sends it in advance to the matching device 3 managed by the bank. Then, the collation device 3 stores the name and the like in the collation destination digital information storage unit 33 as the collation destination digital information (lower diagram of FIG. 5). Note that the reason for "as the collation destination digital information" is that the input name or the like is a character code that can be controlled by the collation device 3 which is a computer.

(Verification processing operation; operation of terminal device)
First, the operation of the terminal device 1 will be described with reference to FIG.

Step S101;
When the new customer starts photographing the license (FIG. 3) on the terminal device 1 in order to prove that the name or the like entered in the input form is correct, the photographing unit 11 of the terminal device 1 displays the image of the license. The display unit 12 displays the image of the license while reading the verification source image data every one second. Note that 1 second is an example, and may be 0.1 second or the like.

Step S102;
Next, the photographing unit 11 sequentially outputs the collation source image data to the character information recognition unit 31 of the collation device 3 every 6 seconds. At this time, the image capturing unit 11 sequentially outputs the collation source image data to the switching detection unit 13 every time the scanning is performed (every second). The operation of the switching detection unit 13 will be described later. Note that 6 seconds is an example, and may be 10 seconds or the like.

Step S103;
Finally, when the display unit 12 receives the collation result from the collation device 3, the display unit 12 superimposes and displays the collation result on the image of the license being photographed (FIG. 6).

(Verification process operation; operation of verification device)
Next, the operation of the matching device 3 will be described with reference to FIG.

Step S201;
First, the character information recognition unit 31 of the matching device 3 receives the matching source image data from the photographing unit 11 of the terminal device 1.

Step S202;
Next, the character information recognition unit 31 converts the matching source image data by converting the image characters included in the received matching source image data into text characters (character recognition processing; OCR) using the existing OCR technology. The text data to be collated is generated. Then, the character information recognition unit 31 outputs the collation source text data to the collation unit 32.

The character information recognition unit 31 repeatedly executes step S201 and step S202 every time the collation source image data is received from the terminal device 1. In this example, since the collation source image data is received from the terminal device 1 every 6 seconds, the character information recognition unit 31 sequentially outputs the collation source text data to the collation unit 32 every 6 seconds.

Step S203;
Next, the collation unit 32, upon receiving the collation source text data from the character information recognition unit 31, reads out the collation destination digital information (lower figure in FIG. 5) from the collation destination digital information storage unit 33 and collates with the collation source text data. Collate with the previous digital information (character collation processing; OCV). At this time, the collation unit 32 can match the entire one area to be matched for each area, or can match for each part in the area. For example, when the address is defined as one area, all the character strings of the address (XX prefecture, XX city, 23rd address) may be matched, or a part of the character string included in the address (XX prefecture 〇 ○ City) may be abutted.

Step S204;
After that, the matching unit 32 sequentially transmits the matching results obtained by matching the areas to the terminal device 1 for each area, and stores the matching results in the matching result storage unit 34. As described in step S103, the terminal device 1 superimposes and displays the matching result from the matching device 3 on the image of the license being photographed (FIG. 6).

On the left side of FIG. 6, an example of the first screen transition of the collation displayed on the display unit 12 of the collation device 3 is shown. First, the names are collated, and since the collation is OK, "collation OK" is superimposed and displayed in the name column of the license image. Next, the date of birth is collated, it takes time to perform the collation process, and the collation result is not obtained. Therefore, “(unmatched)” is superimposed and displayed in the date of birth column. Next, the address is collated, and since the collation is NG, "collation NG" is superimposed and displayed in the address column. It should be noted that the case of collation failure may be, for example, a case where a text character could not be correctly recognized from the collation source image data in step S202.

Step S205;
Next, the matching unit 32 compares the current matching count with a threshold value indicating the upper limit of the matching count. If the current matching count exceeds the threshold value, the process ends, and the current matching count exceeds the threshold value. If not, the process proceeds to step S206.

Steps S206, S207;
After that, the matching unit 32 refers to the matching result storage unit 34 and determines whether there is a matching NG or an unmatched region. If there is a collated NG or uncollated area, the collation unit 32 collates with the collated digital information in the second collation source text data in order to collate the area again. At this time, the collation unit 32 skips the region of collation OK without collating, and collates only the region of collation NG or the region not collated. On the other hand, if there is no matching NG or unmatched area, that is, if all areas are matching OK, the process ends.

On the right side of FIG. 6, an example of the screen transition for the second matching is shown. The collation processing is not performed on the name that has been collated OK, and only the collation result of the first collation is displayed, and the collation processing is started from the date of birth. Then, the date of birth is collated, and since the collation is OK, "collation OK" is superimposed and displayed in the date of birth column. Next, the address is collated, and since the collation is OK, "collation OK" is superimposed and displayed in the address column. The reason why the collation result changed to collation OK in the second collation is that the text characters could be correctly recognized by using the collation source text data different from that in the first collation. That is, since the collation process is repeatedly executed using a plurality of collation source text data corresponding to the same license, the collation success rate is improved.

(Replacement detection operation)
Next, with reference to FIG. 7, the replacement detection operation of the reference source object will be described. FIG. 7 is a diagram showing a processing flow of the switching detection operation performed by the terminal device 1. If another driver's license is replaced after the verification OK is displayed for the name, a combination of the name and address that does not exist will be generated. If the customer has malicious intent, if the name and address corresponding to the combination are entered in the input form, they will be collated and the person can be misrepresented. Therefore, in this operation, the replacement of the license is prevented, and the false verification of the collator is prevented. The switching detection operation is performed only by the terminal device 1.

Step S301;
First, when the terminal device 1 starts photographing a license, the photographing unit 11 of the terminal device 1 reads an image of the license as collation source image data every one second, and switches every time it is read (every second). The data is sequentially output to the detection unit 13. One second is an example of a time span in which it is assumed that the license cannot be replaced.

Step S302;
The substitution detection unit 13 stores the collation source image data output immediately before from the image capturing unit 11 in the previous frame storage unit 14 as a previous frame, and then outputs the collation source image data (after 1 second). And the previous frame are used to calculate the optical flow of the license, and the moving distance of the license for one second is calculated.

Step S303;
Next, the substitution detection unit 13 compares the moving distance of the license with a threshold value that indicates the variation distance of the position of the verification source target object due to the changing operation, and if the moving distance of the license exceeds the threshold value, the process proceeds to step S304. move on. The threshold value is preferably about 2 cm in consideration of camera shake during shooting. On the other hand, if the one-second moving distance of the license does not exceed the threshold value, the process ends.

Step S304;
Finally, when the moving distance of the license exceeds the threshold value, the replacement detection unit 13 determines that the license has been replaced, and outputs an alert signal to the display unit 12 to display alert information. At this time, the substitution detection unit 13 may output the alert signal to the speaker included in the terminal device 1 in addition to the display unit 12 to warn with a sound, or may output to the collation device 3.

After step S303 or step S304, the substitution detecting unit 13 repeatedly executes steps S301 to S304 each time the image capturing unit 11 reads the collation source image data.

(effect)
According to the first embodiment, the collation system 100 collates from the collation destination digital information storage unit 33 that stores the collation destination digital information input in advance, and the collation source image data of the collation source target object that is being photographed. The character information recognition unit 31 for recognizing the original character string to obtain the collation source text data and the collation unit 32 for collating the collation destination digital information with the collation source text data are provided. 32 performs the recognition process and the matching process a plurality of times in the background by using a plurality of matching source images that are continuous in time, so that the matching success rate can be improved.

Further, according to the first embodiment, when the collation unit 32 continuously executes the collation process, only the part that has failed collation or the part that has not been collated is processed, so the collation is successful. Since the collation fails or the non-collation is not performed for the part, the collation success rate can be further improved.

Further, according to the first embodiment, since the replacement detection unit 13 detects the replacement of the verification source target object based on the amount of movement of the verification source target object, the replacement of the verification source target object is performed. Can be reliably detected.

<Second Embodiment>
In the second embodiment, in order to solve the second problem, as a second feature of the present invention, a collation target character is generated with a character font (typeface) suitable for a physical object in the physical space. This improves the accuracy of OCR and OCV. Specifically, two methods are used.

The first is to generate image data in which the collation target digital information is imaged in the same character font as the character font displayed on the collation source object, and whether the image data is similar to the collation source image data. Is determined (direct verification form).

The second is to generate image data in which a learning character string such as an address list stored in advance is imaged with a plurality of character fonts including the character font displayed on the matching source object, and the image data is generated. In that case, OCR learning which is recognized as a learning character string of the image data is performed in advance, and OCR is performed using the learned OCR model generated by this (preliminary learning mode).

As a result, even if the character displayed on the matching source target object is a font unknown to the character information recognition unit 31, it is possible to robustly perform matching.

(Direct verification form)
(Structure of collation system)
FIG. 8 is a diagram showing a functional block configuration of a verification system (direct verification mode) according to the second embodiment. The terminal device 1 is configured to include a photographing unit 11 and a display unit 12. The collation device 3 includes a collation destination digital information storage unit 33, a font information storage unit 35, a collation image generation unit 36, and an image direct collation unit 37. Hereinafter, each functional unit (35 to 37) of the matching device 3 will be described. Each functional unit (11, 12) of the terminal device 1 has the same function as that of the first embodiment, and thus the description thereof is omitted.

The font information storage unit 35 has a function of readablely storing character font information. The font information is, for example, Mincho typeface, Gothic typeface, or the like.

The collation image generation unit 36 has a function of generating an image including characters of the same font as the character font displayed on the collation source object. Specifically, the matching image generation unit 36 generates matching image data that has the same font as the font of the character displayed on the matching source object and includes the same character string as the matching destination character string. For example, if the character string "Tokyo" displayed in the verification source object is "Mincho", the character string "Tokyo" input as the verification target digital information is changed to "Mincho". Output as an image of "Tokyo".

The image direct matching unit 37 has a function of calculating the image similarity between the matching source image data and the matching image data (image data generated by matching target digital information according to font information). Specifically, the image direct matching unit 37 estimates and calculates the image similarity between the matching source image data and the matching image data, determines similarity/dissimilarity, and outputs the determination result to the display unit 12 of the terminal device 1. Output to. The method of obtaining the image similarity is arbitrary, and for example, there is a similar image search method using triplet loss (Non-Patent Document 4) based on deep learning, and an Average Hash method performed without using deep learning. ..

(Operation of collation system)
FIG. 9 is a diagram showing a processing flow in the direct matching mode performed by the matching device 3. In the first embodiment, the method in which the matching unit 32 matches character strings has been described, but here, the image direct matching unit 37 performs the matching process between images.

Step S401;
First, the collation image generation unit 36 obtains the collation destination digital information (name, etc. entered in the input form) from the collation destination digital information storage unit 33, refers to the font information in the font information storage unit 35, and obtains the license. The image data for collation is converted into the same font as the font of the characters displayed on the certificate to generate the image data for collation. For example, when the character font of the license is Mincho type, the name and the like input in the input form are converted into Mincho type to generate image data for collation. It should be noted that as a method of discriminating the font of characters of the license, for example, a method of designating the character font of the license on the input form, a method of detecting the font type when the license is photographed, or the like can be considered.

Step S402;
Next, when the image direct matching unit 37 receives the matching source image data from the photographing unit 11 of the terminal device 1, the image direct matching unit 37 calculates the image similarity between the received matching source image data and the matching image data generated in step S401. Then, the similarity/dissimilarity is determined. For example, as a result of calculating the image similarity between two image data, if the similarity is equal to or more than the threshold value, it is determined that they are similar and the collation is OK. On the other hand, when the degree of similarity is less than the threshold value, it is determined that they are not similar to each other, and the collation is NG.

Step S403;
Finally, the image direct matching unit 37 transmits the matching result of step S402 to the terminal device 1. As described in the first embodiment, the terminal device 1 superimposes and displays the matching result from the matching device 3 on the image of the license being photographed.

(Pre-learning form)
(Structure of collation system)
Next, the pre-learning mode will be described. FIG. 10 is a diagram showing a functional block configuration of a matching system (preliminary learning mode) according to the second embodiment. The terminal device 1 is configured to include a photographing unit 11 and a display unit 12. The collation device 3 has a font information storage unit 35, a learning character string digital information storage unit 38, a learning image generation unit 39, an OCR learning unit 40, in addition to the configuration described in the first embodiment. And a learned OCR model storage unit 41. Hereinafter, each functional unit (35, 38 to 41) of the matching device 3 will be described. Each functional unit (11, 12) of the terminal device 1 has the same function as that of the first embodiment, and thus the description thereof is omitted.

The font information storage unit 35 has a function of readablely storing character font information. The font information is, for example, Mincho typeface, Gothic typeface, or the like.

The learning character string digital information storage unit 38 has a function of readablely storing any learning character string collected for OCR learning as learning character string digital information. The learning character string digital information is, for example, a list of names such as Taro Nippon, a list of addresses such as Chiyoda-ku, Tokyo, a list of dates of birth such as January 1, 2000, and the like. Name, address, date of birth. It is known information that is likely to be used as such.

The learning image generation unit 39 has a function of generating a learning image including a plurality of learning character strings displayed in a plurality of fonts including the same font as that of the character displayed on the matching source object. Specifically, the learning image generation unit 39 generates learning image data in which the arbitrary learning character string is image-displayed in a plurality of fonts. For example, the character string of the address list included in the learning character string digital information is converted into “Mincho type” and output as an image of the address list of “Ming type”.

The OCR learning unit 40 has a function of using the learning image data to generate a learning OCR model for learning so as to recognize an image of a learning character string of each font as the learning character string. For example, a learning OCR model for learning to recognize the character string of the address list from the image of the address list of "Ming-cho" is generated.

The learned OCR model storage unit 41 has a function of readablely storing the learning OCR model generated by the OCR learning unit 40.

(Operation of collation system)
FIG. 11 is a diagram showing a processing flow in the pre-learning mode performed by the matching device 3. In the first embodiment, the method of performing OCR based on only the glyphs of the characters displayed on the collation source object when performing OCR has been described. Here, the font (typeface) of the characters displayed on the collation source object is described. Is used as key information for character recognition to perform OCR.

Step S501;
First, the learning image generation unit 39 acquires learning character string digital information from the learning character string digital information storage unit 38, and generates learning image data in which the font information stored in the font information storage unit 35 is converted to the same font. To do. For example, when the font information includes Mincho font, learning image data of the name list based on the Mincho font is generated for each name. The address list and the date of birth list are also generated in the same manner. More specifically, for example, when “Tanaka” and “Sato” are included in the name list and the font information is “Mincho type” and “Block type”, the image of “Tanaka” of the Mincho type and “Tanaka of the block type” Image, the image of Mincho type "Sato", and the image of block type "Sato" are generated and output. As the font information, a method of designating a font of the same type as the font of the license on the input form, a method of designating an arbitrary font, and the like can be considered.

Step S502;
Next, the OCR learning unit 40 uses the learning image data generated in step S501 to perform learning for recognizing (determining/converting) the learning character string of each font as an image of the learning character string. The learning result is generated as a learning OCR model and stored in the learned OCR model storage unit 41. For example, a learning OCR model that recognizes the character string “Tanaka” in the case of the image of “Tanaka” in the Mincho type, and also recognizes the character string of “Tanaka” in the image of the block type “Tanaka” is generated. ..

The matching device 3 previously executes (preliminary learning) steps S501 and S502 for all learning character strings and all fonts.

Step S503;
After that, when the character information recognition unit 31 receives the collation source image data from the image capturing unit 11 of the terminal device 1, the character information recognition unit 31 has learned about each image such as the name, the date of birth, and the address of the predetermined font included in the collation source image data. By identifying the images of the same font, such as name, date of birth, and address, among the plurality of images included in the OCR model, and by identifying the learning character strings corresponding to the identified images, respectively, the received collation source is identified. A character string is recognized (OCR) from image data. For example, when the Mincho type "Tanaka" is displayed in the collation source image data, the character string "Tanaka" is specified from the learned OCR model. As a result, the character information recognition unit 31 converts the image character included in the matching source image data into a text character.

Step S504;
After that, the subsequent process described in step S202 (the process of outputting the collation source text data to the collation unit 32) to the process described in step S207 are executed.

(effect)
According to the second embodiment, the collation image generation unit 36 generates collation image data that has the same font as the font of the character displayed on the collation source object and includes the same character string as the collation destination digital information. Since the image direct matching unit 37 uses the matching image data to calculate the image similarity with the matching source image data, even if the characters displayed on the matching source target object are unknown fonts, they are robust. The character recognition rate can be improved.

In addition, according to the second embodiment, the learning image generation unit 39 includes a plurality of learning character strings that are displayed in a plurality of fonts including the same font as the character font displayed in the matching source object. An OCR model for learning that generates learning image data and uses the learning image data to learn so that the images of the learning character strings of each font are recognized as the learning character strings, respectively. Since the character information recognition unit 31 recognizes the character string of the matching source from the matching source image using the learning OCR model, even if the character displayed on the matching source object is an unknown font. It is possible to robustly collate and improve the character recognition rate.

The direct matching form and the pre-learning form can be combined with the first embodiment. For example, as illustrated in FIG. 12, in addition to the direct verification form shown in FIG. 8, a replacement detection unit 13 and a previous frame storage unit 14 are added to the terminal device 1 to detect the replacement of the verification source object. You may do it.

<Third Embodiment>
In the third embodiment, in order to solve the third problem, as a third feature of the present invention, a physical object is sensed three-dimensionally, and it is confirmed whether or not there is a physical discrepancy between the physical object and the actual physical object. By doing so, the authenticity of the physical object is determined.

(Structure of collation system)
FIG. 13 is a diagram showing a functional block configuration of the matching system according to the third embodiment. The configuration of the terminal device 1 is the same as that of the first embodiment. In addition to the configuration described in the first embodiment, the matching device 3 includes an authenticity determination unit 42, a verification source physical information storage unit 43, a biometric information authenticity determination unit 44, and a verification source biometric information storage unit 45. , And are configured. The third embodiment can be combined with the second embodiment.

The authenticity determination unit 42 has a function of determining the authenticity of the verification source object based on whether the characteristics of the verification source object included in the verification source image data match the features of the verification source object. .. Specifically, when the physical characteristics of the matching source object match the physical characteristics of the object serving as the model, the authenticity determining unit 42 determines whether the matching source target object is true. The determination result is output, and if they do not match, a false determination result indicating that the verification source target object is a counterfeit product is output.

The collation source physical information storage unit 43 has a function of storing, as reference source physical information, information about the original physical characteristics of the collation source target object so that the physical information can be referred to. The verification source physical information is, for example, the shape, size, thickness of the license, the content of the display information, the color, or the like.

The biometric information authenticity determination unit 44 determines the authenticity of the verification source object based on whether the characteristics of the verification source object included in the verification source image data match the features of the verification source object. Equipped with. Specifically, the biometric information authenticity determination unit 44 indicates that the verification source target object is a genuine product when the biometric characteristics of the verification source object match the biometric characteristics of the model object. A true determination result is output, and if they do not match, a false determination result indicating that the verification source target object is a counterfeit product is output.

The verification source biometric information storage unit 45 has a function of storing information related to the original biological characteristics of the verification source target object as the verification source biometric information so as to be referred to. The verification source biometric information is, for example, a face, a contour of a face related to a fingerprint, a fingerprint shape, or the like.

(Operation of collation system)
(basic action)
First, the basic operation of authenticity determination will be described with reference to FIG. FIG. 14 is a diagram showing a basic processing flow of authenticity determination performed by the collation device 3. Here, the authenticity is determined based on the physical characteristics of the matching source object in addition to the information input in the input form.

Step S601;
First, when the authenticity determination unit 42 receives the collation source image data from the image capturing unit 11 of the terminal device 1, the authenticity determination unit 42 calculates physical characteristics such as a license included in the collation source image data. For example, in the case of a license, the positional relationship such as the position of the black line frame displayed in the license and the arrangement of photographs is calculated with respect to the external shape of the license. When the matching source object has a three-dimensional shape, three-dimensional physical characteristics are calculated using the matching source image data continuously captured from a plurality of angles. For example, in the case of a card like a license, three-dimensional characteristics (height etc.) in the thickness direction are calculated in addition to the physical characteristics of the front surface and the back surface.

Step S602;
Next, the authenticity determination unit 42 acquires information on the original physical characteristics of the license, which is the physical characteristic of the verification source, from the verification physical information storage unit 43, and obtains information such as the license obtained from the verification source image data. It is determined whether the physical feature matches the reference physical feature.

As a method for determining the coincidence/mismatch of the physical features, there is a method of obtaining a feature amount manually determined in advance and obtaining a match with the photographed physical features. For example, in the case of a card-shaped object, a characteristic that a black straight line runs vertically and horizontally from a point 1 cm to the right and 0.5 cm below the upper left apex is used as the original physical characteristic. In addition, both means for automatically discovering and calculating the original physical features by using deep learning such as CNN (Convolutional Neural Network) and then obtaining the similarity can be considered. Either method may be used.

Step S603;
Finally, the authenticity determination unit 42 transmits the authenticity determination result of step S602 to the terminal device 1. For example, the true/false determination unit 42 transmits a false determination result indicating that the license being photographed is a fake one when the physical features of the two do not match or the similarity is lower than the threshold value.

(Modification of basic operation)
Next, with reference to FIG. 15, an operation (modification) of determining authenticity will be described. FIG. 15 is a diagram illustrating a processing flow of a modified example of the authenticity determination performed by the matching device 3. Here, the authenticity is determined based on the biometric characteristics of the matching source object in addition to the information input in the input form.

Step S701;
First, when the biometric information authenticity determination unit 44 receives the collation source image data from the imaging unit 11 of the terminal device 1, the biometric information authenticity determination unit 44 calculates the facial features of the person from the photograph of the license included in the collation source image data. For example, the biometric information authenticity determination unit 44 calculates eye positions, nose positions, face shapes, and the like.

Step S702;
Next, the biometric information authenticity determination unit 44 acquires information on the biometric characteristics of the person, which is the verification source biometric information, from the verification source biometric information storage unit 45, and determines the biometric characteristics of the person obtained from the verification source image data. , It is determined whether or not it matches the biometric information of the verification source. The method described in Non-Patent Document 3 or the like is used as a method for determining whether or not the biological characteristics match. The verification source biometric information is calculated in advance by using images captured continuously from a plurality of angles in consideration of having a three-dimensional shape.

(effect)
According to the third embodiment, the authenticity determination unit 42 determines whether the physical characteristics of the matching source object included in the matching source image data match the physical characteristics of the matching source object. Since the authenticity of the verification source object is determined, the authenticity of the verification source object can be reliably determined. In particular, since a three-dimensional physical characteristic is used, it can be determined as false if, for example, a copy of the license is used.

Further, according to the third embodiment, the biometric information authenticity determination unit 44 determines whether the biometric characteristics included in the verification source target object included in the verification source image data match the biometric characteristics of the person. Based on this, since the authenticity of the verification source object is determined, it is possible to reliably determine the authenticity of the verification source object such as replacement of the photograph part. Furthermore, the verification source biological object using a substitute photo instead of the actual biometric information. It is possible to prevent the replacement of information.

<Use>
In each of the embodiments, the case of opening a bank account has been described as an example, but the invention can be applied to various scenes in which identity verification processing is performed using identity verification documents.

<Other>
The matching system 100 described in the present embodiment can be realized by a computer including a CPU, a memory, a hard disk, and the like. It is also possible to create a collation program for causing a computer to function as the collation system 100 and a storage medium for the collation program.

100... Collation system 1... Terminal device 11... Imaging unit 12... Display unit 13... Switching detection unit 14... Previous frame storage unit 3... Collation device 31... Character information recognition unit 32... Collation unit 33... Collation destination digital information storage unit 34... Collation result storage unit 35... Font information storage unit 36... Collation image generation unit 37... Image direct collation unit 38... Learning character string digital information storage unit 39... Learning image generation unit 40... OCR learning unit 41... Learning Completed OCR model storage unit 42... Authenticity determination unit 43... Collation source physical information storage unit 44... Biometric information truth determination unit 45... Collation source biometric information storage unit

Claims (9)

A collation destination digital information storage unit for storing a pre-input collation destination character string;
A character information recognition unit that recognizes a character string of a matching source from a matching source image of a matching source target being photographed,
A collating unit that collates the character string of the collation source and the character string of the collation destination,
The character information recognition unit and the collation unit,
A collation system, characterized in that the recognition process and the collation process are executed a plurality of times in the background using a plurality of collation source images that are temporally consecutive. The collating unit,
The collation system according to claim 1, wherein when the collation process is continuously executed, only a portion in which the collation fails is processed. The collation system according to claim 1, further comprising: a replacement detection unit that detects replacement of the verification source object based on the amount of movement of the verification source object. The collation system according to any one of claims 1 to 3, further comprising an image generation unit that generates an image including a character of the same font as the character font displayed on the collation source object. The image generating unit is a collation image generating unit that generates a collation image including the same character string as the collation destination character string in the same font as the character displayed on the collation source object,
The collation system according to claim 4, further comprising an image direct collation unit that calculates an image similarity between the collation source image and the collation image. The image generation unit is a learning image generation unit that generates a learning image including a plurality of learning character strings displayed in a plurality of fonts including the same font as the font of the character displayed in the matching source object. There
An OCR learning unit is further provided for generating a learning OCR model for learning by using the learning image to recognize an image of a learning character string of each font as the learning character string.
The character information recognition unit,
The collation system according to claim 4, wherein a character string of a collation source is recognized from the collation source image using the learning OCR model. Further comprising a determination unit that determines the authenticity of the matching source object based on whether or not the characteristics of the matching source object included in the matching source image match the characteristics of the matching source object. The collation system according to any one of claims 1 to 6, which is characterized in that: In the verification method performed by the verification system,
A first step of storing a character string of a collation destination input in advance;
A second step of recognizing the character string of the matching source from the matching source image of the matching source object being photographed,
Performing a third step of collating the collation source character string with the collation destination character string,
In the second step and the third step, respectively,
A collation method comprising performing the recognition process and the collation process multiple times in the background using a plurality of collation source images that are temporally continuous. A collation program causing a computer to function as the collation system according to any one of claims 1 to 7.

Images