WO2020042035A1

WO2020042035A1 - Method and device for automatic fingerprint image acquisition

Info

Publication number: WO2020042035A1
Application number: PCT/CN2018/103082
Authority: WO
Inventors: Linpeng TANG; Cheng TAI; Zhuo QIU; Qin Liu; Qingdi ZHANG; Ruihua SUN
Original assignee: Moqi Technology (beijing) Co., Ltd.
Priority date: 2018-08-29
Filing date: 2018-08-29
Publication date: 2020-03-05

Abstract

A ruler with markers is provided. A method of automatically acquiring fingerprints by using the ruler with markers, which comprises: placing the ruler in a position relative to the fingerprint; detecting the locations of the markers through a image capturing device automatically and visualizing them on a user interface; helping the user to adjust the camera of the image capture device according to the locations of the markers so that the fingerprint is in the optimal position of the camera view; activating autofocus when the fingerprint is in the optimal position of the camera view; using the locations of the markers to autofocus on the fingerprint; locking the focus and starting the image capturing process when autofocus is finished; and using locations of the markers to correct projective deformations of the fingerprint image and rescale it to 1: 1 ratio.

Description

Method and Device for Automatic Fingerprint Image Acquisition

FIELD OF THE INVENTION

This invention relates generally to fingerprint recognition. More particularly, it relates to automatic capturing of fingerprint images.

BACKGROUND OF THE TECHNOLOGY

There exist two types of verifications for fingerprint identification, 1: 1 verification and 1: N verification. In 1: 1 verification, there is one fingerprint enrolled on a device, and the only verification is whether the attempted user is the correct match to the enrolled fingerprint. A non-limiting example includes fingerprint verification to unlock a phone. By contrast, 1: N verification involves searching a fingerprint against a large database which may contain thousands to billions of fingerprint images to determine the identity of the fingerprint. One important use case of 1: N verification is latent fingerprint matching, where “latent” fingerprint may refer to the accidental impression of a fingerprint rather than one deliberately collected from a subject. A non-limiting example includes crime scene fingerprint matching. Problems with 1: N verification include the fingerprint exhibiting poor quality or distortion, or only a small portion of the fingerprint is usable for matching purposes.

Conventional fingerprint identification technologies have been successfully used in solving crimes when the database of fingerprints is small or medium size. The conventional procedure for using fingerprints in solving crimes typically includes three steps: fingerprint acquisition, forensic experts mark features, and matching backend.

In fingerprint acquisition, the police or another investigative body collects the fingerprint from the crime scene. Depending on the surface where the fingerprint is attached, different methods may be used. For instance, on a flat surface, a multi-spectrum lighting source may be used to make the fingerprint visible. If the fingerprint is attached to a curtain or other fabric, chemical compounds may be used to make the fingerprint more visible in order to take a photo. Ultimately, a set of fingerprint images or samples will be obtained.

In the next step, the fingerprint images (or samples) are taken back to an office or lab for forensic experts to mark specific features. Typically, one fingerprint takes about half an hour to mark, and an entire crime scene could take about ten hours.

In the last step, the marked features are sent to a matching backend to find the top candidates from the database. This typically takes about a few hours to complete. Conventional technologies rely heavily on minutia features of the fingerprint for matching. Minutia (or micro-scale) features include ending and bifurcations of the ridge/valley lines in the fingerprint, as non-limiting examples. Macro-scale features, such as distance between the fingerprint core and delta, may also be used to perform filtering to accelerate the process.

While fingerprint matching has been widely applied in solving crimes, latent fingerprint acquisition still requires a lot of efforts. In many cases, the scale ruler is typically placed alongside the fingerprint, and the fingerprint image may be captured with a digital camera. The images are then brought to the police station, where experts need to rescale the image to 1: 1 ratio according to the scale ruler in the image. This requires a lot of manual effort. Moreover, if the camera is not placed just above the latent fingerprint and parallel to the surface, the fingerprint might be deformed in other ways that can not be easily corrected. Such deformations will lower the recognition accuracy. In other cases, fingerprint samples can be made with a tape, but this process may damage the fingerprint, and also requires additional manual effort. An image may be captured on the fingerprint sample with a camera using the same method as above, or a scanner machine can be used to obtain a 1: 1 digital copy of the sample.

In addition, fingerprint acquisition from a live person is needed in many cases as well. As a non-limiting example, the householder's fingerprints need to be excluded when matching latent fingerprints collected from the crime scene. Currently this is done by manual inspection, using ink to collect fingerprints from the householder, or even bringing a fingerprint scanner to the crime scene. However, such methods either require expertise from the police officer, or additional devices are needed.

In all cases, current fingerprint acquisition method requires a lot of manual effort, and may cause deformations that lower the recognition accuracy for the matching backend. Moreover, portable computing devices have been gaining popularity in police jobs. Therefore there is a strong need for automatic fingerprint acquisition on mobile computing devices.

SUMMARY OF THE INVENTION

Embodiments of the present invention may improve over conventional methods in terms of speed, accuracy and ease of use. Embodiments of the present invention may include some or all of the following features. It uses a specially designed ruler with markers to locate the fingerprint, either a latent or live, as well as correct deformations. It uses a portable computing device to capture the image, and is able to detect the markers and correct the deformation automatically with one or more processors. The corrected images can then be sent to the matching backend to find the top candidates.

The present invention provides a method of automatically capturing fingerprint images through use of a ruler with markers, which comprises: placing the ruler in a location relative to the fingerprint; detecting the locations of the markers through a image capturing device automatically and visualizing them on the user interface; helping the user to adjust the camera of the image capturing device with locations of the markers so that the fingerprint is in optimal location of the camera view; activating the autofocus when the fingerprint is in the optimal position of the camera view; using the locations of the markers to autofocus on the fingerprint; and locking the focus and starting the image capturing process when autofocus is finished.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention has other advantages and features which will be more readily apparent from the following detailed description of the invention and the appended claims, when taken in conjunction with the accompanying drawings, in which:

FIG 1. is a design of the ruler with markers. The four markers are placed on the four corners of a square. The center of the marker is carved out where the latent fingerprint is placed. The specific shape or the size of the ruler may vary to suit different use cases.

FIG. 2 shows traditional scale rulers used by the police.

FIG. 3 is a fingerprint image acquisition pipeline through use of a portable computing device.

FIG. 4 is an example user interface of the system when (1) / (2) the ruler with markers and the fingerprint are not in optimal location and size in the camera view, (3) when the ruler with markers are in optimal location and size in the camera view (in green color) .

FIG. 5 is the fingerprint image post-processing pipeline.

FIG. 6 is the user interface showing the fingerprint image before and after deformation correction, cropping and rescaling to 1: 1 ratio.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments described herein provide for a system that automatically captures, rescales and aligns fingerprints through use of portable computing devices. It uses a specially designed ruler with markers (FIG. 1) to locate the fingerprint, as well as aligns and rescales the image through the already-known relative locations of the markers. This allows us to capture 1: 1 images of fingerprints that can be matched accurately in the matching backend.

In previous methods, scale rulers (FIG. 2) have been used along with digital cameras or portable computing devices when capturing latent fingerprint images. The images can then be rescaled to 1: 1 ratio manually according to the distance scales on the ruler. This method has two issues: (1) it requires manual effort to rescale the images to 1: 1 ratio (2) it can not correct deformations if the digital camera or the portable computing device is not held just above and parallel to the surface of the latent fingerprint when taking the image. Incorrectly rescaled or deformed images can lead to decreased accuracy in fingerprint matching.

FIG. 2 shows the design of the ruler with markers. Four markers are placed on the four corners of a square. The center of the square is carved out to place the latent fingerprint. The specific design of the ruler, e.g. how many and what kind of markers are used, the shape and size of the ruler might vary to suit different use cases, and it should known a priori so the image can be automatically corrected using the relative locations of the markers. The ruler is placed on the surface of the latent fingerprint, with the fingerprint inside the hollow center of the ruler so the fingerprint is visible to the camera of the portable computing device.

The markers are designed such that they can be detected automatically and reliably by one or more processors. AruCo is a widely used open source library that achieves this goal, and Aruco markers are used here as an example, but other kinds of markers, non-limiting examples including squares, circles or other markers that allow automatic and reliable detection can be used as well.

The pipeline of taking a 1: 1 picture of the latent fingerprint using the ruler with markers is illustrated in FIG. 3.

First, a video stream is opened before capturing a photo on a portable computing device. The video stream helps the user to preview the photo before initiating the capturing process. The system automatically detects the markers in the video stream. Using coordinates of the detected markers, the system can infer automatically whether the ruler with the markers along with the fingerprint is in optimal location in the camera view of the portable computing device, e.g. whether it is near to the center of the view, whether it's too close or too far to the camera, or whether the angle between the device camera and the surface of the fingerprint is too large. Different methods can be used to determine whether the markers are in suboptimal conditions. As a non-limiting example, if the area inside the markers is larger than a threshold, the camera may be too close to the markers and the fingerprint and the image may be out of focus. These suboptimal conditions lowers the quality of the final fingerprint image and should be avoided. Additional light source may be used to improve the image quality in this process as well.

User interface (UI) is designed to avoid such situations. Non-limiting examples include a polygon mask overlaid on the video stream on the convex hull of the detected markers. The mask intuitively shows the users the region where the fingerprint should be in the camera view. As a nonlimiting example, the mask is green when the markers and the fingerprint are within optimal region in the camera view of the portable computing device. Otherwise, the mask can be set to other colors, and a message can be shown on the UI that prompts the user to adjust the location and angle of the portable computing device.

As another nonlimiting example, assume the length of the four sides of the polygon is x1, x2, x3, x4, and the length of the two diagonals are y1, y2. For a portable computing device, say the maximal side length of the polygon is configured to be maxCaptureLength, and the minimal side length is configured to be minCaptureLength. These two parameters are determinded by experiment to ensure that the fingerprint image captured is within the focal length of the camera and has a high resolution. Specifically, the distance between the camera and the markers is adjusted so that the fingerprint image is located within the focal length of the camera, and the closest distance and the farthest distance are obtained when the image is clear. The length of the longest side of the polygon is maxCaptureLength at the nearest distance, and the length of the shortest side of the polygon at the longest distance is minCaptureLength. If max (x1, x2, x3, x4) >= maxCaptureLength, then the camera is too close to the fingerprint; and if min (x1, x2, x3, x4) <= minCaptureLength, then the camera is too far away from the fingerprint, and it needs to be adjusted accordingly. If min (x1, x2, x3, x4) /max (x1, x2, x3, x4) <= 0.8 or min (y1, y2) /max (y1, y2) <= 0.8, then the camera is not directly above and parallel to the latent fingerprint, and it needs to be adjusted as well. When there is 1 or more markers with the ruler, the four corners of each marker can be detected. By measuring the side lengths and the diagonals of each marker, as well as by measuring the distance between the markers, the camera can be adjusted such that the fingerprint is in optimal location of the camera view as well.

When the markers and the fingerprint are in optimal location of the camera view and correspondingly the mask in green color, the system starts the autofocus activity. This phase adjusts the camera parameter to focus the image on the fingerprint surface to in order to improve imaging quality. Because the fingerprint is placed inside the ruler with markers or in other known locations relative to the ruler, the system can detect the markers and focus to the fingerprint area, and the whole process can be done automatically.

When autofocus completes, the system starts capturing the photo automatically. However, the portable computing device may have been moved during the autofocus phase and the resulting photo can be blurry or the latent fingerprint may not even be in the camera view. Such movement is detected using the gyrometer, and the image capturing process is aborted if big movement is detected.

When capturing the photo, metering parameters, such as ISO, shutter speed and aperture also need to be properly tuned to achieve highest quality for the fingerprint image. The metering region is used in modern cameras on portable computing devices to obtain the best configuration for these parameters. Similar to the focus area, the system also sets the metering region to be inside the inner corners of the four markers to obtain the best configuration. It can then be fined tuned to further improve imaging quality for the latent fingerprint.

After the photo is captured, it's then post-processed as shown in FIG. 5. First, the precise marker locations are detected again on the captured photo. If the detection fails, the location of the markers in the video frame just before autofocus starts can be reused.

Second, perspective transform (or other similar transformations) is applied using the locations of the markers, so 1: 1 image of the fingerprint is obtained. The perspective transform can be solved in the following way: assume the inner corners of the four markers are (x ₁, y ₁) , (x ₂, y ₂) , (x ₃, y ₃) , (x ₄, y ₄) , where (x ₁, y ₁) is at the lower left corner and the four markers are in clockwise direction. In the transformed image, the inner corners should then be at locations (u ₁, v ₁) = (0, 0) , (u ₂, v ₂) = (0, 512) , (u ₃, v ₃) = (512, 512) , (u ₄, v ₄) = (512, 0) . Using these relations, we can solve the parameters of the perspective transform (m ₁₁, m ₁₂, m ₁₃, m ₂₁, m ₂₂, m ₂₃, m ₃₁, m ₃₂) through a system of linear equations:

v _i (1+m ₃₁x _i+m ₃₂y _i) = (m ₁₃+m ₁₁x _i+m ₁₂y _i)

v _i (1+m ₃₁x _i+m ₃₂y _i) = (m ₂₃+m ₂₁x _i+m ₂₂y _i)

for i=1..4.

Using the solved parameters, the transformed coordinates of all the pixels inside the 4 inner corners of the markers can be calculated, and then the transformed image is obtained via bilinear interpolation. When there is 1 or more markers, the 4 corners of each marker can be detected. Because the size of the markers are known beforehand, the 4 corners can be used to perform perspective transform and correct the image as well.

Finally, the region inside the ruler with the markers are cropped out to obtain the standardized image. In addition, the image can be further processed, nonlimiting examples including contrast normalization or enhancement, either on the portable computing device or on one or more other processors. The results are rendered on the mobile computing device to facilitate further operations, nonlimiting examples including quality assessment and results confirmation. The rescaled and cropped image is then sent to the matching backend. The image before and after post-processing is shown in FIG. 6.

Similar methods can be used to collect fingerprint from a live person. One's finger may be placed inside the ruler with markers, and the finger's image can be captured by the portable computing device as well. Nonlimiting example usage of this approach includes excluding the householder's fingerprints from the crime scene, collecting fingerprints from a suspect, or verifying a person's identity.

In the above pipeline, markers have been used to locate the fingerprint and facilitate further processing. The latent fingerprint can also be detected automatically on the mobile computing devices or on other processors using various methods, nonlimiting examples including wavelets and convolutional deep neural networks. The detected latent fingerprint region, including its key feature points, such as the top and bottom of the fingertip, the core and delta of the fingerprint, may be used to facilitate further processing in similar ways as the locations of the markers, nonlimiting examples including autofocus to the region, alignment to a standardize fingerprint, rescaling to a standard fingerprint size and cropping to the fingerprint region as well.

Claims

A method of automatically capturing fingerprint images through use of a ruler with markers, which comprises:

placing the ruler in a location relative to the fingerprint;

detecting the locations of the markers through an image capturing device automatically and visualizing them on the user interface;

helping the user to adjust the camera of the image capturing device according to the locations of the markers so that the fingerprint is in the optimal position of the camera view;

activating the autofocus when the fingerprint is in the optimal position of the camera view; using the locations of the markers to autofocus on the fingerprint; and

locking the focus and starting the image capturing process when autofocus is finished.
The method of claim 1, which further comprises: using locations of the markers to correct projective deformations of the fingerprint image and rescale it to 1: 1 ratio.
The method of claim 1 or 2, which further comprises: using the locations of the markers to determine the metering parameter of the image capturing process.
The method of any of claim 1-3, which further comprises: using locations of the markers to crop the image and only keep the region of the fingerprint.
The method of any of claims 1-4, which further comprises: normalizing and enhancing the fingerprint image for further operations.
The method of any of claims 1-5, wherein the fingerprint is placed inside the ruler with markers.
The method of any of claims 1-6, wherein the locations of the markers are visualized on the user interface and guide the users to place camera in optimal relative location to the fingerprint.
The method of any of claims 1-7, wherein the locations of the markers are used to automatically determine the fingerprint is in optimal location and angle in the camera view.
The method of any of claims 1-8, wherein the locations of the markers are used to infer the location of the fingerprint, and the camera autofocuses to the fingerprint region automatically.
The method of any of claims 1-9, wherein the locations of the markers are used to infer the location of the fingerprint, and the metering parameter is determined from the fingerprint region automatically.
The method of any of claims 1-10, wherein the locations of the markers are used to automatically correct deformations of the fingerprint image.
The method of any of claims 1-11, wherein the fingerprint image is further rescaled to 1: 1 automatically through the locations of the markers in the corrected image.
The method of any of claims 1-12, wherein the fingerprint image is further cropped to standard format through the locations of the markers.
The method of any of claims 1-13, wherein the photo can be taken by any imaging devices, and detection of the makers and further processing of the image can be carried out in any computing devices as well.
The method of any of claims 1-14, wherein still image capturing is skipped and the whole process can be carried out on a video stream.
The method of any of claims 1-15, wherein the rescaled and cropped fingerprint image can be further processed, including contrast normalization and enhancement, on the portable computing device or on one or more other processors.
The method of any of claims 1-16, wherein the processed fingerprint image is then rendered on the mobile computing device for further operations, including quality assessment and matching results confirmation on the mobile computing device.
The method of any of claims 1-17, wherein the fingerprint to be captured is latent.
The method of claim 1-16, where a person's finger can be put inside or near the ruler with markers, and a fingerprint image collected with the portable computing device as well.
The method of claim 19, where 3D information of the person's finger may be collected with various approaches, including stereo camera, structured light and photometrics.
The method of claim 19 or 20, where the 3D information of the person's finger may be used in fingerprint matching system to improve accuracy.
The method of any of claims 1-21, wherein the image capturing process is aborted if the image capturing device have been moved during autofocus phase.
A method of capturing fingerprint images through use of a ruler with markers, which comprises:

placing the ruler in a location relative to the fingerprint;

capturing image containing the ruler and the fingerprint with an image capturing device; and

using locations of the markers to correct projective deformations of the fingerprint image and rescale it to 1: 1 ration.
A method of automatically capturing fingerprint images, which comprises:

detecting the locations of key feature points through an image capturing device automatically and visualizing them on the user interface;

helping the users to adjust the camera of the image capturing device with locations of the key feature points so the fingerprint is in optimal location of the camera view;

activating the autofocus when the fingerprint is in the optimal position of the camera view;

using the locations of the key feature points to autofocus on the fingerprint; and

locking the focus and starting the image capturing process when autofocus is finished.
The method of claim 24, the key feature points are the top and bottom of the fingertip, or the core and delta of the fingerprint.
A method of capturing fingerprint images, which comprises:

capturing image containing key feature points and the fingerprint through an image capturing device; and

using locations of the key feature points to correct projective deformations of the fingerprint image and rescale it to 1: 1 ration.
The method of claim 26, the key feature points are the top and bottom of the fingertip, or the core and delta of the fingerprint.
A ruler with at least 1 marker, wherein the ruler is placed close to the fingerprint to facilitate automatic acquisition.
The ruler of claim 28, wherein the ruler can be a square or other shape, and it may have four or other number of markers.
The ruler of claim 28 or 29, wherein the markers are on the corners of the ruler.
The ruler of any of the claims 28-30, wherein the center of the ruler is carved out.
The ruler of any of claims 28-31, wherein the markers are the ArUco markers, or other any patterns that can be detected automatically by a computing device.
The ruler of any of claims of 28-32, wherein the ruler can be made with soft materials, and it can therefore be placed on a non-flat surface.
The ruler of claim 33, wherein the flexible material is plastic or paper.
The ruler of any of claims 28-34, wherein a person's finger can be placed and the fingerprint captured with a portable computing device.