JP7256512B2 - Personal identification system, personal identification method and program - Google Patents

Description

The present invention relates to an individual identification system, an individual identification method and a program.

An authentication system using a PIN (Personal Identification Number) code and an authentication system using an IC (Integrated Circuit) card are generally used for room entry management or usage restrictions on electronic devices. With these authentication systems, it is difficult to detect spoofing or a change of person (for example, a person who operates the keyboard of a computer terminal changes in the middle) after a successful login. Therefore, there is a demand for continuous personal authentication (continuous authentication) even after regular login (see Patent Document 1, for example).

The continuous authentication method should preferably be a method that does not require the person to be authenticated to perform special operations such as entering a new password, that is, a method that allows the person to be authenticated without being aware of it.

JP 2017-142614 A

However, in the information processing apparatus disclosed in Patent Literature 1, the keyboard needs to be provided with special equipment such as an acceleration sensor and a pressure sensor for natural continuous authentication as described above.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a personal identification system, a personal identification method, and a program that enable continuous authentication to be performed naturally without providing special equipment.

In order to achieve the above object, the personal identification system according to the first aspect of the present invention includes:
an imaging unit that captures an image of an entire hand that performs an operation input to an electronic device as a subject, with an image sensor continuously for a certain period of time during which a plurality of operation inputs are performed with the hand;
Synthesizing a plurality of images captured by the image sensor or synthesizing a plurality of images obtained by processing the plurality of images to generate a synthetic image showing how the hand movement spreads during the predetermined period of time. an image synthesizing unit to generate;
a normalization unit that normalizes the synthesized image synthesized by the image synthesizing unit with reference to the maximum value of pixels in the synthesized image to generate a normalized image;
an authentication unit that performs personal authentication based on the normalized image generated by the normalization unit;
with
The image sensor captures a distance image in which the value of each pixel indicates the distance between the image sensor and the subject as an image of the entire hand as a subject;
The image synthesizing unit performs a process of dividing the value of each pixel by the maximum value that the pixel can take on a plurality of images captured by the image sensor,
The authentication unit
calculating a degree of similarity between the normalized image and the template image by performing template matching for comparing the normalized image with a pre-stored template image;
If the calculated degree of similarity exceeds the threshold, the person is authenticated.

A personal identification system according to a second aspect of the present invention includes:
an imaging unit that captures an image of an entire hand that performs an operation input to an electronic device as a subject, with an image sensor continuously for a certain period of time during which a plurality of operation inputs are performed with the hand;
an image synthesizing unit for synthesizing a plurality of images obtained by processing a plurality of images captured by the image sensor to generate a synthetic image showing how the movement of the hand spreads during the predetermined time period;
a normalization unit that normalizes the synthesized image synthesized by the image synthesizing unit with reference to the maximum value of pixels in the synthesized image to generate a normalized image;
an authentication unit that performs personal authentication based on the normalized image generated by the normalization unit;
with
The image sensor is
Capturing an image with visible light as an image of the entire hand as a subject,
The image synthesizing unit
A plurality of images captured by the image sensor are converted into a binary image that distinguishes the hand region from other regions, and the value of each pixel of the binary image is divided by the maximum value that the pixel can take. and
The authentication unit
calculating a degree of similarity between the normalized image and the template image by performing template matching for comparing the normalized image with a pre-stored template image;
If the calculated degree of similarity exceeds the threshold, the person is authenticated.

The authentication unit
Is the sum of the squares of the difference in value between each pixel of the normalized image and each pixel of the template image, or the sum of the absolute values of the difference between each pixel of the normalized image and each pixel of the template image ? , a normalized cross-correlation of the values of each pixel of the normalized image and each pixel of the template image, or a zero-mean normalized cross-correlation of the values of each pixel of the normalized image and each pixel of the template image. Calculate either the correlation or as the similarity,
You can do it.

A personal identification method according to a third aspect of the present invention includes:
an imaging step of capturing an image of an entire hand that performs an operation input on an electronic device as a subject, with an image sensor continuously for a certain period of time during which a plurality of operation inputs are performed with the hand;
Synthesizing a plurality of images captured in the imaging step or synthesizing a plurality of images obtained by processing the plurality of images to generate a synthetic image showing how the movement of the hand spreads during the predetermined period of time. an image synthesizing step to generate;
a normalization step of normalizing the synthesized image synthesized in the image synthesizing step with reference to the maximum value of pixels in the synthesized image to generate a normalized image;
an authentication step of performing personal authentication based on the normalized image generated in the normalization step;
including
The image sensor captures a distance image in which the value of each pixel indicates the distance between the image sensor and the subject as an image of the entire hand as a subject;
In the image synthesizing step, a plurality of images captured by the image sensor are processed by dividing the value of each pixel by the maximum value that the pixel can take,
In said authentication step,
calculating a degree of similarity between the normalized image and the template image by performing template matching for comparing the normalized image with a pre-stored template image;
If the calculated degree of similarity exceeds the threshold, the person is authenticated.
A personal identification method according to a fourth aspect of the present invention includes:
an imaging step of capturing an image of an entire hand that performs an operation input on an electronic device as a subject, with an image sensor continuously for a certain period of time during which a plurality of operation inputs are performed with the hand;
an image synthesizing step of synthesizing a plurality of images obtained by processing the plurality of images captured in the imaging step to generate a synthetic image showing how the hand movement spreads during the predetermined time;
a normalization step of normalizing the synthesized image synthesized in the image synthesizing step with reference to the maximum value of pixels in the synthesized image to generate a normalized image;
an authentication step of performing personal authentication based on the normalized image generated in the normalization step;
including
The image sensor is
Capturing an image with visible light as an image of the entire hand as a subject,
In the image synthesizing step,
A plurality of images captured by the image sensor are converted into a binary image that distinguishes the hand region from other regions, and the value of each pixel of the binary image is divided by the maximum value that the pixel can take. and
In said authentication step,
calculating a degree of similarity between the normalized image and the template image by performing template matching for comparing the normalized image with a pre-stored template image;
If the calculated degree of similarity exceeds the threshold, the person is authenticated.

A program according to a fifth aspect of the present invention comprises
the computer,
An imaging unit that captures an image of the entire hand that performs operation input on an electronic device as a subject, with an image sensor continuously for a certain period of time during which a plurality of operation inputs are performed with the hand;
Synthesizing a plurality of images captured by the image sensor or synthesizing a plurality of images obtained by processing the plurality of images to generate a synthetic image showing how the hand movement spreads during the predetermined period of time. an image synthesizing unit to generate,
a normalization unit that normalizes the synthesized image synthesized by the image synthesizing unit with reference to the maximum value of pixels in the synthesized image to generate a normalized image;
an authentication unit that performs personal authentication based on the normalized image generated by the normalization unit;
function as
The image sensor captures a distance image in which the value of each pixel indicates the distance between the image sensor and the subject as an image of the entire hand as a subject;
The image synthesizing unit performs a process of dividing the value of each pixel by the maximum value that the pixel can take on a plurality of images captured by the image sensor,
The authentication unit
calculating a degree of similarity between the normalized image and the template image by performing template matching for comparing the normalized image with a pre-stored template image;
If the calculated degree of similarity exceeds the threshold, the person is authenticated.
A program according to a sixth aspect of the present invention comprises
the computer,
An imaging unit that captures an image of the entire hand that performs operation input on an electronic device as a subject, with an image sensor continuously for a certain period of time during which a plurality of operation inputs are performed with the hand;
an image synthesizing unit for synthesizing a plurality of images obtained by processing a plurality of images captured by the image sensor to generate a synthetic image showing how the movement of the hand spreads during the predetermined time period;
a normalization unit that normalizes the synthesized image synthesized by the image synthesizing unit with reference to the maximum value of pixels in the synthesized image to generate a normalized image;
an authentication unit that performs personal authentication based on the normalized image generated by the normalization unit;
function as
The image sensor is
Capturing an image with visible light as an image of the entire hand as a subject,
The image synthesizing unit
A plurality of images captured by the image sensor are converted into a binary image that distinguishes the hand region from other regions, and the value of each pixel of the binary image is divided by the maximum value that the pixel can take. and
The authentication unit
calculating a degree of similarity between the normalized image and the template image by performing template matching for comparing the normalized image with a pre-stored template image;
If the calculated degree of similarity exceeds the threshold, the person is authenticated.

According to the present invention, the motion of a hand that performs an operation input on an electronic device is continuously imaged for a certain period of time during which a plurality of operation inputs are performed, and a normalized image is generated that indicates the spread of the hand motion during that time. Since the spread of hand movements when performing a plurality of operation inputs differs from person to person and cannot be imitated, an individual can be identified from the normalized image. In this way, continuous authentication can be naturally performed simply by imaging the movement of the hand with the camera, without providing special equipment.

1 is a block diagram showing the configuration of a personal identification system according to Embodiment 1 of the present invention; FIG. It is a figure which shows an example of the attachment position of an image sensor. Figure 2 shows the data flow of the personal identification system of Figure 1; (A) to (D) are diagrams showing examples of normalized images. It is a figure which shows the hardware constitutions of an information processing apparatus. 4 is a flowchart showing the flow of template image creation processing in the information processing apparatus; 7 is a flowchart showing the flow of continuous authentication processing of the information processing device; It is a figure which shows an example of the distance image image|photographed. (A) is a diagram showing an example of an image in which only one hand is shown. (B) is a diagram showing an example of an image in which both hands are shown. It is a figure which shows an example of a synthetic image. FIG. 4 is a diagram showing an example of a normalized image; FIG. FIG. 10 is a diagram showing an example of a normalized image that is inverted upside down; FIG. 11 is a graph showing evaluation results of determination when the sum of squares of pixel value differences is used as the degree of similarity; FIG. FIG. 11 is a graph showing evaluation results of determination when the sum of absolute values of differences in pixel values is used as similarity. FIG. 10 is a graph showing evaluation results of determination when normalized cross-correlation of pixel values is used as similarity. FIG. 10 is a graph showing evaluation results of determination when zero-mean normalized cross-correlation of pixel values is used as similarity; FIG. It is a flowchart which shows the flow of the continuous authentication process of the personal identification system which concerns on Embodiment 2 of this invention. (A) and (B) are diagrams showing an example of a normalized image when the electronic device is a smartphone.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In all figures, the same reference numerals are attached to the same or corresponding components. The personal identification system 1 according to this embodiment is a system that continuously performs personal authentication.

Embodiment 1.
First, Embodiment 1 of the present invention will be described. As shown in FIG. 1, a personal identification system 1 according to this embodiment includes an image sensor 2 and an information processing device 3 .

The image sensor 2 is attached to the central upper part of the display 5 of the computer terminal 4 to be secured, as shown in FIG. 2, for example. The field of view of the image sensor 2 captures the entire keyboard 6 of the computer terminal 4, and its position is fixed.

In this embodiment, the image sensor 2 captures an image of hands (both hands) operating a keyboard 6 of a computer terminal 4 as an electronic device. The image sensor 2 is a camera that captures a distance image. A distance image is an image in which the value of each pixel indicates the distance from the image sensor 2 to the subject. In the distance image, normally, the closer the distance, the higher the luminance value.

As imaging principles for capturing such a distance image, there are a method using parallax between two cameras, a method using parallax between one projector and one camera, and a method using one projector and two cameras. There is a method using the parallax between Various patterns are projected onto the object from the projector, and the distance is measured by measuring the patterns with the camera. A method of measuring the distance based on the time difference between when light such as infrared light is emitted from a light source and reflected by an object and returned, the light receiving position, or the like can also be adopted.

The information processing device 3 is a computer that performs personal authentication based on the image captured by the image sensor 2 . The information processing device 3 includes an imaging unit 10 , an image synthesizing unit 11 , a normalizing unit 12 , an authenticating unit 13 , and storage units 20 and 21 .

The imaging unit 10 controls the image sensor 2 . Specifically, the imaging unit 10 outputs an imaging start instruction to the image sensor 2, and outputs an imaging end instruction after a certain period of time T (see FIG. 3) has elapsed. The fixed time T is the time during which a plurality of characters are manually input to the keyboard 6 of the computer terminal 4, and is set to 5 minutes, for example. The image sensor 2 starts continuous imaging when an imaging start instruction is input, and ends continuous imaging when an imaging end instruction is input. The image sensor 2 continuously captures images at regular sampling intervals, and sends the captured images to the imaging unit 10 .

That is, the imaging unit 10 controls the image sensor 2 to capture an image of the entire hand that performs operation input on the keyboard 6 of the computer terminal 4 as a subject for a certain period of time T during which a plurality of operation inputs are performed with that hand. (Refer to FIG. 3) Images are taken continuously. For example, as shown in FIG. 3, the image sensor 2 captures images at sampling times t ₁ , t ₂ , t _{3 ,} t ₄ , t ₅ , t _{6 ,} . to generate images G ₁ , G ₂ , G ₃ , _{G 4} , G ₅ , G _{6 ,} . The generated images G _n (n=1 to N) are sent to the imaging unit 10 of the information processing device 3 and then stored in the storage unit 20 .

Note that the fixed time T is not limited to 5 minutes, and may be shorter than 5 minutes or longer than 5 minutes. It is desirable that the fixed time T be a time during which an individual's characteristics appear in the degree of spread of the hand movement in the generated image. Also, the sampling interval for imaging can be arbitrarily determined.

The image synthesizing unit 11 synthesizes an image G n ′ (n=1 to N) obtained by processing a plurality of images _{G n} (n=1 to N) captured by the imaging unit 10, and in a certain time T A synthetic image SG is generated showing how the movement of the hand spreads. Specifically, as shown in FIG. 3, the image synthesizing unit 11 first divides each pixel of the image G _n (n=1 to N) by 255, which is the maximum luminance value, so that the image G _n ′ (n=1 to N) are generated. Then, the image synthesizing unit 11 adds the images G _n ′ (n=1 to N) to generate a synthetic image SG. As a result, the synthesized image SG is an image obtained by synthesizing the motions of the hands that have performed a plurality of input operations on the keyboard 6 . The generated synthetic image SG is stored in the storage unit 20 .

The normalization unit 12 normalizes the synthesized image SG synthesized by the image synthesizing unit 11 to generate a normalized image SG'. Specifically, the normalization unit 12 reads out the synthetic image SG from the storage unit 20 . Then, the normalization unit 12 searches for the luminance value of the pixel having the maximum luminance in the synthesized image SG. Then, the normalization unit 12 normalizes the synthesized image SG so that the maximum luminance value is 255 based on the searched luminance value, and obtains a normalized image SG'.

The authentication unit 13 performs personal authentication based on the normalized image SG′ generated by the normalization unit 12 . FIGS. 4(A) to 4(D) show normalized images SG' showing how different people's hand movements spread. Each normalized image SG' is an image representing a typing habit that is difficult to imitate, and differs from person to person. Therefore, personal authentication can be performed based on this normalized image SG'. The storage unit 21 stores in advance a template image TG indicating the degree of spread of hand movement associated with an individual. The authentication unit 13 performs template matching to compare the normalized image SG' and the template image TG, and calculates the similarity R between the normalized image SG' and the template image TG. The authentication unit 13 performs personal authentication based on the calculated similarity R. FIG.

FIG. 5 shows the hardware configuration of the information processing device 3 of FIG. As shown in FIG. 5 , the information processing device 3 of the personal identification system 1 includes a control section 31 , main storage section 32 , external storage section 33 , operation section 34 , display section 35 and input/output section 36 . The main storage section 32 , the external storage section 33 , the operation section 34 , the display section 35 and the input/output section 36 are all connected to the control section 31 via the internal bus 30 . Note that the information processing device 3 may be incorporated in the computer terminal 4 .

The control unit 31 is composed of a CPU (Central Processing Unit) and the like. Each component of the personal identification system 1 shown in FIG. 1 is realized by the CPU executing the program 39 stored in the external storage unit 33 . In this embodiment, the control unit 31 constitutes the image synthesizing unit 11 , the normalizing unit 12 and the authenticating unit 13 .

The main storage unit 32 is composed of a RAM (Random-Access Memory) and the like. A program 39 stored in the external storage unit 33 is loaded into the main storage unit 32 . In addition, the main storage unit 32 is used as a work area (temporary storage area for data) for the control unit 31 .

The external storage unit 33 is composed of a non-volatile memory such as a flash memory, a hard disk, a DVD (registered trademark) -RAM (Digital Versatile Disc Random-Access Memory), and a DVD (registered trademark) -RW (Digital Versatile Disc ReWritable). . A program 39 to be executed by the control unit 31 is stored in advance in the external storage unit 33 . The external storage unit 33 also supplies the control unit 31 with data used in executing the program 39 according to instructions from the control unit 31 and stores the data supplied from the control unit 31 . Storage units 20 and 21 in FIG. 1 correspond to external storage unit 33 .

The operating unit 34 includes a pointing device such as a keyboard and mouse, and an interface device that connects the keyboard, pointing device, and the like to the internal bus 30 . Information about the content of the operator's operation is input to the control unit 31 via the operation unit 34 . Note that the operation unit 34 may be the keyboard 6 .

The display unit 35 is composed of a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), an organic EL (ElectroLuminescence), or the like. When the operator inputs operation information, the display unit 35 displays an operation screen. Note that the display unit 35 may be the display 5 .

The input/output unit 36 is an interface that performs data input/output with various devices. The input/output unit 36 is connected to the image sensor 2 , outputs command data output from the control unit 31 to the image sensor 2 , and inputs captured image data from the image sensor 2 . In this embodiment, the control unit 31 and the input/output unit 36 constitute the imaging unit 10 .

Various components of the information processing device 3 are executed by a program 39 using the control unit 31, the main storage unit 32, the external storage unit 33, the operation unit 34, the display unit 35, the input/output unit 36, etc. as hardware resources. By doing so, it demonstrates its function.

Next, the operation of the personal identification system 1 according to this embodiment will be described. In the personal identification system 1, first, the template image generation process is performed, and then the continuous authentication process is performed.

First, the process of generating a template image will be described. As shown in FIG. 6, first, the imaging unit 10 captures an image of the entire hand, which is the object for inputting characters on the keyboard 6 of the computer terminal 4, using the image sensor 2. A plurality of range images G _n (n=1 to N) are obtained by continuously capturing images for a certain period of time T (step S1).

Subsequently, the image synthesizing unit 11 divides the brightness value of each pixel of the plurality of acquired range images G _n (n=1 to N) by 255 to generate range images G _n ′ (n=1 to N). Generate (step S2).

Subsequently, in step S2, the image synthesizing unit 11 adds a plurality of distance images G _{n '} (n=1 to N) obtained by dividing the distance images G _n (n=1 to N) by 255. to generate a composite image SG (step S3).

Subsequently, the normalization unit 12 searches for the maximum luminance value of the hand region of the synthesized image SG (step S4). The normalization unit 12 searches for a portion having the maximum luminance value in the hand region in the synthesized image SG, and obtains the luminance value thereof. Let X be a numerical value obtained by dividing the luminance value at this time by 255.

Subsequently, the normalization unit 12 normalizes the synthesized image SG with the found maximum value to generate a normalized image SG' having a maximum luminance value of 255 (step S5). The normalization unit 12 divides each pixel value of the synthesized image SG by X. Since the normalized image SG' is an image showing how the hand movement spreads, it is appropriate to perform normalization using the maximum luminance value of the hand region.

Subsequently, the normalization unit 12 vertically inverts the normalized image SG' (step S6). This normalized image SG' is an image as illustrated in FIGS. 4(A) to 4(D). The normalization unit 12 stores this normalized image SG' in the storage unit 21 as the template image TG (step S7). This completes the process of generating the template image TG.

Next, the continuous authentication process will be described. As shown in FIG. 7, first, the imaging unit 10 uses the image sensor 2 to capture an image of the entire hand that performs operation input on the keyboard 6 of the computer terminal 4 as a subject. A plurality of distance images G _n (n=1 to N) are obtained by continuously capturing images for a predetermined time period T (step S10; image capturing step). FIG. 8 shows an example of the distance image G _n (n=1 to N) acquired at this time. As shown in FIG. 8, in the distance image G _n (n=1 to N), the hand operating the keyboard 6 is displayed so as to be identifiable from other parts.

Subsequently, the image synthesizing unit 11 divides the brightness value of each pixel of the acquired plurality of distance images G _n (n=1 to N) by 255 to obtain the distance images G _n ′ (n=1 to N). Generate (step S11). The brightness value of each pixel of the distance image G _n ′ (n=1 to N) is 1/255 of that of each pixel of the distance image G _n (n=1 to N).

Subsequently, the image synthesizing unit 11 obtains the number of hands included as subjects in the distance image G _n (n=1 to N) (step S12). The image synthesizing unit 11 determines whether or not the number of hands is two (step S13). If the number of hands is not two (step S13; No), the information processing device 3 returns to step S10. FIG. 9A shows a distance image with one hand. In this case, the determination in step S13 is negative, and the information processing device 3 returns to step S10.

On the other hand, as shown in FIG. 9B, if the number of hands is two (step S13; Yes), the image synthesizing unit 11 generates a plurality of distance images G _n '( n=1 to N) are added together to generate a synthesized image SG (step S14; image synthesizing step). FIG. 10 shows an example of the synthetic image SG.

Subsequently, the normalization unit 12 searches for the maximum luminance value of the hand region in the synthesized image SG (step S15). Let X be a numerical value obtained by dividing the luminance value at this time by 255. Subsequently, the normalization unit 12 normalizes the synthetic image SG with the found maximum value to generate a normalized image SG' (step S16; normalization step). Specifically, the normalization unit 12 divides each pixel value of the synthesized image SG by X. FIG. 11 shows an example of the normalized image SG'. The normalization unit 12 vertically inverts the normalized image SG' (step S17). FIG. 12 shows an image flipped upside down in this way.

Subsequently, the authentication unit 13 performs template matching (step S18). Here, template matching is performed between the normalized image SG' generated in step S17 and the template image TG stored in the storage unit 20, and the degree of similarity R between the two is calculated.

上記式（１）のＲ_ｓｓｄは、正規化画像ＳＧ’とテンプレート画像ＴＧとの各画素の輝度値の差の二乗和（ＳＳＤ：Sum of Squares Difference）を示す。Ｒ_ｓｓｄは、０に近づけば近づくほど（低ければ低いほど）、類似度Ｒが高いということになる。認証部１３が、類似度Ｒとして二乗和Ｒ_ｓｓｄを算出する場合には、計算量を少なくすることができるという特徴がある。

上記式（２）のＲ_ｓａｄは、正規化画像ＳＧ’とテンプレート画像ＴＧとの各画素の輝度値の差の絶対値和を示す。Ｒ_ｓａｄは、０に近づけば近づくほど（低ければ低いほど）、類似度Ｒが高いということになる。認証部１３が、類似度Ｒとして絶対値Ｒ_ｓａｄ（ＳＡＤ：Sum of Absolute Difference）を算出する場合には、外れ値（ノイズ等の原因より統計的に見て誤差成分の大きい画素の輝度値）の影響を受けにくいという利点がある。

上記式（３）のＲ_ｎｃｃは、正規化画像ＳＧ’とテンプレート画像ＴＧとの各画素の輝度値の正規化相互相関を示す。認証部１３が、類似度Ｒとして正規化相互相関Ｒ_ｎｃｃを算出する場合には、正規化画像ＳＧ’が撮像された時の周囲の明るさと、テンプレート画像ＴＧが撮像された時の周囲の明るさとの違いによる影響を受けにくいという利点がある。

上記式（４）のＲ_ｚｎｃｃは、正規化画像ＳＧ’とテンプレート画像ＴＧとの各画素の輝度値の零平均正規化相互相関を示す。認証部１３が、類似度Ｒとして、零平均正規化相互相関Ｒ_ｚｎｃｃを算出する場合には、正規化相互相関Ｒ_ｎｃｃよりも、正規化画像ＳＧ’が撮像された時の周囲の明るさと、テンプレート画像ＴＧが撮像された時の周囲の明るさとの違いによる影響を受けにくいという利点がある。 As the degree of similarity R, any one of R _ssd , R _sad , R _ncc and R _zncc represented by the following formulas can be adopted. Here, I(i, j) indicates the brightness value of each pixel (i, j) of the normalized image SG′, and T(i, j) indicates the brightness value of each pixel (i, j) of the template image TG. Indicates luminance value. i and j indicate the position of each pixel on the horizontal axis and the position on the vertical axis.

R _ssd in the above formula (1) represents the sum of squares (SSD: Sum of Squares Difference) of the difference in luminance value of each pixel between the normalized image SG′ and the template image TG. The closer R _ssd is to 0 (the lower it is), the higher the similarity R is. When the authenticating unit 13 calculates the sum of squares R _ssd as the degree of similarity R, the amount of calculation can be reduced.

R _sad in the above equation (2) indicates the sum of absolute values of differences in luminance values of pixels between the normalized image SG′ and the template image TG. The closer R _sad is to 0 (the lower it is), the higher the similarity R is. When the authentication unit 13 calculates the absolute value R _sad (SAD: Sum of Absolute Difference) as the degree of similarity R, outliers (brightness values of pixels with statistically large error components due to causes such as noise) has the advantage of being less susceptible to

R _ncc in the above equation (3) represents the normalized cross-correlation of the luminance value of each pixel between the normalized image SG′ and the template image TG. When the authentication unit 13 calculates the normalized cross-correlation R _ncc as the degree of similarity R, the ambient brightness when the normalized image SG′ was captured and the ambient brightness when the template image TG was captured There is an advantage that it is less susceptible to the difference between the

R _zncc in the above equation (4) represents the zero-mean normalized cross-correlation of the brightness values of each pixel between the normalized image SG′ and the template image TG. When the authenticating unit 13 calculates the zero-mean normalized cross-correlation R _zncc as the similarity R, the brightness of the surroundings when the normalized image SG′ was captured and the normalized cross-correlation R _ncc This has the advantage of being less susceptible to the difference in ambient brightness when the template image TG was captured.

Subsequently, the authentication unit 13 performs personal authentication depending on whether the calculated degree of similarity R exceeds a threshold (step S19; authentication step). Specifically, if the degree of similarity R obtained as described above is higher than the threshold value, the identity of the person is authenticated, and use of the computer terminal 4 is allowed to continue. On the other hand, if the degree of similarity R is lower than the threshold, it is assumed that the person is a different person, and use of the computer terminal 4, for example, is prohibited.

FIG. 13 shows a graph where the similarity R is the sum of squares _Rssd of the pixel value differences between the normalized image SG′ and the template image TG. As shown in FIG. 13, when the normalized image SG′ and the template image TG are of the same person, the value of the sum of squares R _ssd is close to 0. It's getting bigger. Therefore, if a threshold value is set between the area of the same person (dotted line) and the area of a different person (solid line) in this graph, the person can be authenticated with high accuracy.

FIG. 14 shows a graph in which the similarity R is defined as the sum of absolute values R _sad of the pixel value differences between the normalized image SG′ and the template image TG. As shown in FIG. 14, when the normalized image SG′ and the template image TG are of the same person, the value of the absolute value sum R _sad is close to 0, and when they are of different persons, the value is 0 getting bigger. Therefore, if a threshold value is set between the area of the same person (dotted line) and the area of a different person (solid line) in this graph, the person can be authenticated with high accuracy.

FIG. 15 shows a graph where the similarity R is the normalized cross-correlation _Rncc of the brightness values of the pixels of the normalized image SG′ and the template image TG. The vertical axis is _Rncc . As shown in FIG. 15, when the normalized image SG′ and the template image TG are of the same person, the value of the normalized cross-correlation R _ncc is close to 1, and when they are of different persons, is smaller than 1. Therefore, if a threshold value is set between the area of the same person (dotted line) and the area of a different person (solid line) in this graph, the person can be authenticated with high accuracy.

FIG. 16 shows a graph in which the similarity R is defined as the zero-mean normalized cross-correlation R _zncc of the luminance value difference of each pixel between the normalized image SG′ and the template image TG. The vertical axis is _Rzncc . As shown in FIG. 16, when the normalized image SG′ and the template image TG are of the same person, the value of the zero-mean normalized cross-correlation R _zncc is close to 1, and when they are of different persons, is less than 1. Therefore, if a threshold value is set between the area of the same person (dotted line) and the area of a different person (solid line) in this graph, the person can be authenticated with high accuracy.

In this way, it has been clarified that the degree of similarity R can be used to clearly separate the degree of similarity R of the same person from the degree of similarity R of a different person. Therefore, by comparing the degree of similarity R with a threshold value, personal authentication becomes possible.

Embodiment 2.
Next, Embodiment 2 of the present invention will be described. A personal identification system 1 according to the present embodiment differs from the personal identification system 1 according to the first embodiment in that it includes an image sensor 2 and an information processing device 3 .

The configuration of the personal identification system 1 according to this embodiment is the same as that of the personal identification system 1 according to the first embodiment shown in FIG. However, in the present embodiment, the image sensor 2 is a sensor (two-dimensional image sensor sensitive to the visible light range) that captures an image using visible light, not a distance image.

As shown in FIG. 17, in the present embodiment, between image acquisition (step S10) and image division (step S11), the image synthesizing unit 11 binarizes the image (step S30 ). Specifically, the image synthesizing unit 11 converts the image captured by the imaging unit 10 into a binary image that distinguishes between the hand area and other areas. This binary image is an image in which the luminance of the hand area is high and the luminance of the other areas is low.

Other processes in FIG. 17 are the same as those in the first embodiment. After that, the imaging unit 10 divides the value of each pixel of the binary image by 255 (step S11), and the image synthesizing unit 11 adds the divided binary images to generate a synthesized image SG (step S14). ). Then, the authentication unit 13 generates a normalized image SG' based on the generated synthesized image SG (step S17). Template matching is performed using this normalized image SG'.

As described in detail above, according to the above-described embodiments, the motion of the hand for inputting an operation to the keyboard 6 of the computer terminal 4 is continuously imaged while a plurality of input operations are being performed, generates a normalized image SG' showing how the motion spreads. Since the spread of hand movements when performing a plurality of input operations is completely different from person to person, an individual can be identified from the normalized image SG' showing the spread of hand movements. In this way, continuous authentication can be naturally performed simply by imaging the movement of the hand with the image sensor 2 without providing special equipment.

In each of the above embodiments, a normalized image SG′ (FEI (Finger Energy Image ) image) to perform continuous authentication. In the normalized image SG', the image appears white where the hand hardly moves, and the image becomes blurred where the hand frequently moves. As a result, the normalized image SG' becomes an image showing the spread of the movement of the hand operating the keyboard 6, and an image showing individual characteristics.

According to each of the embodiments described above, personal authentication is performed based on the degree of spread of hand movements, which indicates differences in hand movements (fingering) during typing on the keyboard 6 . Therefore, there is no need to detect typing accuracy, speed, or key press strength.

In addition, since it is not necessary to input a specific keyword or phrase for authentication, the authentication can be performed without interrupting the work. Therefore, the convenience of the computer terminal 4 can be maintained. Also, there is no need to perform authentication using personal physical features such as fingerprints and irises.

In each of the above-described embodiments, personal authentication is performed based on the extent of movement of the hand operating the keyboard 6 of the computer terminal 4, but the present invention is not limited to this. For example, as shown in FIGS. 18(A) and 18(B) , a normalized image SG′ is generated to show how the movement of the hand operating the smartphone spreads, and based on the generated normalized image SG′, , authentication may be performed. In this case, the position of the image sensor 2 and the position of the smartphone are shifted each time the image is captured. There is a need to.

In each of the above embodiments, the captured image is divided by 255, but the present invention is not limited to this. You may make it synthesize|combine the imaged image as it is. Also, in each of the above embodiments, the normalized image SG' is vertically inverted, but the present invention is not limited to this. Template matching may be performed without inverting the normalized image SG'.

In addition, the hardware configuration and software configuration of the personal identification system 1 are examples, and can be arbitrarily changed and modified.

The main processing part of the personal identification system 1, which is composed of the control unit 31, the main storage unit 32, the external storage unit 33, the operation unit 34, the display unit 35, the input/output unit 36, the internal bus 30, etc., is It can be realized using a normal computer system regardless of the system. For example, a computer program for executing the above operation is stored in a computer-readable recording medium (flexible disk, CD-ROM, DVD (registered trademark) -ROM, etc.) and distributed, and the computer program is stored in a computer. The personal identification system 1 that performs the above processing may be configured by installing the personal identification system 1 in the . Alternatively, the personal identification system 1 may be configured by storing the computer program in a storage device of a server device on a communication network such as the Internet, and downloading the program to a normal computer system.

When the functions of the personal identification system 1 are realized by sharing the responsibility of an OS (operating system) and an application program, or by cooperation between the OS and an application program, only the application program portion is stored in a recording medium or storage device. good too.

It is also possible to superimpose a computer program on a carrier wave and distribute it via a communication network. For example, the computer program may be posted on a bulletin board system (BBS, Bulletin Board System) on a communication network and distributed over the network. Then, the computer program may be configured to be started and executed in the same manner as other application programs under the control of the OS so that the above processes can be executed.

The present invention is capable of various embodiments and modifications without departing from the broader spirit and scope of the invention. Moreover, the embodiment described above is for explaining the present invention, and does not limit the scope of the present invention. That is, the scope of the present invention is indicated by the claims rather than the embodiments. Various modifications made within the scope of the claims and within the meaning of equivalent inventions are considered to be within the scope of the present invention.

The present invention can be applied to continuous personal authentication.

1 personal identification system, 2 image sensor, 3 information processing device, 4 computer terminal, 5 display, 6 keyboard, 10 imaging unit, 11 image synthesis unit, 12 normalization unit, 13 authentication unit, 20, 21 storage unit, 30 inside bus, 31 control unit, 32 main storage unit, 33 external storage unit, 34 operation unit, 35 display unit, 36 input/output unit, 39 program

Claims (7)

an imaging unit that captures an image of an entire hand that performs an operation input to an electronic device as a subject, with an image sensor continuously for a certain period of time during which a plurality of operation inputs are performed with the hand;
Synthesizing a plurality of images captured by the image sensor or synthesizing a plurality of images obtained by processing the plurality of images to generate a synthetic image showing how the hand movement spreads during the predetermined period of time. an image synthesizing unit to generate;
a normalization unit that normalizes the synthesized image synthesized by the image synthesizing unit with reference to the maximum value of pixels in the synthesized image to generate a normalized image;
an authentication unit that performs personal authentication based on the normalized image generated by the normalization unit;
with
The image sensor captures a distance image in which the value of each pixel indicates the distance between the image sensor and the subject as an image of the entire hand as a subject;
The image synthesizing unit performs a process of dividing the value of each pixel by the maximum value that the pixel can take on a plurality of images captured by the image sensor,
The authentication unit
calculating a degree of similarity between the normalized image and the template image by performing template matching for comparing the normalized image with a pre-stored template image;
If the calculated similarity exceeds the threshold, the person is authenticated;
personal identification system. an imaging unit that captures an image of an entire hand that performs an operation input to an electronic device as a subject, with an image sensor continuously for a certain period of time during which a plurality of operation inputs are performed with the hand;
an image synthesizing unit for synthesizing a plurality of images obtained by processing a plurality of images captured by the image sensor to generate a synthetic image showing how the movement of the hand spreads during the predetermined time period;
a normalization unit that normalizes the synthesized image synthesized by the image synthesizing unit with reference to the maximum value of pixels in the synthesized image to generate a normalized image;
an authentication unit that performs personal authentication based on the normalized image generated by the normalization unit;
with
The image sensor is
Capturing an image with visible light as an image of the entire hand as a subject,
The image synthesizing unit
A plurality of images captured by the image sensor are converted into a binary image that distinguishes the hand region from other regions, and the value of each pixel of the binary image is divided by the maximum value that the pixel can take. and
The authentication unit
calculating a degree of similarity between the normalized image and the template image by performing template matching for comparing the normalized image with a pre-stored template image;
If the calculated similarity exceeds the threshold, the person is authenticated;
personal identification system. The authentication unit
Is the sum of the squares of the difference in value between each pixel of the normalized image and each pixel of the template image, or the sum of the absolute values of the difference between each pixel of the normalized image and each pixel of the template image? , a normalized cross-correlation of the values of each pixel of the normalized image and each pixel of the template image, or a zero-mean normalized cross-correlation of the values of each pixel of the normalized image and each pixel of the template image. Calculate either the correlation or as the similarity,
3. Personal identification system according to claim 1 or 2. an imaging step of capturing an image of an entire hand that performs an operation input on an electronic device as a subject, with an image sensor continuously for a certain period of time during which a plurality of operation inputs are performed with the hand;
Synthesizing a plurality of images captured in the imaging step or synthesizing a plurality of images obtained by processing the plurality of images to generate a synthetic image showing how the movement of the hand spreads during the predetermined period of time. an image synthesizing step to generate;
a normalization step of normalizing the synthesized image synthesized in the image synthesizing step with reference to the maximum value of pixels in the synthesized image to generate a normalized image;
an authentication step of performing personal authentication based on the normalized image generated in the normalization step;
including
The image sensor captures a distance image in which the value of each pixel indicates the distance between the image sensor and the subject as an image of the entire hand as a subject;
In the image synthesizing step, a plurality of images captured by the image sensor are processed by dividing the value of each pixel by the maximum value that the pixel can take,
In said authentication step,
calculating a degree of similarity between the normalized image and the template image by performing template matching for comparing the normalized image with a pre-stored template image;
If the calculated similarity exceeds the threshold, the person is authenticated;
Personal identification method. an imaging step of capturing an image of an entire hand that performs an operation input on an electronic device as a subject, with an image sensor continuously for a certain period of time during which a plurality of operation inputs are performed with the hand;
an image synthesizing step of synthesizing a plurality of images obtained by processing the plurality of images captured in the imaging step to generate a synthetic image showing how the hand movement spreads during the predetermined time;
a normalization step of normalizing the synthesized image synthesized in the image synthesizing step with reference to the maximum value of pixels in the synthesized image to generate a normalized image;
an authentication step of performing personal authentication based on the normalized image generated in the normalization step;
including
The image sensor is
Capturing an image with visible light as an image of the entire hand as a subject,
In the image synthesizing step,
A plurality of images captured by the image sensor are converted into a binary image that distinguishes the hand region from other regions, and the value of each pixel of the binary image is divided by the maximum value that the pixel can take. and
In said authentication step,
calculating a degree of similarity between the normalized image and the template image by performing template matching for comparing the normalized image with a pre-stored template image;
If the calculated similarity exceeds the threshold, the person is authenticated;
Personal identification method. the computer,
An imaging unit that captures an image of the entire hand that performs operation input on an electronic device as a subject, with an image sensor continuously for a certain period of time during which a plurality of operation inputs are performed with the hand;
Synthesizing a plurality of images captured by the image sensor or synthesizing a plurality of images obtained by processing the plurality of images to generate a synthetic image showing how the hand movement spreads during the predetermined period of time. an image synthesizing unit to generate,
a normalization unit that normalizes the synthesized image synthesized by the image synthesizing unit with reference to the maximum value of pixels in the synthesized image to generate a normalized image;
an authentication unit that performs personal authentication based on the normalized image generated by the normalization unit;
function as
The image sensor captures a distance image in which the value of each pixel indicates the distance between the image sensor and the subject as an image of the entire hand as a subject;
The image synthesizing unit performs a process of dividing the value of each pixel by the maximum value that the pixel can take on a plurality of images captured by the image sensor,
The authentication unit
calculating a degree of similarity between the normalized image and the template image by performing template matching for comparing the normalized image with a pre-stored template image;
If the calculated similarity exceeds the threshold, the person is authenticated;
program. the computer,
An imaging unit that captures an image of the entire hand that performs operation input on an electronic device as a subject, with an image sensor continuously for a certain period of time during which a plurality of operation inputs are performed with the hand;
an image synthesizing unit for synthesizing a plurality of images obtained by processing a plurality of images captured by the image sensor to generate a synthetic image showing how the movement of the hand spreads during the predetermined time period;
a normalization unit that normalizes the synthesized image synthesized by the image synthesizing unit with reference to the maximum value of pixels in the synthesized image to generate a normalized image;
an authentication unit that performs personal authentication based on the normalized image generated by the normalization unit;
function as
The image sensor is
Capturing an image with visible light as an image of the entire hand as a subject,
The image synthesizing unit
A plurality of images captured by the image sensor are converted into a binary image that distinguishes the hand region from other regions, and the value of each pixel of the binary image is divided by the maximum value that the pixel can take. and
The authentication unit
calculating a degree of similarity between the normalized image and the template image by performing template matching for comparing the normalized image with a pre-stored template image;
If the calculated similarity exceeds the threshold, the person is authenticated;
program.

Images