JP6926351B1 - Terminal devices, information processing methods, and programs - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a terminal device, an information processing method, and a program capable of stably authenticating a user regardless of a usage state. SOLUTION: A terminal device 1 has succeeded in authentication based on an authentication information acquisition unit that acquires a plurality of authentication information and one authentication information or a plurality of authentication information acquired by the authentication information acquisition unit. For authentication that updates a plurality of stored authentication information by the authentication judgment unit that determines whether or not, and when the authentication judgment unit determines that the authentication is successful, the authentication information acquisition unit acquires a plurality of authentication information. It is equipped with an information update unit, and in the background, the authentication determination unit determines that the authentication in its first usage state is successful when one authentication information is less than a predetermined authentication threshold, and one authentication information. Whether or not the authentication in the second usage state of the user succeeded based on the second authentication information as the authentication in the first usage state failed when is equal to or more than the authentication threshold and equal to or more than the predetermined authentication tolerance value. Is determined. [Selection diagram] Fig. 1

Description

The present invention relates to terminal devices, information processing methods, and programs.

Normally, the terminal device is provided with an authentication function for preventing unauthorized use, and each function can be used when the authentication is successful. In addition to general passwords, various types of biometric information are also used in the authentication process. For example, Patent Document 1 discloses a technique in which an image including an image of a user's face, iris, etc. is photographed with a camera, and the user is authenticated by the feature amount of the face, iris, etc. obtained by analyzing the photographed image. Has been done.

Japanese Unexamined Patent Publication No. 2020-042404

In the technique disclosed in Patent Document 1, when the contact sensor detects that the user has come into contact with the information processing device, the camera starts taking an image including an image of the user's face, iris, or the like. Then, when the contact sensor cannot detect that the user is in contact with the information processing device, the camera stops taking an image including an image of the user's face, iris, or the like. Therefore, in the technology disclosed in Patent Document 1, for example, when the information processing device is used hands-free, or when the information processing device is used while being stored in a bag, pocket, or the like, the user can use the information processing device. Since there is no contact, the user cannot be authenticated. Therefore, the technique disclosed in Patent Document 1 has a problem that the user cannot be stably authenticated as the person himself / herself depending on the usage state of the information processing device.

The present invention solves the above-mentioned problems, and an object of the present invention is to provide a terminal device, an information processing method, and a program capable of stably authenticating a user regardless of a usage state.

In order to achieve the above object, the terminal device according to the present invention is
An authentication information acquisition unit that acquires multiple authentication information,
Based on at least one of authentication information or credentials among the plurality of authentication information acquired in the authentication information acquiring unit, and determines the authentication assessment section whether the authentication is successful,
If it is determined that the authentication by the authentication determination unit has succeeded, by the plurality of authentication information acquired by the authentication information acquiring section, and a authentication information update section for updating the plurality of authentication information stored ,
In the background, the authentication determination unit performs authentication in which a first authentication value obtained by comparing one authentication information acquired by the authentication information acquisition unit with one registration information registered in itself is predetermined. If it is less than the threshold value, it is determined that the authentication in the first usage state of the user is successful, and the first authentication value is a value including the authentication allowable range value indicating the case where the user is the person or not in the authentication threshold value. whether authentication in the second use state of the own based on some the second authentication information acquired by the authentication information acquiring section as the authentication allowable value or more of authentication in the first use state when the failure has succeeded If the first authentication value is equal to or more than the authentication threshold value and less than the authentication allowable value, it is determined whether or not the authentication is successful based on the three authentication information acquired by the authentication information acquisition unit. If the determination is made and the authentication is successful based on the three authentication information, it is assumed that the user is authenticated based on the one authentication information .

According to the terminal device according to the present invention, when authentication by one authentication information is not possible in the background, it is determined whether or not the authentication in its own second usage state is successful based on the second authentication information. Therefore, it is possible to authenticate the user stably regardless of the usage state of the terminal device.

It is a front view of the terminal apparatus which concerns on embodiment of this invention. It is a block diagram of the terminal apparatus shown in FIG. It is a figure which shows an example of the hardware composition of the terminal apparatus shown in FIG. It is a figure of the information processing block of the terminal apparatus shown in FIG. It is a figure which shows the table of the biometric information database for authentication acquired from the terminal apparatus shown in FIG. It is a figure which shows the table of the behavior information database for authentication acquired from the terminal apparatus shown in FIG. It is a figure which shows the inclination information table of the terminal apparatus shown in FIG. It is a flowchart of the authentication process which concerns on embodiment. It is the continuation flowchart of the flowchart of the authentication process shown in FIG. 6A. It is a flowchart of the face recognition value acquisition process which concerns on embodiment. It is a flowchart of auxiliary authentication processing which concerns on embodiment. It is a flowchart of the voice acquisition process which concerns on embodiment. It is a flowchart of voice authentication processing which concerns on embodiment.

Hereinafter, the terminal device, the information processing method, and the program according to the embodiment of the present invention will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are designated by the same reference numerals.

The terminal device 1 according to the present embodiment is based on biological information such as an image, fingerprint, and voice of the user's face, and behavior information based on a unique behavior when the user operates the terminal device 1, an operation state, and the like. This is a device that can authenticate the user himself / herself and execute various functions in the terminal device 1. In particular, in the terminal device 1, if the image of the user's face cannot be taken depending on the usage situation, such as when the terminal device 1 is stored in a bag or pocket, and authentication by the face image cannot be performed, the user's own voice is used. The user can be authenticated.

Further, the user authentication process performed in the terminal device 1 is executed in the background while the terminal device 1 is in operation, and the accuracy of authentication is improved by updating the user's biometric information and behavior information. It is a process that can be made to do.

A front view of the terminal device 1 is shown in FIG. The terminal device 1 is a so-called smartphone, and the terminal device 1 has an in-camera 11A for photographing the user's face in front, a speaker 12A, a microphone 12B which is a microphone for talking, and a tilt for detecting the tilt of the terminal device 1. A touch panel that also serves as a detection unit 13, an operation input unit 14, and a display unit 19, a left fingerprint sensor 15A and a right fingerprint sensor 15B that detect a user's fingerprint, and a position detection unit 16 that detects the current position of the terminal device 1. Be prepared. Further, the terminal device 1 is provided with a main camera 11B on the back surface capable of photographing a person, a landscape, an object, etc. as seen by the user.

Here, in the following, the in-camera 11A and the main camera 11B are collectively referred to as a photographing unit 11. Hereinafter, the speaker 12A and the microphone 12B, which is a microphone for telephone calls, are collectively referred to as an audio input / output unit 12. Further, in the following, the left fingerprint sensor 15A and the right fingerprint sensor 15B are collectively referred to as a fingerprint detection unit 15.

FIG. 2 is a block diagram showing the configuration of the terminal device 1. The terminal device 1 includes a communication unit 10, a photographing unit 11, a voice input / output unit 12, a tilt detection unit 13, an operation input unit 14, a fingerprint detection unit 15, a position detection unit 16, and a terminal storage unit 17. And a terminal control unit 18 and a display unit 19.

The communication unit 10 communicates with an external server, cloud, etc. via a communication network (not shown) to transmit / receive various data, and a wireless signal for telephone communication between a data communication unit (not shown) and a base station (not shown). Includes a voice communication unit that transmits and receives data. The data communication unit can be configured by using a wireless LAN (Local Area Network), Wi-fi (registered trademark), Bluetooth (registered trademark), or the like. Further, the voice communication unit can be configured by using a communication device that transmits / receives a wireless signal for telephone communication with the base station.

The photographing unit 11 includes the in-camera 11A and the main camera 11B shown in FIG. The photographing unit 11 captures a still image or a moving image such as a camera using an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Sensor) image sensor, a video camera, or the like, and captures the still image or the moving image. Various cameras that can be acquired can be used.

The audio input / output unit 12 includes the speaker 12A shown in FIG. 1 and the microphone 12B. The speaker 12A outputs voice received in a voice call, music data acquired from the outside via a communication network, and the like. The microphone 12B is a device that picks up the voice of the user.

The tilt detection unit 13 is a device capable of detecting tilt, shaking, etc. of the terminal device 1. The tilt detection unit 13 can be configured by using various sensors that can detect the tilt of the terminal device 1, such as an acceleration sensor, an angle sensor, and a magnetic sensor that detects the geomagnetism. The number and types of sensors constituting the tilt detection unit 13 may be either a single number or a plurality of sensors.

The operation input unit 14 is a device capable of inputting an operation from the user shown in FIG. The fingerprint detection unit 15 is a sensor that detects the user's fingerprint. The fingerprint detection unit 15 includes the left fingerprint sensor 15A and the right fingerprint sensor 15B shown in FIG. The fingerprint detection unit 15 is not limited to the fingerprint sensor, and any sensor, device, or the like capable of detecting the user's fingerprint may be used.

The position detection unit 16 is a device capable of detecting the current position of the terminal device 1. The position detection unit 16 can be configured by using a device such as GPS (Global Positioning System) that can detect the current position of the terminal device 1.

The terminal storage unit 17 includes an authentication processing program 170 for performing user authentication processing, an authentication biometric information database 171 that summarizes user biometric information acquired by the terminal device 1, and user behavior acquired by the terminal device 1. It includes an authentication behavior information database 172 that summarizes information, and a tilt information table 173 for storing the tilt state of the terminal device 1. Further, the terminal storage unit 17 stores programs of various applications executed by the terminal device 1.

The authentication processing program 170 is a program that performs processing for authenticating a user based on the biometric information and behavior information of the user acquired by the terminal device 1. The authentication biometric information database 171 is a database for storing information on the biometric information of the user and the authentication value used for authentication.

The authentication behavior information database 172 is a database for storing information on user-specific behavior when operating the terminal device 1, authentication pass conditions, and the like. Here, the behavior peculiar to the user is the behavior when the user operates the terminal device 1, the distance between the screen of the display unit 19 and the user's face, the keystroke, the way of holding, the position where the terminal device 1 is used, and the specification. Refers to user-specific things such as the number of connections to the communication network, launching of a specific application, and operation.

The tilt information table 173 is a table for storing the tilt angle of the terminal device 1 detected by the tilt detection unit 13, the acquisition date and time, and the waiting time for acquisition. The details of the authentication processing program 170, the biological information database 171 for authentication, the behavior information database 172 for authentication, and the inclination information table 173 will be described later.

The terminal control unit 18 executes various programs stored in the terminal storage unit 17. Further, the terminal control unit 18 includes a communication unit 10, a photographing unit 11, an audio input / output unit 12, a tilt detection unit 13, an operation input unit 14, a fingerprint detection unit 15, and a position detection unit 16. Data is acquired, processed, and stored in various databases and tables of the terminal storage unit 17. Further, the terminal control unit 18 can cause the photographing unit 11 to take a picture at an arbitrary timing by transmitting an instruction to take a picture to the taking picture unit 11.

The display unit 19 displays the processing contents of various programs executed by the terminal control unit 18. Further, the display unit 19 can also display images such as still images and moving images shot by the shooting unit 11, data input from the operation input unit 14, and the like. The display unit 19 is laminated on the operation input unit 14 and constitutes the touch panel shown in FIG.

Next, an example of the hardware configuration of the terminal device 1 will be described with reference to FIG. The terminal device 1 includes a processor 21 for executing various programs, a memory 22 for deploying various programs, a display controller 23 for outputting various display data, a display device 24 for displaying various display data, and photographing. It includes an I / O port 25 for connecting a unit 11, an audio input / output unit 12, and a storage device 26 for storing various programs and various data, and a communication device 27 for communicating with the outside and transmitting and receiving various data. .. The processor 21, the memory 22, the display controller 23, the display device 24, the I / O port 25, the storage device 26, and the communication device 27 are connected to each other via the data bus 28.

The processor 21 reads various programs stored in the storage device 26, expands them in the memory 22, and executes them. The processor 21 can be configured by using a processing device such as a CPU (Central Processing Unit) and an MPU (Micro-processing Unit). Further, the memory 22 can be configured by using a storage element and a storage medium such as a volatile or non-volatile semiconductor memory such as a RAM (Random Access Memory) and a flash memory.

The display controller 23 is a controller that outputs various display data to the display device 24. The display controller 23 can be configured by using a video signal output device such as a video card, a GPU (Graphics Processing Unit), or a graphic board. Further, the display device 24 can be configured by using a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) monitor.

The I / O port 25 is for connection that can connect the photographing unit 11, the audio input / output unit 12, the tilt detection unit 13, the operation input unit 14, the fingerprint detection unit 15, and the position detection unit 16. It is a port. The I / O port 25 can be configured by using various ports to which devices can be connected, such as a USB (Universal Serial Bus) port and an IEEE1394 port.

The storage device 26 is a device that stores various programs executed by the processor 21 and various data for use in the various programs. The storage device 26 can be configured by using a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

The communication device 27 communicates with an external server, cloud, etc. via a communication network (not shown), and transmits / receives various data between a data communication unit and a base station (not shown), which is a wireless signal for telephone communication. Includes a voice communication unit that transmits and receives data. The data communication unit can be configured by using a wireless LAN, Wi-fi (registered trademark), Bluetooth (registered trademark), or the like. Further, the voice communication unit can be configured by using a communication device that transmits / receives a wireless signal for telephone communication with the base station.

The information processing block shown in FIG. 4 is realized in the terminal control unit 18 by executing the authentication processing program 170 stored in the terminal storage unit 17 of the terminal device 1 shown in FIG. 2 by the processor 21 described above. As a result, the terminal device 1 is based on biological information such as an image, fingerprint, and voiceprint of the user's face, and behavior information based on the user's peculiar behavior when operating the terminal device 1, the operation state, and the like. The person can be authenticated and various functions in the terminal device 1 can be executed.

The information processing block includes an authentication information acquisition unit 181 that acquires biometric information and behavior information for authentication from the communication unit 10, the photographing unit 11, and the like, an authentication determination unit 182 that authenticates whether or not the user is the person, and a display unit 19. The authentication result display unit 183 for displaying the authentication result, and the authentication information update unit 184 for updating the information of various databases and tables stored in the terminal storage unit 17 according to the instruction from the authentication determination unit 182.

The authentication information acquisition unit 181 acquires biometric information and behavior information for authentication from the communication unit 10, the photographing unit 11, and the like. The authentication determination unit 182 authenticates the user based on the authentication biometric information and behavior information acquired from the authentication information acquisition unit 181 and the authentication values, pass conditions, etc. stored in various databases of the terminal storage unit 17. I do. The authentication result display unit 183 receives the user's authentication result from the authentication determination unit 182, and causes the display unit 19 to display a message, an image, or the like according to the authentication result. The authentication information update unit 184 updates the data stored in various databases and tables stored in the terminal storage unit 17 based on the instruction from the authentication determination unit 182. The biological information and behavior information acquired by the authentication information acquisition unit 181 are examples of authentication information within the scope of claims. Further, each biometric information acquired by the authentication information acquisition unit 181 is an example of one information and two information within the scope of claims. Each behavior information acquired by the authentication information acquisition unit 181 is an example of three pieces of information within the scope of claims.

Next, the configurations of the authentication biometric information database 171, the authentication behavior information database 172, and the tilt information table 173 stored in the terminal storage unit 17 are described below with reference to FIGS. 5A to 5C. explain. First, as shown in FIG. 5A, in the table of the biometric information database 171 for authentication, the types of biometric information such as face and voice, the registration information which is the biometric information of the user himself, the registration information, and the registration information are shown in FIG. The authentication value obtained by comparing with the biometric information acquired by the authentication information acquisition unit 181 is stored.

In the present embodiment, three types of data of voiceprint, mouth habit, and intonation are stored in the authentication biometric information database 171 as voice biometric information. Data such as voiceprint, habit, intonation, etc. of voice are data that can be used for personal authentication by the filler function.

The registered information stored in the table of the authentication biometric information database 171 is the biometric information of the user himself / herself. The registration information is information registered in advance before the authentication process is performed on the terminal device 1, and is updated when the user himself / herself can be authenticated. For the registered information, for example, if the type of biometric information is a face, the feature amount obtained from the face image is used. If the type of biometric information is iris, the iris data is stored, and if the type of biometric information is fingerprint, the feature amount obtained from the fingerprint image is stored. Of the types of biometric information, "face" is an example of the first biometric information within the scope of claims. Further, among the types of biometric information, "voice" is an example of the second biometric information within the scope of claims.

In the present embodiment, the determination of similarity of biometric information is performed by the authentication value. The authentication value is a value obtained based on the result of comparing the registered information with the biological information acquired by the authentication information acquisition unit 181 shown in FIG. The authentication value approaches 0 when the registered information and the biometric information acquired by the authentication information acquisition unit 181 are similar, and approaches 1 when they are not similar. The biometric information database 171 for authentication includes an average value of authentication values, an authentication threshold that is a threshold for determining the authentication value, and an authentication permissible value that indicates the case where the user is gray in the authentication threshold. And are included. The authentication value stored in association with the biometric information of voice is an authentication value obtained by averaging each authentication value of voiceprint, habit, and intonation of voice.

First, the average value of the authentication values is the average value of the authentication values obtained by comparing the registered information with the biometric information acquired by the authentication information acquisition unit 181. The authentication threshold is determined by comparing the registered information with the biometric information acquired by the authentication information acquisition unit 181 and determining that the user is the user himself / herself if the authentication value obtained based on the comparison result is equal to or less than this value. It is a standard value for. The authentication threshold value is a value that fluctuates according to the authentication status of the user, and an upper limit value is set in advance. The upper limit value is a value at which the user should not be authenticated only with the user himself / herself and biometric information when the value exceeds the value. For example, if the default value of the authentication threshold value is 0.4 between 0, which approaches the case where the registered information and the biometric information acquired by the authentication information acquisition unit 181 are similar, and 1 which approaches the case where the information is not similar, the authentication is performed. The upper limit of the threshold value is 0.45.

Further, the authentication allowable value is obtained by comparing the registered information with the biometric information acquired by the authentication information acquisition unit 181 and when the authentication value obtained based on the comparison result is equal to or more than this value, the user is the user himself / herself. It is a reference value for determining that there is no such thing. Since the authentication permissible value is a value including the authentication permissible range value indicating the case where the user is gray as described above, it is a value that fluctuates according to the fluctuation between the authentication threshold value and the authentication permissible range value. An upper limit is set in advance for the authentication tolerance, and this is called the maximum authentication tolerance. The maximum authentication allowable value is a value at which the user should be judged as another person when the value is equal to or more than this value. For example, it is 0.5, which is intermediate between 0, which approaches the case where the registered information and the biometric information acquired by the authentication information acquisition unit 181 are similar, and 1 which approaches the case where the information is not similar.

The value between the authentication threshold value and the authentication tolerance value is called an authentication tolerance range value, and is a value indicating whether or not the user is the user himself / herself in gray. When the authentication value is within the authentication allowable range value, it is not judged only by the biometric information whether or not the user is the user himself / herself, and when the behavior information peculiar to the user matches the pass condition, the user is authenticated with the user himself / herself. If they do not match, the user will not be authenticated. User authentication based on behavior information is hereinafter referred to as auxiliary authentication. The authentication allowable range value is a predetermined value that the user can generally think of as long as the authentication value falls within this range.

For example, it is 0.08, which is less than 10% of 0 that approaches a similar case and 1 that approaches a dissimilar case. When the authentication threshold reaches the upper limit, the authentication threshold becomes the value obtained by subtracting the upper limit from the maximum authentication allowance. For example, if the upper limit of the authentication threshold is 0.45 and the maximum authentication value is 0.5, the authentication allowable range value is 0.05. Therefore, when the authentication threshold value is the upper limit value, the value of the authentication allowable range value is smaller than that when the authentication threshold value is not the upper limit value.

Next, the table of the behavior information database 172 for authentication will be described below with reference to FIG. 5B. In the table of the authentication behavior information database 172, the types of user behavior such as communication connection and event execution, the acquisition information acquired by the authentication information acquisition unit 181 shown in FIG. 4, the latest status in each behavior, and the latest status in each behavior are displayed. The pass conditions for each behavior are stored.

The acquired information includes, for example, the connection destination address, SSID (Service Set Identifier), BSSID (Basic Service Set Identifier) if the behavior type is a communication connection, and the schedule book in advance if the behavior type is event execution. If the location information such as the name and address of the place where the event is performed stored in is the distance between the face and the terminal device if the behavior type is the distance, and if the behavior type is the device connection, the connected device is selected. If the name, ID (Identifier), etc. to be indicated are location information, the latitude / longitude, address, etc. indicating the current location of the terminal device 1 are stored.

The latest status of each behavior is, for example, if the type of behavior is a communication connection, the total number of times the communication connection destination indicated in the acquired information has been connected so far. The initial value of the total number of connections to the communication connection destination is 0, and the number of times is added by the connection to the communication connection destination. If the behavior type is event execution, the distance between the location stored in the acquired information and the user's current location is stored.

If the type of behavior is the distance between the face and the terminal device 1, the latest situation stores the average distance between the face and the terminal device 1 calculated when the user is authenticated as the user himself / herself. NS. The average distance between the face and the terminal device 1 is updated each time the user is authenticated as the user himself / herself. The initial value of the average distance between the face and the terminal device 1 is the distance obtained when the biometric information shown in FIG. 5A is registered in advance before the user is authenticated by the terminal device 1.

If the type of behavior is device connection, the latest status stores whether or not the device is connected to the device indicated by the name, ID, or the like stored in the acquired information. The device connection is, for example, a connection between the device paired by Bluetooth® and the terminal device 1. If the type of behavior is location information, the latest status stores the latitude / longitude, address, etc. indicating the location where the user was when authenticated as the user himself / herself. Further, the pass condition of each behavior is a condition in which the condition for guaranteeing the reliability of each behavior is predetermined.

Next, the table of the tilt information table 173 is shown in FIG. 5C. The tilt information table 173 stores an angle indicating the tilt of the terminal device 1 acquired from the tilt detection unit 13 shown in FIG. 4, the acquisition date and time when the angle was acquired, and a waiting time which is an interval for detecting the tilt. doing. The angle indicating the inclination of the terminal device 1 is acquired from the inclination detection unit 13 by the authentication information acquisition unit 181 shown in FIG. 4 and updated every time the standby time elapses. Also, when updating the angle, the acquisition date and time when the angle was acquired is also updated.

When the terminal device 1 according to the present embodiment completes the execution of the initialization process of the post-power-on process or returns from the sleep state, the terminal device 1 enters a locked state in which the operation of each function is not permitted until the authentication is successful. When authentication is required when entering this locked state or operating each function, the terminal control unit 18 shown in FIG. 2 executes the authentication processing program 170 stored in the terminal storage unit 17. Determine if the user is the user himself / herself. The processing of the authentication processing program 170 executed by the terminal control unit 18 will be described below with reference to the flowcharts of each processing shown in FIGS. 6A to 10.

First, the flowchart of the authentication process shown in FIG. 6A is referred to. The terminal control unit 18 shown in FIG. 4 executes the face recognition value acquisition process (step S101). The face recognition value acquisition process will be described below with reference to the face recognition value acquisition process shown in FIG. 7.

The terminal control unit 18 sets the number of retries to 0 (step S201). The authentication information acquisition unit 181 shown in FIG. 4 causes the photographing unit 11 to take a face photograph of the user operating the terminal device 1. Specifically, the authentication information acquisition unit 181 causes the in-camera 11A to take a photograph of the face of the user facing the front of the terminal device 1. The authentication information acquisition unit 181 acquires a user's face photograph taken from the photographing unit 11 (step S202). The authentication information acquisition unit 181 determines whether or not the acquired user's face photo is blurred (step S203).

When the user's face photograph is not blurred (step S203; NO), the authentication information acquisition unit 181 determines whether or not the user's face can be detected from the user's face photograph taken by the photographing unit 11 (step S204). ). When the user's face can be detected from the user's face photograph (step S204; YES), the authentication information acquisition unit 181 obtains the feature amount of the detected user's face image. The authentication information acquisition unit 181 transmits the obtained feature amount of the user's face image to the authentication determination unit 182 shown in FIG.

The authentication determination unit 182 acquires the authentication biological information database 171 stored in the terminal storage unit 17 shown in FIG. From the table of the biometric information database 171 for authentication shown in FIG. 5A, the authentication determination unit 182 sets the feature amount of the face image stored in the registration information associated with the "face" among the types of biometric information, and the authentication value. Acquire the authentication tolerance and authentication threshold of. The authentication determination unit 182 compares the feature amount of the face image of the registered information acquired from the biometric information database 171 for authentication with the feature amount of the face image received from the authentication information acquisition unit 181 and based on the comparison result. Find the face recognition value.

Further, when the user's face photograph acquired in step S203 is blurred (step S203; YES), or when the user's face cannot be detected from the user's face photograph in step S204 (step S204; NO), authentication is performed. The user information acquisition unit 181 determines whether or not the number of retries is less than the specified number (step S206). This specified number of times is a predetermined number of times that the retry can be performed, and it is assumed that the specified number of times is stored in the terminal storage unit 17 shown in FIG.

When the number of retries is less than the specified number (step S206; YES), the authentication information acquisition unit 181 adds 1 to the number of retries (step S207). The authentication information acquisition unit 181 causes the photographing unit 11 to retry taking a photograph of the user's face (step S208).

If the number of retries is not less than the specified number (step S206; NO), the authentication information acquisition unit 181 causes the authentication determination unit 182 to acquire the face authentication value stored in the terminal storage unit 17 (step S209). .. Specifically, the authentication information acquisition unit 181 causes the authentication determination unit 182 to acquire the authentication biometric information database 171 stored in the terminal storage unit 17 shown in FIG. The authentication determination unit 182 acquires the authentication value associated with the "face" among the types of biometric information from the table of the biometric information database 171 for authentication shown in FIG. 5A. If the user's face cannot be detected from the user's face photo, lock the user who is currently operating to prevent further operations, display a message to the effect that another authentication method will be used, and so on. May be good.

The authentication determination unit 182 ends the face authentication acquisition process. Here, the process returns to FIG. 6A. The authentication determination unit 182 determines whether or not the face authentication value obtained in the face authentication value acquisition process is equal to or greater than the authentication threshold value acquired from the authentication biometric information database 171 (step S102). When the face authentication value obtained in the face authentication value acquisition process is equal to or greater than the authentication threshold (step S102; YES), the authentication determination unit 182 determines that the obtained face authentication value is the authentication allowable value acquired from the authentication biometric information database 171. It is determined whether or not it is as follows (step S103).

When the obtained face authentication value is equal to or less than the authentication allowable value (step S103; YES), the user using the terminal device 1 is gray whether or not he / she is the user himself / herself. Therefore, the authentication determination unit 182 executes the auxiliary authentication process, which is the process of authentication based on the behavior information (step S104). The auxiliary authentication process will be described below with reference to the flowchart shown in FIG.

First, the authentication determination unit 182 causes the authentication information acquisition unit 181 shown in FIG. 4 to acquire the communication connection destination currently connected from the communication unit 10. The authentication determination unit 182 receives the current communication connection destination of the acquired communication unit 10 from the authentication information acquisition unit 181.

Subsequently, the authentication determination unit 182 acquires the authentication behavior information database 172 from the terminal storage unit 17 shown in FIG. The authentication determination unit 182 acquires the acquisition information, the number of times, and the acceptance conditions associated with the "communication connection" among the types of behavior stored in the table of the authentication behavior information database 172 shown in FIG. 5B.

For example, as shown in FIG. 5B, the SSIDs ABC_WLAN and 123 WLAN are stored in the acquired information of the "communication connection". In this ABC_WLAN, it is stored that the number of connections is 31 times and the number of connections is 100 times or more as a pass condition. Further, in 123 WLAN, it is stored that the number of connections is 157 times and the number of connections is 100 times or more as a pass condition. In the following, the case where the passing condition is satisfied is referred to as trusting, and the case where the passing condition is not satisfied is referred to as unreliable.

The authentication determination unit 182 compares the current communication connection destination of the communication unit 10 received from the authentication information acquisition unit 181 with the acquisition information acquired from the authentication behavior information database 172, and the current communication connection destination trusts. It is determined whether or not it is a communication connection destination (step S301).

Here, for example, it is assumed that the SSID 123 WLAN has been acquired as the current communication connection destination of the communication unit 10. The 123 WLAN in the acquisition information of the behavior type "communication connection" stored in the authentication behavior information database 172 has been connected 156 times, and the pass condition has been connected 100 times or more. Therefore, since the current communication connection destination is a reliable communication connection destination (step S301; YES), the authentication determination unit 182 authenticates the user using the terminal device 1 as the user himself / herself.

Subsequently, the authentication determination unit 182 makes the authentication interval longer than the current authentication interval (step S302). This is because if the current communication connection destination is a reliable communication connection destination, the user himself / herself is considered to be in a reliable environment such as home or work. In this case, the authentication interval may be longer than the current authentication interval, the frequency of authentication may be reduced, and the minimum number of authentications required may be performed. Further, the authentication determination unit 182 sets the fact that the auxiliary authentication has been completed as a return value (step S303).

Further, in step S301, for example, it is assumed that the SSID ABC_WLAN has been acquired as the current communication connection destination of the communication unit 10. The ABC_WLAN in the acquisition information of the behavior type "communication connection" stored in the authentication behavior information database 172 has been connected 31 times, and the pass condition has been connected 100 times or more. Therefore, the authentication determination unit 182 determines that the current communication connection destination is not a reliable communication connection destination (step S301; NO). In this case, the authentication determination unit 182 determines whether or not the trusted event is being executed immediately before or at present (step S304).

Here, for example, it is assumed that the calendar provided in the terminal device 1 stores "△ ● movie theater" as the location of the event to be held at the current date and time. The authentication determination unit 182 compares the current position information acquired from the position detection unit 16 by the authentication information acquisition unit 181 with the position information of the "△ ● movie theater" which is the location of the event to be held at the current date and time. do. For example, the distance between the current location information and the location information of the event location “△ ● movie theater” is 72 m. In this case, the authentication determination unit 182 also determines that the trusted event has been executed (step S304; YES). After that, the authentication determination unit 182 executes step S302 and step S303.

Further, the authentication determination unit 182 causes the authentication information acquisition unit 181 to acquire the content of the event currently being executed from the operation input unit 14 immediately before or at the present time. The authentication determination unit 182 acquires from the calendar provided in the terminal device 1 whether or not there is a schedule at the immediately preceding or current date and time, and information on the place where the schedule is performed. If there is no schedule immediately before or at the current date and time, the authentication determination unit 182 determines that the trusted event is not executed (step S304; NO). In this case, the authentication determination unit 182 calculates the distance between the face and the terminal device 1 (step S305).

Further, in step S304, when there is a schedule immediately before or at the present, the authentication determination unit 182 causes the authentication information acquisition unit 181 to acquire the current position information from the position detection unit 16. Subsequently, the authentication determination unit 182 acquires the authentication behavior information database 172 from the terminal storage unit 17 shown in FIG.

The authentication determination unit 182 acquires the acquisition information and the pass condition corresponding to the "event execution" among the types of behavior stored in the table of the authentication behavior information database 172 shown in FIG. 5B. For example, as shown in FIG. 5B, "○ × park" and "△ ● movie theater" are stored as places where the event is held in the acquired information of "event execution", and "distance is 100 m" as a pass condition for both. It is assumed that "within" is remembered.

Here, for example, it is assumed that "○ × park" is stored in the calendar provided in the terminal device 1 as the place of the event to be held at the current date and time. The authentication determination unit 182 compares the current position information acquired from the position detection unit 16 by the authentication information acquisition unit 181 with the position information of "○ × park" which is the location of the event to be held at the current date and time. .. For example, the distance between the current location information and the location information of the event location "○ × Park" is 113 m. In this case, it is assumed that the event is not executed with confidence (step S304; NO), and the distance between the face and the terminal device 1 is calculated (step S305).

The distance between the user's face and the terminal device 1 is calculated based on the ratio of the user's face in the photograph of the user's face facing the front of the terminal device 1 taken by the in-camera 11A shown in FIG.

Subsequently, the authentication determination unit 182 acquires the authentication behavior information database 172 from the terminal storage unit 17 shown in FIG. The authentication determination unit 182 acquires the average distance and the pass condition corresponding to the "distance between the face and the terminal device" among the types of behavior stored in the table of the authentication behavior information database 172 shown in FIG. 5B. do. For example, as shown in FIG. 5B, the average distance of the "distance between the face and the terminal device" is 262 mm, and the passing condition is stored as plus or minus 20 mm of the average distance.

The authentication determination unit 182 determines whether or not the distance between the user's face and the terminal device 1 calculated in step S109 is within the set range set in the pass condition acquired from the authentication behavior information database 172 (step S306). .. Specifically, since the average distance acquired from the authentication behavior information database 172 is 262 mm and the pass condition is plus or minus 20 mm of the average distance, it is determined whether or not the range is in the range of 242 mm to 282 mm.

When the distance between the user's face and the terminal device 1 calculated in step S305 is within the set range, specifically, when it is within the range of 242 mm to 282 mm (step S306; YES), the authentication determination unit 182 , Step S302 and step S303 are executed.

Further, when the distance between the user's face and the terminal device 1 calculated in step S305 is not within the set range, specifically, when the distance is not within the range of 242 mm to 282 mm (step S306; NO), the authentication determination unit. 182 finds the current location of the terminal device 1 (step S307). Specifically, the authentication determination unit 182 causes the authentication information acquisition unit 181 shown in FIG. 4 to acquire the latitude and longitude indicating the current location of the terminal device 1 from the position detection unit 16.

Subsequently, the authentication determination unit 182 determines whether or not the latitude / longitude indicating the current location of the terminal device 1 acquired from the position detection unit 16 is within the set range set in the pass condition of the authentication behavior information database 172 (step). S308). Specifically, the authentication determination unit 182 acquires the authentication behavior information database 172 from the terminal storage unit 17 shown in FIG. The authentication determination unit 182 includes the latitude and longitude stored in the latest status corresponding to the "position information" among the types of behavior stored in the table of the authentication behavior information database 172 shown in FIG. 5B, and the acceptance conditions. And get. The authentication determination unit 182 obtains the distance based on the difference between the latitude and longitude indicating the current location of the terminal device 1 acquired by the authentication information acquisition unit 181 and the latitude and longitude acquired from the authentication behavior information database 172. The authentication determination unit 182 determines whether or not the obtained distance is within the set range set in the pass condition of the authentication behavior information database 172.

When the latitude / longitude indicating the current location of the terminal device 1 is within the setting range set in the pass condition of the authentication behavior information database 172 (step S308; YES), the authentication determination unit 182 executes steps S302 and S303. do. Further, when the latitude / longitude indicating the current location of the terminal device 1 is not within the setting range set in the pass condition of the authentication behavior information database 172 (step S308; NO), the authentication determination unit 182 cannot perform auxiliary authentication. This is set as the return value (step S309). The authentication determination unit 182 ends the auxiliary authentication process.

Here, the process returns to FIG. 6A. The authentication determination unit 182 determines whether or not auxiliary authentication has been performed (step S105). When the auxiliary authentication cannot be performed (step S105; NO), the authentication determination unit 182 causes the authentication result display unit 183 shown in FIG. 4 to display on the display unit 19 that the voice input of the keyword is requested (step S106). ). This keyword is an arbitrary keyword stored in advance in the terminal storage unit 17 shown in FIG.

The authentication determination unit 182 determines whether or not the voice-input keyword is correct (step S107). When the voice-input keyword is incorrect (step S107; NO), the authentication determination unit 182 causes the authentication result display unit 183 to display on the display unit 19 that the password input is requested (step S108). This password is an arbitrary password stored in advance in the terminal storage unit 17 shown in FIG.

The authentication determination unit 182 determines whether or not the entered password is correct (step S109). If the entered password is incorrect (step S109; NO), the authentication determination unit 182 then causes the authentication determination unit 182 to display the login screen on the display unit 19 on the authentication result display unit 183 shown in FIG. (Step S110). This login screen is personal authentication by the OS included in the terminal device 1 such as password input and fingerprint authentication.

Further, when the face authentication value obtained in the face authentication value acquisition process in step S102 is not equal to or greater than the authentication threshold value (step S102; NO), and when auxiliary authentication can be performed in step S105 (step S105; YES), step S107 If the keyword input by voice is correct (step S107; YES), or if the password input in step S109 is correct (step S109; YES), the authentication determination unit 182 executes the voice acquisition process (step S111).

In other words, when the face authentication value obtained in the face authentication value acquisition process in step S102 is not equal to or greater than the authentication threshold (step S102; NO), and when auxiliary authentication can be performed in step S105 (step S105; YES), in other words. , When the user's face authentication is successful. In this case, the terminal device 1 is in a state in which the user's face image can be acquired, such as a state in which the user is grasped and a state in which the user is operating various functions.

If the keyword input by voice in step S107 is correct (step S107; YES), or if the password input in step S109 is correct (step S109; YES), the keyword or password displayed on the display unit 19 is displayed. The user can see the request for input. In this case, as in the states in steps S102 and S105, the terminal device 1 is in a state in which the user's face image can be acquired, such as a state in which the user is grasped and a state in which the user is operating various functions.

The usage state of the terminal device 1 capable of acquiring the user's face image, such as a state of being held by the user and a state of being operated by the user, is an example of the first usage state within the scope of the claims.

Subsequently, the voice acquisition process will be described below with reference to the flowchart shown in FIG. The authentication determination unit 182 determines whether or not there is a user's utterance (step S401). Specifically, the authentication determination unit 182 determines that the user has uttered a voice when the authentication information acquisition unit 181 can acquire the user's voice from the voice input / output unit 12 shown in FIG. Further, when the authentication information acquisition unit 181 cannot acquire the user's voice from the voice input / output unit 12 shown in FIG. 4, the authentication determination unit 182 determines that the user has not spoken. The audio input / output unit 12 is the microphone 12B shown in FIG.

If there is no voice from the user (step S401; NO), the authentication determination unit 182 ends the voice acquisition process. If there is a user's utterance (step S401; YES), the authentication determination unit 182 sets the recording time to 0 seconds (step S402). The authentication determination unit 182 causes the authentication information acquisition unit 181 to acquire the user's voice from the voice input / output unit 12 shown in FIG. 4 (step S403).

The authentication determination unit 182 determines whether or not the recording time exceeds the specified time (step S404). This specified time is the time required to acquire predetermined voice data. For example, an arbitrary time such as 10 seconds or 30 seconds is set. The set specified time is stored in the terminal storage unit 17.

When the recording time does not exceed the specified time (step S404; NO), the authentication determination unit 182 returns to step S403 and causes the authentication information acquisition unit 181 to acquire the user's voice from the voice input / output unit 12. When the recording time exceeds the specified time (step S404; YES), the authentication determination unit 182 analyzes the acquired voice data (step S405). Specifically, the authentication determination unit 182 acquires voice data of each feature amount of voice print, habit, and intonation included in the type of biometric information of the biometric information database 171 for authentication shown in FIG. 5A. Is analyzed and obtained. The authentication determination unit 182 ends the voice acquisition process.

Here, the process returns to FIG. 6A. The authentication determination unit 182 causes the authentication information update unit 184 shown in FIG. 4 to update various data stored in the authentication biometric information database 171 and the authentication behavior information database 172 shown in FIG. 2 (step S112). .. Specifically, the authentication information update unit 184 is stored in the registration information in the registration information associated with the biometric information type "face" in the table of the authentication biometric information database 171 shown in FIG. 5A. The registration information is updated by adding the feature amount of the face image received from the authentication information acquisition unit 181 by the authentication determination unit 182 to the feature amount of the face image. In addition, the authentication information update unit 184 sets the types of biometric information "voice (voice pattern)", "voice (voice)", and "voice (intonation)" in the table of the biometric information database 171 for authentication shown in FIG. 5A. The registration information is updated by adding the voice print, habit, and intonation feature quantities of the voice obtained in the voice acquisition process shown in FIG. 9 to the associated registration information.

Subsequently, the authentication information update unit 184 adds 1 to the number of times stored in the latest status associated with the behavior type "communication connection" of the table of the authentication behavior information database 172 shown in FIG. 5B. Update. Further, the latest status corresponding to the behavior type "distance between the face and the terminal device" stored in the table of the authentication behavior information database 172 shown in FIG. 5B is displayed in the stored average distance and step S109. Update with the average distance obtained from the "distance between the face and the terminal device" calculated in.

By updating the biometric information stored in the authentication biometric information database 171 and the behavioral information stored in the authentication behavior information database 172 in this way, the accuracy of the user's biometric information and behavioral information is improved. Therefore, the accuracy of user authentication can be improved.

Now move to FIG. 6B. The authentication determination unit 182 determines whether or not the biometric authentication and the auxiliary authentication have succeeded in a predetermined set number of times (step S113). The set number of times is an arbitrary number of times, for example, 10 times in a row, 20 times in total after the terminal device 1 is started, and the like. When biometric authentication and auxiliary authentication succeed in a predetermined set number of times (step S113; YES), the authentication determination unit 182 obtains an average value of face authentication values obtained by authentication for the set number of times (step S114). ..

Specifically, the authentication determination unit 182 acquires the authentication biological information database 171 shown in FIG. 2 from the terminal storage unit 17. The authentication determination unit 182 acquires the average value of the authentication values associated with the "face" among the types of biometric information from the table of the biometric information database 171 for authentication shown in FIG. 6A. The authentication determination unit 182 adds the face authentication value obtained in step S101 and the average value of the authentication values acquired from the authentication biometric information database 171 and divides by 2, to calculate the average value of the face authentication values. If biometric authentication and auxiliary authentication are not successful for a predetermined set number of times (step S113; NO), the process of step S121 to step S123 is skipped and the process proceeds to step S124.

The authentication determination unit 182 transmits the average value of the face authentication values obtained in step S114 to the authentication information update unit 184. The authentication information update unit 184 compares the average value of the received face authentication values with the preset upper limit value of the authentication threshold value. When the average value of the face authentication values is equal to or more than the preset upper limit value of the authentication threshold value, the authentication information update unit 184 sets the type of biometric information in the table of the authentication biometric information database 171 shown in FIG. 6A. Of these, the upper limit of the authentication threshold is written and updated in the authentication threshold associated with the "face".

When the average value of the face authentication values is equal to or less than the preset upper limit value of the authentication threshold value, the authentication information update unit 184 sets the authentication biometric information in the table of the authentication biometric information database 171 shown in FIG. 6A. The average value of the face authentication values obtained in step S114 is written and updated in the authentication threshold value associated with the "face" among the types (step S115).

Subsequently, the authentication information update unit 184 updates the authentication allowable value (step S116). Specifically, when the average value of the face authentication values obtained in step S114 is equal to or greater than the preset upper limit value of the authentication threshold value, the authentication information update unit 184 sets the preset maximum authentication allowable value. Set to the authentication tolerance. Further, when the average value of the face authentication values obtained in step S114 is equal to or less than the upper limit of the preset authentication threshold, the average value of the face authentication values obtained in step S114 and the default authentication allowable range value are used. If the value obtained by adding is equal to or less than the maximum authentication allowable value, the added value is regarded as the authentication allowable value. If the sum of the average value of the face authentication values obtained in step S114 and the default authentication allowable range value is equal to or greater than the maximum authentication allowable value, the maximum authentication allowable value is set as the authentication allowable value.

The authentication information update unit 184 acquires the authentication biological information database 171 shown in FIG. 2 from the terminal storage unit 17. The authentication information update unit 184 writes the obtained authentication tolerance value in the authentication tolerance value associated with the "face" of the types of biometric information in the table of the authentication biometric information database 171 shown in FIG. 6A. Update.

The authentication information acquisition unit 181 shown in FIG. 4 acquires the inclination angle of the terminal device 1 from the inclination detection unit 13. Subsequently, the authentication information acquisition unit 181 acquires the current date and time information from a timer (not shown) (step S117). The authentication information acquisition unit 181 transmits the acquired tilt angle of the terminal device 1 and the current date and time information to the authentication determination unit 182.

The authentication determination unit 182 transmits the received tilt angle of the terminal device 1 and the current date and time information to the authentication information update unit 184. The authentication information update unit 184 writes and saves the received inclination angle of the terminal device 1 and the current date and time information in the inclination information table 173 stored in the terminal storage unit 17 shown in FIG. 2 (step S118). ).

The authentication determination unit 182 acquires the waiting time stored in the table of the inclination information table 173 shown in FIG. 5C. The authentication determination unit 182 transmits the acquired waiting time to the authentication information acquisition unit 181. The authentication information acquisition unit 181 waits for data acquisition from the communication unit 10, the photographing unit 11, and the like during the received standby time (step S119).

When the standby time ends, the authentication information acquisition unit 181 acquires the inclination angle of the terminal device 1 from the inclination detection unit 13. Subsequently, the authentication information acquisition unit 181 acquires the current date and time information from a timer (not shown) (step S120). The authentication information acquisition unit 181 transmits the acquired tilt angle of the terminal device 1 and the current date and time information to the authentication determination unit 182.

The authentication determination unit 182 acquires the angle of the terminal device 1 stored in the table of the tilt information table 173 shown in FIG. 5C. The authentication determination unit 182 compares the tilt angle of the terminal device 1 received from the authentication information acquisition unit 181 with the angle of the terminal device 1 acquired from the tilt information table 173, and determines whether or not the angle has changed. Is determined (step S121).

When the change in the angle of the terminal device 1 is an angle of a predetermined set value, for example, 30 degrees or more (step S121; NO), the authentication determination unit 182 moves the terminal device 1 by the user to perform some operation. Is determined to have been performed, and the process returns to step S101 shown in FIG. 6A. After that, the authentication determination unit 182 executes the processes after step S101.

Further, when the change in the angle of the terminal device 1 is equal to or less than the angle of the preset value (step S121; YES), the authentication determination unit 182 determines that the terminal device 1 has not been moved by the user. Subsequently, the authentication determination unit 182 determines whether or not it is time to authenticate the user (step S122). The timing for authenticating the user is the timing at which the preset authentication interval time has elapsed.

When it is time to authenticate the user (step S122; YES), the authentication determination unit 182 returns to step S101 shown in FIG. 6A. After that, the authentication determination unit 182 executes the processes after step S101. If it is not time to authenticate the user (step S122; NO), the authentication determination unit 182 returns to step S118. After that, the authentication determination unit 182 executes steps S118 to S122.

Here, the process returns to FIG. 6A. In step S103, when the face authentication value obtained by the authentication determination unit 182 is not equal to or less than the authentication allowable value acquired from the authentication biometric information database 171 (step S103; NO), the authentication determination unit 182 performs voice authentication processing. Execute (step S123).

When the authentication value of the face is not equal to or less than the authentication allowable value acquired from the biometric information database 171 for authentication, in other words, the authentication by the user's face fails. This indicates that the usage state of the terminal device 1 is a usage state in which authentication by the user's face cannot be performed. For example, when the terminal device 1 is used while being stored in a bag, a pocket, or the like, authentication by the user's face fails because the user's face image is not acquired. Further, for example, when the terminal device 1 is used hands-free from a distant position, the face image is unclear even if the user's face image can be acquired, and the user's face image in the image is too small and the face authentication value. Authentication by the user's face fails for reasons such as not being asked for. In this case, by switching from the user's face authentication to the user's voice authentication, the user can be stably authenticated as the person himself / herself. The usage state of the terminal device 1 that authenticates by the user's voice is an example of the second usage state within the scope of claims.

The voice authentication process will be described below with reference to the flowchart shown in FIG. The authentication determination unit 182 determines whether or not there is a user's utterance (step S501). Specifically, the authentication determination unit 182 determines that the user has uttered a voice when the authentication information acquisition unit 181 can acquire the user's voice from the voice input / output unit 12 shown in FIG. Further, when the authentication information acquisition unit 181 cannot acquire the user's voice from the voice input / output unit 12 shown in FIG. 4, the authentication determination unit 182 determines that the user has not spoken. The audio input / output unit 12 is the microphone 12B shown in FIG.

When there is no voice from the user (step S501; NO), the authentication determination unit 182 sets the return value to the effect that voice authentication could not be performed (step S510). When there is a user's utterance (step S501; YES), the authentication determination unit 182 causes the authentication information acquisition unit 181 shown in FIG. 4 to acquire the user's voice from the voice input / output unit 12 (step S502). .. The authentication determination unit 182 analyzes the acquired voice data (step S503). Specifically, the authentication determination unit 182 acquires voice data of each feature amount of voice print, habit, and intonation included in the type of biometric information of the biometric information database 171 for authentication shown in FIG. 5A. Is analyzed and obtained.

The authentication determination unit 182 obtains the voice authentication value (step S504). The voice authentication value is obtained by averaging the voiceprint, habit, and intonation authentication values of the voice. Specifically, first, the authentication determination unit 182 acquires the authentication biological information database 171 stored in the terminal storage unit 17 shown in FIG. The authentication determination unit 182 corresponds to "voice (voice pattern)", "voice (mouth habit)", and "voice (intonation)" among the types of biometric information from the table of the biometric information database 171 for authentication shown in FIG. 5A. Acquires each feature amount stored in the attached registration information, the authentication permissible value of the authentication value, and the authentication threshold.

Subsequently, the authentication determination unit 182 includes each feature amount of the registered information "voice (voiceprint)", "voice (mouth habit)", and "voice (intonation)" of the registration information acquired from the biological information database 171 for authentication, and step S503. Compare the voiceprint, habit, and intonation feature of the voice obtained in. The authentication determination unit 182 obtains each authentication value of "voice (voiceprint)", "voice (mouth habit)", and "voice (intonation)" based on the comparison result. The authentication determination unit 182 obtains the average value of each of the obtained authentication values. This average value becomes the voice authentication value.

The authentication determination unit 182 determines whether or not the voice authentication value obtained in step S504 is equal to or greater than the authentication threshold value (step S505). When the voice authentication value is equal to or greater than the authentication threshold value (step S505; YES), the authentication determination unit 182 determines whether or not the voice authentication value is equal to or less than the authentication allowable value (step S506). When the voice authentication value is not equal to or higher than the authentication threshold value (step S505; NO), the authentication determination unit 182 sets the return value to the effect that voice authentication has been performed (step S509).

When the voice authentication value is equal to or less than the authentication allowable value (step S506; YES), the authentication determination unit 182 executes the auxiliary authentication process (step S507). For the auxiliary authentication process, the process of the flowchart shown in FIG. 8 is executed. If the voice authentication value is not equal to or less than the authentication permissible value (step S506; NO), the authentication determination unit 182 sets the return value to the effect that voice authentication could not be performed (step S510).

The authentication determination unit 182 determines whether or not auxiliary authentication has been performed (step S508). When the auxiliary authentication is possible (step S508; YES), the authentication determination unit 182 sets the return value to the effect that voice authentication is possible (step S509). If the auxiliary authentication cannot be performed (step S508; NO), the authentication determination unit 182 sets the return value to the effect that voice authentication could not be performed (step S510). The authentication determination unit 182 ends the voice authentication process.

Here, the process returns to FIG. 6A. The authentication determination unit 182 determines whether or not voice authentication has been performed (step S124). When voice authentication is possible (step S124; YES), the authentication determination unit 182 causes the authentication information acquisition unit 181 shown in FIG. 4 to acquire the user's voice from the voice input / output unit 12 (step S125).

The authentication determination unit 182 causes the authentication information update unit 184 shown in FIG. 4 to update the feature amount of the voice stored in the authentication biological information database 171 shown in FIG. 2 (step S126). Specifically, first, the authentication determination unit 182 analyzes the voice data acquired in step S125, and obtains each feature amount of the voice print, habit, and intonation of the voice. The authentication determination unit 182 tells the authentication information update unit 184 that each feature amount is obtained from the table of the authentication biometric information database 171 for the types of biometric information "voice (voice pattern)", "voice (voice)", and "voice". (Intonation) ”is added to the corresponding registration information, and the registration information is updated. The authentication determination unit 182 returns to step S101 and executes the processes after step S101.

Further, in step S124, if voice authentication cannot be performed (step S124; NO), the authentication determination unit 182 determines that the user using the terminal device 1 is not the user himself / herself. The authentication determination unit 182 causes the authentication result display unit 183 shown in FIG. 4 to display to the display unit 19 that the authentication has not been performed. Subsequently, the authentication determination unit 182 calls the existing biometric authentication means provided in the terminal device 1. Here, it is assumed that fingerprint authentication is called as an existing biometric authentication means. The authentication determination unit 182 executes fingerprint authentication (step S127). The various biometric information used in the existing biometric authentication means provided in the terminal device 1 is an example of the four authentication information within the scope of the claims.

When fingerprint authentication is possible (step S128; YES), the authentication information acquisition unit 181 causes the photographing unit 11 to take a face photograph of the user operating the terminal device 1 according to the instruction from the authentication determination unit 182. The authentication information acquisition unit 181 acquires an image of the user's face photograph taken from the photographing unit 11 and obtains a feature amount of the user's face image. The authentication information acquisition unit 181 transmits the feature amount of the user's face image obtained to the authentication determination unit 182. The authentication determination unit 182 transmits the received feature amount of the user's face image to the authentication information update unit 184 shown in FIG. The authentication information update unit 184 stores the feature amount of the received user's face image in the registration information associated with the biometric information type "face" in the table of the authentication biometric information database 171 shown in FIG. 5A. In addition to the feature amount of the face image that has been performed, the feature amount is updated (step S129). The authentication determination unit 182 returns to step S101 and executes the steps after step S101.

If fingerprint authentication cannot be performed (step S128; NO), the authentication determination unit 182 causes the authentication result display unit 183 shown in FIG. 4 to display the login screen on the display unit 19 (step S110). This login screen is personal authentication by the OS included in the terminal device 1 such as password input and fingerprint authentication.

In the above embodiment, when the authentication value obtained from the biometric information and the authentication threshold value are the same, the authentication value obtained from the biometric information is equal to or less than the authentication threshold value, or is obtained from the biometric information. In either case, where the authenticated authentication value is equal to or greater than the authentication threshold value, it may be determined whether or not the authentication is successful. If the authentication value obtained from the biometric information and the authentication permissible value are the same, the authentication value obtained from the biometric information is less than or equal to the authentication permissible value, or the authentication value obtained from the biometric information is In either case, which is equal to or greater than the authentication allowable value, it may be determined whether or not the authentication is successful.

As described above, the terminal device 1 according to the above-described embodiment is based on biological information such as an image, fingerprint, and voice of the user's face, and behavior based on a peculiar behavior when the user operates the terminal device 1, an operating state, and the like. Based on the information, the user can be authenticated and various functions in the terminal device 1 can be executed. In particular, in the terminal device 1, the user cannot be authenticated by the image of the user's face due to the usage status of the terminal device 1, for example, when the terminal device is used while being put in a bag, a pocket, or the like. It is possible to authenticate the user himself / herself by his / her voice. As a result, the user can be stably authenticated regardless of the usage state of the terminal device 1.

Further, the user authentication process performed in the terminal device 1 is executed in the background while the terminal device 1 is in operation, and the accuracy of authentication is improved by updating the user's biometric information and behavior information. Can be made to. As a result, security can be ensured without imposing a burden on the user.

(Modification example 1)
In the above embodiment, as the authentication by the user's biometric information, the authentication by the user's face image, the voice authentication, and the fingerprint authentication are used. Not limited to this, the authentication based on the user's biometric information may be performed by any method such as finger vein authentication and iris authentication. Further, in the above embodiment, the authentication by the user's face image and the voice authentication are used as the conditions for the determination to use the auxiliary authentication, but the auxiliary authentication should be used according to the determination result by other biometric information. You may do it.

(Modification 2)
In the above embodiment, the user authentication process is realized by executing the authentication process program 170 shown in FIG. All or part of each step performed by the authentication processing program 170 is realized by a semiconductor chip such as an ASIC (Application Specific Integrated Circuit) or a system LSI (Lage-scale Integration), a circuit composed of various circuit elements, or the like. You may do so.

(Modification example 3)
In the above embodiment, the connection to the trusted connection destination, the execution of the trusted event, and the distance between the user's face and the face of the terminal device 1 are used as the determination conditions for the auxiliary authentication. Not limited to this, other methods may be used or included. For example, it is determined whether or not the device owned by the user and the terminal device 1 are connected by Bluetooth (registered trademark), and if they are connected, the user is authenticated. In order to use the devices connected by Bluetooth®, it is necessary to "pair" the devices. Therefore, the connection of the device by Bluetooth (registered trademark) has strong individual specificity, and it is possible to authenticate the user himself / herself by using it as auxiliary authentication. Further, when it is determined whether or not the user is the user himself / herself based on the pattern, regularity, etc. of the user's action route acquired by the position detection unit 16 shown in FIG. 2, when the pattern, regularity, etc. of the action route match. It may be authenticated with the user himself / herself.

(Modification example 4)
In the above embodiment, when one of the auxiliary authentications is successful, the user is authenticated. Not limited to this, the user may be authenticated only when a plurality of auxiliary authentications are all successful. As a result, the accuracy of authentication can be further improved.

(Modification 5)
In the above embodiment, when the biometric authentication and the auxiliary authentication are successful, the authentication determination unit 182 makes the authentication interval longer than the current authentication interval in step S302 of the flowchart shown in FIG. 8 and lowers the authentication frequency. ing. However, not limited to this, when biometric authentication and auxiliary authentication are successful, the authentication interval does not need to be longer than the current authentication interval, and the frequency of authentication does not have to be reduced. Specifically, step S302 of the flowchart shown in FIG. 8 does not have to be performed.

(Modification 6)
In the above embodiment, when the user is successfully authenticated, the user continuously operates the terminal device 1, and the inclination of the terminal device 1 is not changed, the user himself / herself continues to operate the terminal device 1. It can be judged that it is. In this case, the authentication threshold value and the authentication permissible value of biometric authentication may be set loosely, the authentication interval may be lengthened, and the like. By doing so, it is possible to save the use of the resources of the terminal device 1 while performing the minimum necessary authentication for the user himself / herself in the background.

(Modification 7)
In the above embodiment, background authentication is performed at a predetermined authentication interval. Not limited to this, authentication may be performed randomly in the background without defining the timing and interval of authentication. For example, if any of the various sensors mounted on the terminal device 1 causes a spatial change such as a change in the position of the terminal device 1 or a change in the inclination, the authentication may be performed at any time. In addition, authentication may be performed when the user performs an operation for performing a special process on the terminal device 1, an irregular operation, or the like.

(Modification 8)
In the above embodiment, in step S112 of the authentication process shown in FIG. 6A, the authentication determination unit 182 is connected to the authentication information update unit 184 shown in FIG. The registration information of various data stored in the authentication behavior information database 172 is updated. Not limited to this, the authentication determination unit 182 sends the authentication information update unit 184 to the authentication living body when, for example, the user can be authenticated a predetermined number of times or when a predetermined time has elapsed. The registration information of various data stored in the information database 171 and the authentication behavior information database 172 may be updated.

Further, in the above embodiment, in step S112 of the authentication process shown in FIG. 6A, the authentication determination unit 182 stores the authentication information update unit 184 in the authentication biometric information database 171 and the authentication behavior information database 172. Some of the registered information of the various data provided may be updated.

(Modification 9)
In the above embodiment, in the authentication process shown in FIG. 6A, when the user's face image cannot be used to authenticate the user's own person, the user's own person is authenticated by voice. Not limited to this, for example, in the background, the voice of the user may be constantly acquired and the user may be authenticated by the voice. Further, in this case, when the user's own authentication is required and the voice authentication is successful in the background immediately before, the authentication may be regarded as successful.

In addition, when the background voice authentication has not been performed immediately before, or the voice authentication has failed, for example, other biometric authentication such as the user's face image authentication or fingerprint authentication may be performed. You may. Further, when other biometric authentication such as the user's face image authentication or fingerprint authentication fails, the user may be explicitly uttered again to perform voice authentication.

Further, in the embodiment of the present invention, the terminal device 1 can be realized by using a normal computer system without relying on a dedicated system. For example, a program for realizing each function in the terminal device 1 is stored in a recording medium such as a computer-readable CD-ROM (Compact Disc Read Only Memory) or a DVD-ROM (Digital Versailles Disc Read Only Memory). By distributing this program and installing this program on a computer, a computer capable of realizing each of the above-mentioned functions may be configured. When each function is realized by sharing the OS (Operating System) and the application, or by cooperating with the OS and the application, only the application may be stored in the recording medium.

The technical scope of the present invention is not limited to the above embodiments and modifications. The present invention can be freely applied, modified or improved as long as the technical idea described in the claims is concerned.

1 Terminal device, 10 Communication unit, 11 Imaging unit, 11A in-camera, 11B main camera, 12 audio input / output unit, 12A speaker, 12B microphone, 13 detection unit, 14 operation input unit, 15 fingerprint detection unit, 15A left fingerprint sensor , 15B right fingerprint sensor, 16 position detector, 17 terminal storage unit, 18 terminal control unit, 19 display unit, 21 processor, 22 memory, 23 display controller, 24 display device, 25 I / O port, 26 storage device, 27 Communication equipment, 28 data bus, 170 authentication processing program, 171 authentication biometric information database, 172 authentication behavior information database, 173 information table, 181 authentication information acquisition unit, 182 authentication judgment unit, 183 authentication result display unit, 184 authentication Information update department.

Claims (7)

An authentication information acquisition unit that acquires multiple authentication information,
Based on at least one of authentication information or credentials among the plurality of authentication information acquired in the authentication information acquiring unit, and determines the authentication assessment section whether the authentication is successful,
If it is determined that the authentication by the authentication determination unit has succeeded, by the plurality of authentication information acquired by the authentication information acquiring section, and a authentication information update section for updating the plurality of authentication information stored ,
In the background, the authentication determination unit performs authentication in which a first authentication value obtained by comparing one authentication information acquired by the authentication information acquisition unit with one registration information registered in itself is predetermined. If it is less than the threshold value, it is determined that the authentication in the first usage state of the user is successful, and the first authentication value is a value including the authentication allowable range value indicating the case where the user is the person or not in the authentication threshold value. whether authentication in the second use state of the own based on some the second authentication information acquired by the authentication information acquiring section as the authentication allowable value or more of authentication in the first use state when the failure has succeeded If the first authentication value is equal to or more than the authentication threshold value and less than the authentication allowable value, it is determined whether or not the authentication is successful based on the three authentication information acquired by the authentication information acquisition unit. If the determination is made and the authentication is successful based on the three authentication information, it is assumed that the user is authenticated based on the one authentication information .
Terminal equipment. The authentication judging portion further prior Symbol second authentication information and the own second authentication values calculated by comparing the second registration information registered is the authentication threshold value or more and less than the authentication tolerance In this case, it is determined whether or not the authentication is successful based on the three authentication information acquired by the authentication information acquisition unit, and if the authentication is successful based on the three authentication information, the second authentication information is used. Based on this, it is assumed that the user has been authenticated as the person himself / herself .
The terminal device according to claim 1. Further, when the second authentication value is equal to or more than the permissible value for authentication , the authentication determination unit determines that the authentication based on the second authentication information has failed, and uses the four authentication information acquired by the authentication information acquisition unit. Determine if the authentication was successful based on
The terminal device according to claim 2. The one authentication information is the first biometric information of the user acquired by the authentication information acquisition unit.
The second authentication information is the second biometric information of the user acquired by the authentication information acquisition unit.
The three authentication information is behavior information including a user's peculiar behavior.
The terminal device according to claim 2 or 3. The first usage state is a usage state in which the user holds the self.
The second usage state is a usage state in which the self is stored in a bag or a pocket.
The terminal device according to any one of claims 1 to 4. An information processing method executed in a terminal device.
Get multiple credentials,
Based on at least one authentication information or a plurality of authentication information among the acquired plurality of authentication information, it is determined whether or not the authentication is successful.
When it is determined that the authentication is successful, the plurality of stored authentication information is updated by the acquired plurality of authentication information.
In the background, if the first authentication value obtained by comparing one authentication information with one registration information registered in itself is less than the predetermined authentication threshold, the authentication in the first usage state of oneself is successful. If the first authentication value is equal to or greater than the authentication allowance value, which is a value including the authentication allowance range value indicating whether or not the user is the person himself / herself in the authentication threshold, the authentication in the first use state is performed. Determines whether or not the authentication in the second usage state of the user is successful based on the second authentication information, and the first authentication value is equal to or more than the authentication threshold and less than the authentication allowable value. In that case, it is determined whether or not the authentication is successful based on the three authentication information, and if the authentication is successful based on the three authentication information, the user is authenticated as the person based on the one authentication information. Assume ,
Information processing method. On the computer
The process of acquiring multiple credentials,
A process of determining whether or not authentication is successful based on at least one authentication information or a plurality of authentication information among the acquired plurality of authentication information.
A process of updating a plurality of stored authentication information by the acquired plurality of authentication information when it is determined that the authentication is successful.
In the background, if the first authentication value obtained by comparing one authentication information with one registration information registered in itself is less than the predetermined authentication threshold, the authentication in the first usage state of oneself is successful. If the first authentication value is equal to or greater than the authentication allowance value, which is a value including the authentication allowance range value indicating whether or not the user is the person himself / herself in the authentication threshold, the authentication in the first use state is performed. Determines whether or not the authentication in the second usage state of the user is successful based on the second authentication information, and the first authentication value is equal to or more than the authentication threshold and less than the authentication allowable value. In that case, it is determined whether or not the authentication is successful based on the three authentication information, and if the authentication is successful based on the three authentication information, the user is authenticated as the person based on the one authentication information. processing of those was,
A program to execute.

Images