JP2009259269A - Face image recording system and face image recording method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make it possible for a monitor to determine spoofing and report it real-time in each input terminal. <P>SOLUTION: This invention takes an image of a user's face verified through ID verification using ID information in the input terminal 2 by a user and checks whether the user concerned is an identical person via a face registering server 3 through face verification using the face image. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a system for managing access to an important facility (room, floor, building, etc.), fraud prevention using a camera in accessing a computer or a terminal, and grasping an accessor's action.

Conventionally, there are mainly two systems for performing management using a camera.
One is a monitoring system installed in banks and convenience stores. In this type of surveillance system, a person photographed by a camera (surveillance camera) is monitored, or an image photographed by the camera is recorded. The video from the surveillance camera is monitored at another location and checked by a supervisor. The central technique here is a technique for capturing an image of a person or face in an optimal state, a technique for transmitting the captured image at high speed, or a technique for recording the captured image with high compression.

  The other is an authentication system that performs authentication processing for a person. In this type of authentication system, personal authentication (identification) is performed using biological information such as a face image of the person to be authenticated. For example, a face image pattern of a user who has an access right is registered in advance, and the user's face image is input from the camera at the time of access, and is compared with the registered image pattern in real time to determine whether the user is the person. The method using a face image pattern belongs to the (biometrics) personal authentication method based on biometric information typified by fingerprints, but can be input in a non-contact manner from a distant position and extracted for collation with the face image. There is an advantage that the face pattern feature can be left as a history.

  However, in a conventional surveillance system using a surveillance camera, the surveillance camera is installed at a relatively high position separated by a certain distance in order to prevent the user from being aware that his / her face is captured as much as possible. There are many. In such a case, there is a problem that a face image of a person is not always properly captured such that a face image facing the front that can be identified by an individual cannot always be obtained. In other words, there are few problems from the viewpoint of user burden (usability) at the time of input, but it may not necessarily be powerful in terms of fraud deterrence.

  On the other hand, in the conventional personal authentication (identity confirmation) system, the user is consciously input the front face, and access control is performed in real time by collating with face image (or face pattern) data registered in advance. Do. Such a personal authentication method by inputting a face image is easier for the user to compare with other biometric verification methods such as fingerprints, and is superior in terms of the burden on the user when inputting a face image. However, compared with systems that monitor people and systems that record human images, the security level is considerably higher, but the effort to register face image patterns in advance and lighting fluctuations such as outside light affect recognition performance. There is a problem that it is not easy to use because it gives

  As described above, the conventional system has advantages and disadvantages in usability and security level, and there is no intermediate device or system in the balance between the two, and there is a problem that the application range is narrow.

Yamaguchi, Fukui, Maeda, "Face recognition system using moving images", IEICE PRMU97-50, June 1997 Fukui, Yamaguchi, “Face feature point extraction by combination of shape extraction and pattern matching”, IEICE D-II, Vol.J-80-D-II, No.8, August 1997 Toshio Sato et al., "Development of FacePass Security System FacePass Considering Standing Position Fluctuation", Journal of the Institute of Image Information and Television Engineers, Special Issue on Information Media Processing for Security, Vol.56, No.7, pp. 1111–1117, July 2002 Nagao Makoto, "Image Recognition", Corona, 1983

  The present invention has been made in view of the above circumstances, and its purpose is to take advantage of the non-contact property by face matching and to minimize the usability and convenience of the user as much as possible. Another object of the present invention is to provide a system capable of enhancing the security level and fraud deterrent effect in physical security.

  The face image recording system of the present invention has a plurality of input terminals installed at a plurality of points and a server device capable of receiving data from these input terminals. An image input means for inputting a user's face image; and a transmission means for transmitting face image data input by the image input means and history information including the shooting location and shooting time to the server device; When the server device receives the face image data transmitted from the input terminal and the image recording means for recording the face image data, and the face image data from the input terminal, the face image data received from the input terminal And face image data already recorded in the image recording means, face image data received from the input terminal by the matching means, and the face image data When the collation with the face image data already recorded in the image recording means is successful, the history information corresponding to the face image data collated with the face image data received from the input terminal is read from the image recording means, and the history Display means for displaying a trajectory of the user's action based on the shooting location and the shooting time included in the information.

  According to the present invention, it is possible to enhance the security level and fraud deterrent effect in information security and physical security while making the most of the convenience and convenience of the user as much as possible by taking advantage of non-contact by face matching. A face image recording system can be provided.

The figure which shows schematically an example of the model of the face image recording system which concerns on each embodiment of this invention. The figure which shows schematically an example of the model of the face image recording system which concerns on each embodiment of this invention. The figure which shows a network type system structure. The figure which shows the external appearance structural example of a face image input device. The side view which shows the example of installation of a face image input device. The figure which shows the whole structure as a control system of a face image input device. The block diagram which shows schematically the structural example of a face recording server. The flowchart for demonstrating the flow of the input process of the face image in the input terminal which concerns on 1st Embodiment. The flowchart for demonstrating the flow of the input process of the face image in the input terminal which concerns on 1st Embodiment. The flowchart for demonstrating operation | movement of the face recording server which concerns on 1st Embodiment. The flowchart for demonstrating the flow of a process in the face recording server which concerns on 2nd Embodiment. The figure which shows the example of a display of a display part. The figure which shows the example of a display of a display part. The flowchart for demonstrating operation | movement of the face recording server which concerns on 4th Embodiment. The figure which shows the example of a display of a display part. The flowchart for demonstrating operation | movement of the input terminal concerning 5th Embodiment. The flowchart for demonstrating operation | movement of the input terminal concerning 5th Embodiment. The figure which shows the example of a display (structure example) of an action record report. The figure for demonstrating a person detection process. The flowchart for demonstrating operation | movement of the input terminal which concerns on 6th Embodiment. The flowchart for demonstrating operation | movement of the face recording server 3 which concerns on 6th Embodiment. The figure which shows the example of a display of a display part. The flowchart for demonstrating operation | movement of the face recording server which concerns on 7th Embodiment. The flowchart for demonstrating operation | movement of the input terminal which concerns on 7th Embodiment.

Hereinafter, first to seventh embodiments of the present invention will be described with reference to the drawings.
First, a system having a configuration common to the embodiments will be described.
The system according to each embodiment of the present invention can be roughly modeled into two as shown in FIGS. FIG. 1 is a diagram schematically illustrating a system configuration of model A, and FIG. 2 is a diagram schematically illustrating a system configuration of model B.

  First, as shown in FIG. 1, the model A includes an ID authentication device 1, a face image input device 2, a face recording server 3, a face history database 4, and the like. The ID authentication device 1 performs ID authentication processing based on ID information input by the user. Here, the ID information input by the user is information such as ID information stored in the ID card or information (ID number, password) input by key input. The face image input device 2 inputs (photographs) an image (face image) of an area including at least the face of the user, detects a face feature pattern from the input image, and face image data based on the input image Is transmitted to the face recording server 3. The face recording server 3 records the face image data transmitted from the face image input device 2 in the face history database 4 together with history information, or searches the face history database 4 for face image data of a specific person. It is something to do. The face history database 4 is composed of, for example, a large capacity magnetic disk.

  That is, in the model A, the user first inputs his / her ID information by an ID card or key input. Then, the ID authentication device 1 performs user authentication based on the ID information input by the user. Here, as user authentication, it may be determined whether or not the user is the person himself / herself, or may be determined as to who the user is.

  When the user is authenticated by the ID authentication device 1, the face image input device 2 inputs (photographs) the user's face image with a camera (not shown). When the face image of the user is input as face image data, the face image input device 2 detects a face feature pattern from the input image data. Thus, when a facial feature pattern is detected from the input image data, the facial image input device 2 sends the input image data from which the facial feature pattern has been detected to the face recording server 3 together with attribute information of the input image data. To transmit. The face recording server 3 records the input face image data transmitted from the face image input device 2 and its attribute information in the face history database 4.

Further, as shown in FIG. 2, the model B has a configuration in which the model A ID authentication apparatus 1 is omitted, and the user authentication based on the ID information in the model A as described above is omitted.
That is, in the model B, first, the face image input device 2 captures a user's face image. When the face image of the user is input, the face image input device 2 detects a face feature pattern from the face image data based on the input image. Thus, when a facial feature pattern is detected from the input image data, the facial image input device 2 sends the input image data from which the facial feature pattern has been detected to the face recording server 3 together with attribute information of the input image data. To transmit. The face recording server 3 records the input image data transmitted from the face image input device 2 and its attribute information in the face history database 4.

  As an implementation form of the model A and the model B, the model A and the model B may be configured as an integrated device (stand-alone type) having the function of the face image input device 2 and the function of the face recording server 3. As shown in FIG. 6, the face recording server 3 may be connected to a plurality of face image input devices 2 via a network (network type).

  In the following description, it is assumed that the entire system is configured as a network type as shown in FIG. That is, in this system, each face image input device (terminal) 2 transmits the face image data to the face recording server 3 together with the history information. The face recording server 3 stores face image data and history information sent from each face image input device 2 in a face history database 4 connected to the face recording server 3. Further, the face recording server 3 searches (tracks) the face data of the person specified by the operation of the administrator (monitoring person).

Next, the face image input device 2 will be described.
FIG. 4 is a diagram showing an external configuration example of the face image input apparatus 2. In FIG. 4, an automatic door and an electric lock are shown as examples of devices that operate based on the control of the face image input device 2.
In the configuration illustrated in FIG. 4, the face image input device 2 includes an operation unit 11 and a processing unit 12. The processing unit 12 is connected to the operation unit 11, and an electric lock control unit that controls an automatic door and an electric lock is connected to the processing unit 12.

As shown in FIG. 4, the operation unit 11 includes a display unit (display) 21, a camera 22, a speaker 23, a numeric keypad 24, and the like.
The display unit 21 is configured by, for example, a color liquid crystal display device. On the display unit 21, the detection result of the face area (for example, indicated by a circle representing the face area) is displayed on the image input from the camera 22. The camera 22 captures (inputs) a plurality of images (face images) of an area including at least the face of the user. The numeric keypad 24 is a key for inputting ID information and the like. The speaker 23 reproduces audio data from the processing unit 12 and performs audio guidance. When the face image input device 2 shown in FIG. 4 is applied to the model B, the numeric keypad 24 may be omitted.

  Further, the processing unit 12 detects a face feature amount from image data input by the camera 22, performs camera control to control the camera 22, numeric key control to control the numeric key 24, the automatic door or Functions such as door control for controlling the electric lock through the electric lock control, image transmission for transmitting the input image data from the camera 22 to the outside, and image recording for storing the input image data from the camera 22 in the storage device have.

FIGS. 5A and 5B are side views showing an installation example of the face image input device 2.
FIG. 5A is a side view showing an installation example of the face image input apparatus 2 having the wall-hanging type operation unit 11. In FIG. 5A, the operation unit 11 configured as shown in FIG. 4 is installed on a wall. In such a configuration, the face of the user P standing in front of the operation unit 11 is photographed by the camera 22 from obliquely below. The operation unit 11 is connected to the processing unit 12 via a cable.

  FIG. 5B shows an example of a pole-type face image input device 2 that can be stationary. In the configuration example shown in FIG. 5B, the configuration of the operation unit 11 such as the display unit 21 and the camera 22 is provided in a pole-shaped housing, and the processing unit 12 is installed in the housing. In the configuration shown in FIG. 5B, the height of the pole-shaped housing is set to, for example, 140 to 150 cm according to the height of the person. That is, the pole-shaped housing having the configuration shown in FIG. 5B only needs to be high enough to secure the mounting position of the camera 22 that can stably photograph the face facing the user. A display unit 21 is installed in the vicinity.

  Further, when the face image input device 2 having the configuration shown in FIGS. 5A and 5B is applied to a network type system, the processing unit 12 includes the above-described wireless communication such as wireless LAN or Bluetooth. A wireless communication function (transmission means) with the face recording server 3 may be provided. In this way, by installing the wireless communication function in the face image input device 2, it is possible to save the trouble of wiring for connecting to the network line and the face recording server 3, and the face image input device 2 can be easily installed and expanded. It becomes.

FIG. 6 is a diagram illustrating an overall configuration as a control system of the face image input apparatus.
As shown in FIG. 6, the face image input apparatus 2 includes an operation unit 11 and a processing unit 12 as a user interface. The face image input device 2 is connected to the face recording server 3 and a printer via a LAN. The operation unit 11 of the face image input device 2 has an external configuration as shown in FIG. 4, for example, and includes a display 21, a camera 22, a speaker 23, a numeric keypad 24, and the like. In addition, in the configuration example shown in FIG. 6, the illumination unit that illuminates the user's face photographed by the camera 22 is not shown, but the illumination unit that illuminates the user's face is provided near the camera 22. May be.

  The processing unit 12 of the face image input device 2 includes a processing processor 31, a work memory 32, a program memory 33, an I / O board 34, a capture board 35, a disk interface 36, a LAN interface 38, and a system bus 39. Has been. Further, the capture board 35 has an A / D conversion circuit 41 and an image memory 42.

  The processor 31 controls the face image input apparatus 2 as a whole by executing various processes. The work memory 32 is a memory that temporarily stores image data being processed, other data, and the like. The program memory 33 is a memory that stores a control program and the like. The processing processor 31 realizes various functions based on a control program stored in the program memory 33. For example, the processing processor 31 sends a control signal to each camera to fetch face image data, and performs face image data registration processing while using the work memory 32 for the data.

  The I / O board 34 functions as various interfaces and is connected to the display 21, the speaker 23, the numeric keypad 24, and the like. The capture board 35 is an interface for capturing an image captured by the camera 22 into the processing unit 12. The A / D conversion circuit 41 in the capture board 35 captures image data composed of analog signals photographed by the camera 22 and converts them into digital signal image data. The image memory 34 is a buffering buffer that temporarily stores image data. In the image memory 34, image data shot by the camera 22 and A / D converted by the A / D conversion circuit 41 is stored. When the camera 22 includes a USB (Universal Serial Bus) interface, the A / D conversion circuit 41 is not necessary and the USB interface may be provided.

  The disk interface 36 reads / writes data from / to the magnetic disk (recording device) 37. The magnetic disk 37 stores input image data input from the camera 22, history information of the input image data, and the like. The magnetic disk 37 also functions as a temporary buffer memory. The LAN interface 38 is an interface for connecting to a LAN. The system bus 39 is a data bus that transmits and receives data such as control data and image data. The system bus 39 is connected to the processor 31, work memory 32, program memory 33, I / O board 34, image memory 42 in the capture board 35, disk interface 36, and LAN interface 38.

  When the face image input device 2 is used in the form of Model A, the magnetic disk 37 is provided with an ID information table 37a in which ID information such as ID codes and passwords of the respective users is stored. The program memory 33 stores a control program for performing authentication based on ID information (ID authentication). Thereby, when the face image input apparatus 2 is used in the form of model A, the processor 31 serves as the ID authentication unit 1 in the ID information input by the user with the numeric keypad 24 and the ID information table 37a. The ID authentication for the user is performed by collating with the stored ID information.

The functions realized by the processor 31 based on the control program stored in the program memory 33 include a feature amount extraction process, an information generation process, an image process, and an image transmission process.
The feature amount extraction processing is processing for extracting a facial feature amount (feature pattern) from an image photographed by the camera 22. The information generation process is a process of generating attribute information (such as a shooting time) regarding an image shot by the camera 22. The image processing executed by the processing processor 31 is, for example, a process of extracting a face area, an image density correction process, a normalization process, or an image compression process on an image captured by the camera 22. The image transmission process is a process of transferring face image data based on an image taken by the camera 22 to the face recording server 3 together with history information.

Next, a configuration example of the face recording server 3 will be described.
FIG. 7 is a block diagram schematically showing a configuration example of the face recording server 3.
As shown in FIG. 7, the face recording server 3 includes a control unit 51, a program memory 52, a work memory 53, a magnetic disk interface 54, a display unit 55, an operation unit 56, a LAN interface 57, and the like.

  The control unit 51 controls the entire face recording server 3. The program memory 52 is a memory that stores a control program and the like. The work memory 53 is a memory that temporarily stores an image or data being processed. The magnetic disk interface 54 is an interface for accessing the face history database 4.

  The display unit 55 is constituted by a display device, and displays search results of the face history database 4 and the like. The operation unit 83 is configured by an input device such as a keyboard or a mouse, and an operation instruction or the like is input from a monitor who operates the face recording server 3. The LAN interface 57 is an interface for transmitting and receiving data to and from the face image input device 2 via the LAN.

  In the face recording server 3, various processes are realized by the control unit 51 operating based on the control program stored in the program memory 52. For example, the face recording server 3 has a function of recording a face image from the face image input device 2 together with its history in the face history database 4 and a function of searching information stored in the face history database 4. Have.

Hereinafter, seven embodiments realized based on the system configured as described above will be described.
First, the first embodiment will be described.
The first embodiment is applied to the model A network type system. In addition, as an implementation form of the first embodiment, an integrated apparatus or a network system may be used.
In the first embodiment described below, a face image input device (hereinafter referred to as an input terminal) 2 having an ID authentication function is installed at a plurality of points, and these input terminals are connected via a network such as a wireless LAN. It is assumed that the monitor is connected to the face recording server 3 operated by a supervisor or administrator.

First, face image input processing in the input terminal 2 according to the first embodiment will be described.
FIG. 8 is a flowchart for explaining a flow of face image input processing in the input terminal 2 according to the first embodiment. The processing shown in FIG. 8 is executed by the processing processor 31 of each input terminal.
First, a user at a point where the input terminal 2 is installed inputs an ID code and a password as ID information of the user using the numeric keypad 24 of the operation unit 11. The ID code is provided in advance by the administrator to each user, and the password is freely set by each user at the time of registration.

  When ID information (ID code and password) is input by the user, the processor 31 performs an authentication process (ID authentication) based on the ID information (step S11). In this ID authentication, the processor 31 receives the ID information (ID code and password) input by the user and the ID information (ID code and password of each user) stored in the ID information table 37a of the magnetic disk 37. Is matched.

  If there is no ID information in the ID information table 37a that matches the ID information input by the user as a result of this collation (step S11, failure), the processing processor 31 determines that the ID authentication for the user has failed. Thus, when ID authentication fails, the said processor 31 displays on the said display part 21 that the ID information which the user input was not authenticated, and rejects the said user's access.

  When ID information that matches the ID information input by the user by the above collation exists in the ID information database 37a (step S11, success), the processing processor 31 determines that the ID authentication for the user has been successful. When the ID authentication is successful in this way, the processing processor 31 displays, for example, that the ID information input by the user has been authenticated on the display unit 21 and an image (face image) including at least a front-facing face. ) Is urged to be taken by the camera 22. Along with such guidance, the processor 31 performs an image input process for inputting the user's face image by the camera 22 (step S12).

  In the image input process, the camera 22 captures an image for one frame under the control of the processing processor 31, and the captured image is captured into the processing unit 12 by the capture board 35. When the face image captured by the camera 22 is captured in the processing unit 12 as face image data, the processing processor 31 performs face detection processing by detecting facial feature points from the entire captured image data (step 13). ). In this face detection process, the face image area is roughly searched from the image data (input image data) captured by the camera 22, and then the four feature points of the pupil area center point and the nostril area center point in the face area are finely searched. Is detected as a feature point of the face.

  When face feature points cannot be detected from the input image in such face detection processing, that is, when face detection fails (step S13, failure), the processor 31 rejects the user's access. . In this case, the processor 31 returns to the image input process in step S12, and again captures a face image and performs a face detection process.

  In addition, when the face feature point can be detected from the input image by the face detection process, that is, when the face detection is successful (step S13, success), the processing processor 31 is based on the detected face feature point. Then, the face pattern area is cut out (step S14).

  When the face pattern area is cut out from the face image data input by this cut-out process, the processing processor 31 performs a normalization process on the image data of the face pattern area (step S16). In this normalization process, geometric correction and density correction are performed on the image data of the face pattern region. When these processes are completed, the processor 31 calculates (generates) a feature vector (face pattern) as a face feature amount based on the extracted feature points (step S16).

  When the feature vector as the feature amount of the user's face is calculated, the processing processor 31 generates history information including the ID information, shooting time, shooting point, and the like of the user as history information generation processing. When history information is generated by the history information generation process, the processor 31 transmits face history information including the history information, a captured image, a feature vector calculated from the image, and the like to the face recording server 3. A transmission process is executed (step S17). In this transmission process, when data is transmitted to the face recording server 3, the processor 31 uses the magnetic disk 37 as a temporary buffer.

  Further, in the operation example shown in FIG. 8, the face image of one frame taken by the camera 22 is targeted, but a plurality of preset (N) frames may be collected. As shown in FIG. 9, the operation of the input terminal 2 in this case repeats the processing of step S12 and step S13 in FIG. 8 until the number of frames in which facial feature points have been successfully detected is at least N. Is.

  That is, in the operation example shown in FIG. 9, first, when an image is input in step S <b> 12, the processing processor 31 determines whether or not the number of frames (input frame number) input from the camera 22 has become T. . If it is determined by this determination that the number of input frames has not reached T, the processing processor 31 performs face detection processing on the image data based on the image of the input frame (step S22). If a face feature point cannot be detected by this face detection process (step S22, failure), the processing processor 31 returns to step S12 and inputs an image of the next frame from the camera 22.

  When the facial feature point can be detected by the face detection process (step S22, success), the processing processor 31 accepts the image of the frame and the image of the frame (acceptance frame) as the magnetic disk 37. It accumulates in the storage means. When receiving the frame, the processor 31 determines whether or not the number of received frames has reached a predetermined number (M) (step S23). If it is determined by this determination that the number of accepted frames has not reached M, the processor 31 returns to step S12 and inputs an image of the next frame from the camera 22. If it is determined by the above determination that the number of accepted frames has reached M, the processor 31 proceeds to step S25 in which processing is performed on the accumulated accepted frames.

  If it is determined in step S21 that the number of input frames has reached T, the processor 31 determines whether or not the number of frames received so far is N or more (step S24). If it is determined by this determination that the number of accepted frames is not N or more, the processor 31 rejects the face image input process for the user. On the other hand, if it is determined by the above determination that the number of accepted frames is N or more, the processing processor 31 proceeds to step S25.

  In step S25, the processes in steps S14 to S16 are performed on a plurality of images stored in a recording unit such as the magnetic disk 37 as a reception frame. That is, when at least N or more images in which facial feature points can be detected have been acquired, the processing processor 31 extracts face pattern regions, image normalization processing, and feature vectors for those images. Perform the calculation process.

  When the process of step S25 is completed, the processor 31 generates history information including user ID information, shooting time, shooting point, and the like for the input group of face image data. When such history information is generated, the processor 31 obtains face history information including the history information, all received face image data, feature vectors calculated from the face image data, and the face recording server 3. (Step S17).

  Note that the greater the number of frames of the image transmitted to the face recording server 3, the higher the accuracy of the face search in the face recording server 3. However, as the number of frames of face image data to be collected by the input terminal 2 increases, the face image input (registration) process takes longer. Therefore, in the operation example shown in FIG. 9, the parameter M is introduced, the number of input frames within a range that the user can accept as processing at the input terminal is set, and N to M frames within the number of input frames. The number of frames (at least N or more frames) is collected.

  Further, in the explanation of the operations in FIG. 8 and FIG. 9, processing related to guidance displayed on the display unit and voice guidance by the speaker is omitted, but the front of the face is shown to the user at the time of image input processing. Guidance is directed to the camera 22, the user is notified of access permission when face detection is successful, the user is denied access when it fails, and the user is prompted to retry. You may do it.

Next, the operation of the face recording server 3 according to the first embodiment will be described.
FIG. 10 is a flowchart for explaining the operation of the face recording server 3 according to the first embodiment.
First, the face recording server 3 receives face history information from the input terminal 2 (step S31). Here, it is assumed that the face history information received from the input terminal 2 includes at least user ID information and a facial feature pattern (face pattern) extracted from the user's face image data.

  When the face history information is received from the input terminal 2, the control unit 51 of the face recording server 3 records the received face history information in the face history database 4 in association with the ID information (step S32). Further, the control unit 51 determines whether or not the face history information is the first face history information corresponding to the ID information (step S33).

  When it is determined by this determination that the face history information is the first face history information corresponding to the ID information, that is, when face history information having the same ID information as the face history information does not exist, the control unit 51 The face pattern included in the face history information is recorded in the face history database 4 as the representative face pattern of the user corresponding to the ID information (step S34). Here, the representative face pattern is assumed to represent the face pattern of each user. The representative face pattern is used as a face pattern as dictionary data for each user when a matching process (identification process) is performed using a newly acquired face pattern.

  If it is determined in step S33 that the face history information is not the first face history information corresponding to the ID information, that is, face history information having the same ID information as the face history information is already stored in the face history database 4. If the face history information is present, the control unit 51 determines whether the face pattern (representative face pattern) corresponding to the ID information of the face history information already recorded in the face history database 4 and the face pattern of the face history information. A collation (identification) process is performed (step S35). As a result of this matching process, a matching degree between the face pattern of the face history information and the representative face pattern is calculated.

  As a specific processing method of face pattern matching processing, a “mutual subspace method” as described in Document 3 can be applied. Furthermore, as face pattern matching methods, conventional techniques known in the field of digital image processing and pattern recognition (for example, “correlation method between feature vectors” and “subspace method” used in character recognition, etc. ) Can be applied.

  When the collation degree as a result of the collation processing is calculated, the control unit 51 determines whether or not the calculated collation degree is equal to or greater than a predetermined threshold (step S36). If it is determined by this determination that the collation degree is equal to or greater than a predetermined threshold, the control unit 51 determines that the user is not the person himself (“spoofing”) and monitors the display unit 55 or an alarm (not shown). Inform the employee that the user is “spoofing”.

  When it is determined that the user is “spoofing”, the face history information recorded in the face history database 4 in step S32 includes the fact that the user is “spoofing” or the face history information includes Invalidity is recorded. In such a case, the face history information may be deleted from the face history database 4.

  Here, “spoofing” means that a person who has performed ID authentication at the input terminal 2 is impersonating another user. Therefore, “spoofing detection” is based on the face image (face image data) taken by the input terminal 2 to determine whether the person who has performed ID authentication at the input terminal 2 is impersonating another user. Is.

  If it is determined by the above determination that the collation degree is not equal to or greater than a predetermined threshold, the control unit 51 determines that the user is the person himself (not “spoofing”). Thus, when it is determined that the user is the user, the control unit 51 performs a process of updating the representative face pattern of the user with the face pattern of the face history information (step S38).

  In the representative face pattern update process, the representative face pattern may be rewritten to the face pattern of the face history information, and an average face pattern is calculated from the existing representative face pattern and the face pattern of the face history information. May be used as a representative face pattern. In the former case, since the face pattern of the face history information is the latest face pattern of the user, there is an advantage that the representative face pattern of the user can always be rewritten to the latest face pattern. In the latter case, since the representative face pattern is updated based on the existing representative face pattern and the latest face pattern, there is an advantage that a high-quality representative face pattern can be generated.

  If it can be determined that the face pattern of the face history information is not a stable face pattern, the update process of the representative face pattern of the user may not be performed. In step S38, the representative face pattern is not updated, and the administrator interactively designates the face pattern recorded in the face history database 4 as the representative face pattern. good.

  As described above, in the first embodiment, the face image of the user authenticated by ID authentication based on the ID information input by the user at the input terminal 2 is captured, and the face image data based on the face image is used. The face recording server confirms whether or not the user is the person himself / herself by face matching.

  That is, in the first embodiment, the face recording server is a face image of past history information having the same ID information based on the ID information included in the face history information sent from each input terminal. If the data and the face image data of the face history information are collated, and the person of the face history information is not confirmed (identified) by this collation processing, it is determined as “spoofing (not the person)” This is to inform the monitoring personnel.

  Thus, according to the first embodiment, each input terminal in real time (including a near real time as a slight time delay for processing based on face history information from a plurality of input terminals). The “spoofing” can be determined by the face recording server and notified to the monitoring staff.

Hereinafter, a second embodiment will be described.
This second embodiment is applied to the model B network system, for example. Further, as an implementation form of the second embodiment, it may be an integrated device or a network type system. In the second embodiment, a list of face history information of the user (“one-point tracking” result) is monitored in real time.

  In the second embodiment, the appearance configuration and the overall system configuration of a face image input device as an input terminal are the same as those in the first embodiment. However, in the second embodiment, ID authentication (key-in input and code processing thereof) at the input terminal 2 is not required (for model B), and the configuration related to ID authentication of the numeric keypad 24, etc. May be omitted.

  In the second embodiment, the face history information recorded in the face history database 4 is recorded for each cluster as a group of face history information for each user. Accordingly, the face history information recorded in the face history database 4 is given a cluster number assigned for each user, and is grouped for each cluster. In each cluster, a representative face pattern is defined as a representative face pattern of the user.

Next, the operation of the input terminal 2 according to the second embodiment will be described.
In the second embodiment, the input terminal 2 performs substantially the same operation as the first embodiment except for the ID authentication. That is, the input terminal 2 according to the second embodiment performs an operation in which the operation in step S11 is omitted from the operation illustrated in FIG. Therefore, the input terminal 2 generates face history information that does not include user ID information and transmits the face history information to the face recording server 3.

Next, the operation of the face recording server 3 according to the second embodiment will be described.
FIG. 11 is a flowchart for explaining the flow of processing in the face recording server 3 according to the second embodiment.

  First, the face recording server 3 receives face history information from the input terminal 2 (step S41). Here, the face history information received from the input terminal 2 includes a face feature pattern (face pattern) extracted from the user's face image, but does not include user ID information. .

  When the face history information is received from the input terminal 2, the control unit 51 of the face recording server 3 determines whether or not the received face history information is the face history information received by the face recording server 3 first ( Step S42). If it is determined as the first face history information by this determination, that is, if face history information is not accumulated in the face history database 4 (YES in step S42), the control unit 51 proceeds to step S46 described later.

  When it is determined that the face history information is not the first face history information, that is, when it is determined that the face history information is stored in the face history database 4 (NO in step S42), the control unit 51 receives the information. The face pattern of the face history information thus matched is compared with the representative face pattern of each cluster stored in the face history database 4 (step S43). In this matching process, the matching degree between the face pattern of the received face history information and the representative face pattern of each cluster is calculated, and the maximum matching degree among the matching degrees is used as a matching result. Further, a terminal ID indicating an input terminal that has input a face image is included in the face history information, and in the matching process, a cluster to be matched with the face pattern of the received face history information is the same terminal ID as the face history information. May be limited to those having face history information.

  When the maximum matching degree for the face pattern of the face history information is obtained by the matching process, the control unit 51 determines the cluster number of the representative face pattern that is the maximum matching degree (step S43). Further, the control unit 51 determines whether or not the maximum collation degree obtained by the collation process is greater than or equal to a predetermined threshold (step S44).

  If it is determined by this determination that the maximum matching degree is not equal to or greater than a predetermined threshold, the control unit 51 generates one new cluster and records it in the face history database 4. When a new cluster is recorded in the face history database 4, the control unit 51 further records the received face history information in the face history database 4 as face history information belonging to the cluster. At this time, the control unit 51 records the face pattern of the face history information in the face history database 4 as a representative face pattern of the cluster (step S46).

  When the face history information is recorded in the face history database 4 by such an operation, the control unit 51 informs the display unit that the user of the face history information is the first user who took a picture with the input terminal 2. indicate. Thereby, the supervisor of the face recording server 3 can recognize in real time that the first user has appeared on the input terminal 2.

  If it is determined in step S45 that the maximum matching degree obtained by the matching process is equal to or greater than a predetermined threshold, the control unit 51 determines the maximum matching determined by the user of the face history information in step S44. It is determined that the user is the user of the cluster (corresponding cluster). Accordingly, the control unit 51 records the face history information in the face history database 4 so that the face history information belongs to the corresponding cluster (step S48). When the face history information is recorded in the face history database 4, the control unit 51 updates the representative face pattern of the cluster with the face pattern of the face history information (step S49).

  When the face history information is recorded in the face history database 4, the control unit 51 displays the action history of the users of the cluster on the display unit 55. At this time, for example, a screen as shown in FIG. 12 is displayed on the display unit 55 as the action history of the user. In the display example illustrated in FIG. 12, information indicating the user's action history is displayed together with the cluster number of the user and the terminal number indicating the input terminal that has captured the latest face image. The user's action history displayed on the display unit 21 includes, for each face history information in the cluster, a registration number, access date / time (access date / time, access time), face image (of the recorded face image). Etc.) are displayed in order from the latest one.

  In step S49, the representative face pattern of the cluster is updated based on the face pattern of the received face history information (latest face history information). The administrator of the recording server 3 may specify whether to update the representative face pattern of the corresponding cluster with the face pattern of the face history information received. Further, each cluster can have a plurality of representative face patterns, and each time face history information is received, the face recording server 3 may add the face pattern of the face history information as a representative face pattern. . Further, each time the face history information is received, the control unit 51 may merge with the representative face pattern of the cluster corresponding to the face pattern of the received face history information. Here, merging means, for example, averaging a plurality of face patterns or statistically performing principal component analysis to hold only the principal components.

As described above, in the second embodiment, the face record server records the face history information from each input terminal in the face history database 4 and the face history database 4 that are currently sent. The user's action history is displayed based on the face history information belonging to the cluster having the identified representative face pattern and the face pattern of the face history information. Thereby, the manager of the face recording server can easily check the action history of the person every time the person is photographed at each input terminal.
In the second embodiment, since the face history information is classified and recorded as clusters for each user in the history database, the matching process can be made efficient.

The third embodiment will be described below.
The third embodiment is applied to the model B network type system, for example. In the third embodiment, since input terminals installed at a plurality of points are required, a network type system is realized as an implementation form.

  In the third embodiment, the face recording server 3 monitors a user's action trajectory (“multi-point tracking” result) taken by each input terminal 2 in real time. That is, the third embodiment is an extension of the second embodiment, and is photographed at each input terminal based on face history information obtained from input terminals installed at a plurality of points. As a user's action history, the user's action trajectory is notified to the monitoring staff.

  The appearance configuration and system configuration of the face image input apparatus to which the third embodiment is applied are the same as those of the second embodiment. That is, the third embodiment is also the same as the first embodiment, and ID authentication (key-in input and its code processing) at the input terminal 2 in the first embodiment is not necessary.

  In the third embodiment, as in the second embodiment, the face history information recorded in the face history database 4 is classified for each cluster as a group of face history information for each user. To be recorded. In particular, in the third embodiment, since it is necessary to perform collation processing on history information between a plurality of terminals, the processing amount is increased as compared with the second embodiment. For this reason, in the third embodiment, the search results (user cluster information) so far are always retained to eliminate duplication processing and to make the identification processing more efficient.

  The operation of the input terminal 2 according to the third embodiment is also the same as that of the second embodiment. That is, in the third embodiment, the input terminal 2 performs an operation in which the operation in step S11 is omitted from the operation illustrated in FIG. 8 or FIG. Therefore, the input terminal 2 generates face history information that does not include user ID information and transmits the face history information to the face recording server 3. However, in the third embodiment, the face history information includes at least the shooting location and the shooting time for shooting the face image. As the information indicating the shooting location, a terminal number indicating the input terminal that shot the face image is used.

  The operation of the face recording server 3 according to the third embodiment is substantially the same as that of the second embodiment except for the information as the user action history displayed on the display unit 55. That is, in the third embodiment, the operation of the face recording server 3 is the same as the operation shown in FIG. 11 except for step S50.

  In step S50 of the operation shown in FIG. 11, the control unit 51 of the face recording server 3 first stores the history information of the user (cluster) that has been successfully verified with the face pattern of the received face history information in the face history database. Read from 4. Next, based on the history information read from the face history database 4, the control unit 51 displays the user's action trajectory on the display unit 55. At this time, a screen as shown in FIG. 13 is displayed on the display unit 55.

  In the display example shown in FIG. 13, the cluster number and the latest access date and time are displayed, and the point where the user has been photographed in the past based on the user's past face history information (the terminal number of the photographed input terminal) ) And the shooting time are displayed as a trajectory of action. In the display example shown in FIG. 13, the shooting point and the shooting time are displayed in a chart.

  The face history information displayed on the display unit 55 as the user's action trajectory may be limited based on a predetermined condition. For example, the trajectory of the user's action may be displayed by limiting the elapsed time up to the present to face history information within a predetermined period. In this case, it is possible to display only those whose elapsed time up to the present time is a predetermined period, and it is possible to prevent the user's behavior trajectory based on the old face history information from being displayed. Further, the number of face history information to be displayed as a user's action trajectory may be limited. For example, a predetermined number of pieces of face history information are displayed in order from the latest face history information. Thereby, a predetermined number of trajectories can be displayed, and a display that can be easily viewed by the supervisor can be performed.

Hereinafter, a fourth embodiment will be described.
The fourth embodiment is applied to, for example, the model A and model B network type systems. Moreover, as an implementation form of the fourth embodiment, an integrated apparatus or a network type system may be used. In the fourth embodiment, face images are registered in advance and “specific person detection” is performed in real time.
In the fourth embodiment, the appearance configuration of the face image input device as the input terminal 2 and the configuration of the entire system are the same as those in the first embodiment. In the fourth embodiment, the input terminal 2 may be in the form of model A that performs ID authentication (key-in input and its code processing), or in the form of model B that does not require ID authentication. May be.

  In addition, the operation of the input terminal 2 in the fourth embodiment may be a person who performs the operation shown in FIG. 8 or FIG. 9 as in the first embodiment, or the second and third An operation in which the operation in step S11 is omitted from the operation shown in FIG. 8 or FIG.

Next, the operation of the face recording server 3 according to the fourth embodiment will be described.
In the fourth embodiment, the administrator creates a face image information (face pattern) list (hereinafter referred to as “specific person list”) 4 a of a specific person in advance and records it in the face history database 4. Shall be kept. Here, the specific person is a person that the administrator particularly wants to monitor. For example, the specific person may be a suspicious person or a person who needs attention.

  The face pattern registered in the specific person list 4a is a face pattern selected from the face history information stored in the face history database 4. Further, the face pattern registered in the specific person list 4a may be a face pattern created from face image data such as a photograph, a face pattern taken from another device, or a face taken from another device. It may be a face pattern created from an image.

FIG. 14 is a flowchart for explaining the operation of the face recording server 3 according to the fourth embodiment.
First, it is assumed that the administrator of the face recording server 3 registers the specific person list 4a as described above in the face history database 4 in advance. When the face history information is received from the input terminal 2 in this state, the control unit 51 records the received face history information in the face history database 4 (step S61). When the face history information received from the input terminal 2 is recorded in the face history database 4, the control unit 51 includes the face pattern of the received face history information and all the face patterns registered in the specific person list 4a. Is performed (step S62). As a result of this collation processing, the collation degree between the face pattern of the received face history information and all the face patterns registered in the specific person list 4a is obtained.

  When the result of the collation process is obtained, the control unit 51 determines whether or not each collation degree obtained as a result of the collation process is greater than or equal to a threshold value. If it is determined that there is no verification degree equal to or greater than a predetermined threshold, the process is terminated. Further, when the collation degree is equal to or higher than a predetermined threshold based on the above determination, the control unit 51 reads information on the specific person giving the collation degree from the specific person list 4a and displays it on the display unit 55. In this case, for example, a display example as shown in FIG. 15 is displayed on the display unit 55.

  In the display example shown in FIG. 15, the access date and time, the terminal number indicating the input terminal that has input the face history information, the face image of the face history information, and the hit specific person (the face pattern of the face history information and The face image of a person whose collation degree is equal to or greater than a predetermined threshold is displayed.

  The fourth embodiment can be implemented in combination with the first, second, and third embodiments. For example, by using together with the first embodiment, the face history information of the person who is determined to have performed “spoofing” by “spoofing detection” as described in the first embodiment is used as the specific person list 4a. It is also possible to register for. In addition, by carrying out in combination with the fourth embodiment and the second or third embodiment, past history information of a person collated with a specific person is extracted from the face history database, and the person It is possible to display an action history, an action trajectory, and the like.

  As described above, when a specific person list is registered in advance in the face history database 4 and the face recording server receives face history information from the input terminal 2, the face pattern of the received face history information and the specific When the matching process is performed with the face patterns of all persons registered in the person list, and the person in the face history information is identified as the person registered in the specific person list by this matching process, this is indicated. It is intended to notify the monitor.

  Thereby, according to the fourth embodiment, the monitoring staff of the face recording server can easily find a person designated in advance or can easily monitor a specific person, and the security level can be increased. Can be improved.

The fifth embodiment will be described below.
The fifth embodiment is applied to an intermediate model between the model A and the model B, for example. In the fifth embodiment, information indicating a user's action record such as a diary is automatically created.

  In the fifth embodiment, for example, ID information is input at least once at the time of first access or the like. That is, in the fifth embodiment, it is a condition that ID information is input at least once, but a user who has input ID information does not necessarily need to input ID information in subsequent operations. In addition, as an implementation form of the fifth embodiment, an integrated device may be used or a network system may be used.

Next, the operation of the input terminal 2 according to the fifth embodiment will be described.
In the fifth embodiment, the input terminal 2 is substantially the same as the operation in the first embodiment as the operation at the time of normal operation. Of the operations shown in FIG. 8 or FIG. Processing is different. That is, each user needs to input his / her ID information to the input terminal 2 at least once during normal operation. Therefore, a user who has input ID information at least once can omit input of ID information during normal operation. In this case, the input terminal performs an operation in which the operation in step S11 is omitted from the operation illustrated in FIG. 8 or FIG.

FIG. 16 is a flowchart for explaining the operation of the input terminal 2 according to the fifth embodiment.
First, the administrator activates a dedicated operation mode (administrator mode) using the numeric keypad 24 as an operation unit of the input terminal 2. In the administrator mode, an operation mode for creating a user action record as the fifth embodiment can be executed.

  When this manager mode is activated, the manager instructs display or creation of user action records. Then, the processor 31 of the input terminal 2 accesses the face history database 4 via the face recording server 3. When the face history database 4 is accessed, the processor 31 searches for face history information to which ID information is assigned from among the face history information recorded in the face history database 4.

  Thus, when the face history information to which the ID information is assigned is searched, the processing processor 31 uses the face pattern of the face history information to which the ID information is assigned as a matching template (dictionary data) for another face. The collation process with the face pattern of the history information is performed. With respect to the face pattern of the face history information to which ID information is not given by this matching process, the degree of matching with the face pattern of each face history information to which ID information is given is calculated.

  Based on the result of this matching process, the processor 31 assigns ID information of face history information having a face pattern having the highest matching degree to face history information to which no ID information is assigned (step S71). . That is, the face history information to which no ID information is assigned is identified as the face history information of the face pattern having the highest matching degree.

  As a result, when the ID information is assigned to the face history information to which no ID information is assigned, the processing processor 31 summarizes the history of shooting locations, shooting times, etc. in time series for each ID information. An action record report indicating information, that is, an action history of each user is created and output. This action record report may be printed on paper connected to a LAN or the like, or may be displayed on the display unit 21.

FIG. 17 is a flowchart for explaining the operation of the input terminal 2 according to the fifth embodiment, and is a modification of the operation shown in FIG.
That is, in the operation example shown in FIG. 17, in step S71 described above, the ID information is not simply assigned to the ID information with the highest matching degree, but is assigned to the ID information with the highest matching degree and higher than the predetermined threshold value. . Therefore, in the operation example shown in FIG. 17, when the maximum matching degree is smaller than the predetermined threshold, the allocation of ID information to the face history information is suspended (step S73).

  Next, the processor 31 of the input terminal 2 collects the face history information for which the ID information has been assigned for each ID information and merges the face patterns, thereby obtaining one template corresponding to the ID information (matching). Dictionary data) is generated (step S74). As a result, when a template for each ID information is generated, the processing processor 31 determines the face pattern of the face history information (face history information for which ID information allocation is suspended) and the face patterns that have not yet been assigned ID information. A collation process with a template for each ID information is performed.

With respect to the face pattern of the face history information whose ID information is put on hold by this matching process, a matching degree with the template for each ID information is calculated. Based on the result of this collation processing, the processor 31 assigns the ID information of the template having the highest collation degree to the face history information whose ID information assignment is suspended (step S75). As a result, ID information is assigned to all face history information, and an action record report for each ID is created and output in step S72.
Further, in the operation examples shown in FIGS. 16 and 17, an action record report is generated and output for all face history information to which ID information is assigned. By specifying only the ID information, an action record report regarding the user of the ID information may be created and output. In this case, since it is not necessary to perform collation processing for all face patterns of face history information to which ID information is assigned, it is possible to shorten the processing time and quickly create and output a desired user action record report.

  The main difference between the operation shown in FIG. 16 and the operation shown in FIG. 17 is that the processing shown in FIG. 17 is more accurate but the processing time is longer.

  In the process of step S74, a plurality of face patterns are integrated (merged) into one template. As a merging method, for example, averaging as shown in Non-Patent Document 4 is performed. Alternatively, a conventional method of statistically performing principal component analysis and extracting only the principal components can be used. Further, the template created in the process of step S74 may be used as a template representing the face pattern of each ID information from the next collation process. 16 and 17 belong to a technique called “clustering” in the pattern recognition field, and for example, an algorithm called “K average algorithm” shown in Non-Patent Document 4 is used. May be used.

  Moreover, the said action recording report is comprised like the example of a display shown, for example. In the display example shown in FIG. 18, a user ID as ID information and a terminal number indicating an input terminal, a registration number as an action record, an access date (access date and time), a face image, and the like are displayed.

  The action record report is output on the display screen of the terminal (operation unit), and after confirmation by the administrator, is printed out according to the instruction. A printer that prints an action record report may be connected to a network via a LAN, for example, or may be incorporated in a terminal device.

  The face recording server 3 according to the fifth embodiment normally performs the same operation as that of the first, second, third, or fourth embodiment during operation. Further, the face recording server 3 may execute processing similar to the processing in FIG. 16 or FIG. In this case, the manager of the face recording server 3 may be started by activating a dedicated operation mode (administrator mode) by the operation unit 56.

  As described above, in the fifth embodiment, face pattern matching processing is performed on past face history information, and history information as action records for each user is created and output. It is. Thereby, the face history information accumulated in the face history database can be classified, and further, action records for each user can be easily extracted.

Hereinafter, a sixth embodiment will be described.
The sixth embodiment can be applied to each of the systems described above. In the sixth embodiment, a camera 22 for inputting a face image is also used as a surveillance camera for person monitoring. In the system as described above, the camera 22 is in a standby state except when a face image is input. Therefore, in the sixth embodiment, a “person monitoring function” is newly added to increase the use efficiency of the camera 22.

  In the sixth embodiment, a person approaching the camera 22 and a person moving away from the camera 22 are detected based on an image photographed by the input terminal 2, and the camera is in the meantime. The video imaged by the camera 22 is transmitted to the face recording server 3 and recorded.

  In the following description of the sixth embodiment, an example of application in a network type system will be described. However, the present invention can also be applied to a stand-alone apparatus instead of a network type system. When the sixth embodiment is applied to a stand-alone device, the moving image captured by the camera 22 during the time period from when the person approaches the camera 22 until it moves away is the history information of the user. As described above, the data may be recorded on the magnetic disk 37 in the input terminal 2.

  Normally, a user (or a passerby) has a flow of approaching, inputting a face image, and moving away from the camera 22 installed at a certain point. Therefore, in the sixth embodiment, images from the time when the user approaches the camera 22 until the user moves away are acquired regardless of the success or failure of the face image input process. For this reason, the process described in the sixth embodiment includes the face history information for recording the user's face history information described in the first, second, and third embodiments in the face history database 4. This process is executed in parallel with the face history information recording process as a process different from the recording process (referred to as the input mode in the sixth embodiment).

Next, detection of approaching and moving away of a person from the camera 22 will be described.
First, the approach of the person to the camera 22 is detected when a proximity sensor provided in the vicinity of the camera 22 detects the person or when an image region that is likely to be a person is detected from an image captured by the camera 22. Detected. Further, the distance of the person from the camera 22 is detected when the proximity sensor provided in the vicinity of the camera 22 no longer detects the person, or from the image captured by the camera 22, a human-like image area is detected. Detected when it runs out. In other words, the detection of a person includes a proximity sensor and an image process for an image taken by the camera 22.

Next, processing for detecting a person by image processing on an image captured by the camera 22 will be described.
FIG. 19 is a diagram for explaining person detection processing for detecting the presence of an image or a person photographed by the camera 22.
As shown in FIG. 19, first, a background image taken by the camera 22 is registered in the magnetic disk 37 or the like (step S81). In this state, the processor 31 acquires an image taken by the camera 22 as needed (step S82). The processor 31 calculates the difference between the image data captured by the camera 22 and the background image data or binarizes the image data captured by the camera 22 at any time. It is determined whether or not there is (step S83).

  If it is determined that there is a change between the image data captured by the camera 22 and the background image data registered in advance, the control unit 51 determines the image data captured by the camera 22 and the background image. A change component is extracted from difference image data from the data or the like (step S84). When the change component is extracted, the control unit 51 determines whether or not the size of the area of the extracted change component is larger than a predetermined threshold value. When it is determined that the area of the change component extracted by this determination is larger than a predetermined threshold, the control unit 51 detects that a person is present within the shooting range of the camera 22.

Next, the operation of the input terminal 2 according to the sixth embodiment will be described.
FIG. 20 is a flowchart for explaining the operation of the input terminal 2 according to the sixth embodiment.
First, it is assumed that the camera 22 of the input terminal 2 is taking images of a predetermined shooting range as needed. For example, the processing processor 31 captures an image from the camera 22 at predetermined intervals (step S91), and determines whether or not a person exists within the shooting range of the camera 22 by the person detection process as described above. Judgment is made (step S92).

  When the presence of a person is detected by this determination, the processing processor 31 starts transmitting image data of continuous images (moving images) captured by the camera 22 to the face recording server 3 (step S93). . Thus, when transmission of moving image data to the face recording server 3 is started, the processing processor 31 continuously executes transmission of moving image data to the face recording server 3 and the camera 22. A face detection process for detecting the face image of the user is performed on the image taken by the user (step S94). Here, it is assumed that the input terminal 2 is model B. However, if the input terminal 2 is model A, it is determined in step S94 whether or not ID information has been input by the user.

  When a face image is not detected in the face detection process (step S94, NO), the processing processor 31 performs the person detection process to determine whether or not a person is out of the shooting range of the camera 22. (Step S95). If it is determined by this determination that there is no person, that is, if it is determined that there is still a person within the shooting range of the camera 22 (step S95, NO), the processing processor 31 performs step S94. Return to. If it is determined in step S95 that there is no more person (YES in step S95), the processing processor 31 ends the transmission of moving image data to the face recording server 3 (step S96).

  When a face image is detected in the face detection process in step S94 (YES in step S94), the processing processor 31 performs a process of an input mode for generating face history information and transmitting it to the face recording server 3. Execute (Step S97). When the processing in this input mode is executed, the processor 31 further performs the person detection process to determine whether or not the person is no longer in the shooting range of the camera 22 (step S98).

  If it is determined by this determination that there is no person, that is, if it is determined that a person still exists within the shooting range of the camera 22 (NO in step S98), the processor 31 disappears. The person detection process is repeated until it is determined that the If it is determined in step S98 that there are no more persons (step S98, YES), the processing processor 31 ends the transmission of moving image data to the face recording server 3 (step S96).

  As described above, the input terminal 2 detects the presence / absence of a person within the shooting range from the image taken by the camera 22, and if there is a person, the moving image is parallel to the input mode processing. The image data is transmitted to the face recording server 3 until there is no more person. As a result, it is possible to transmit images from the time when the person approaches the camera 22 until the person moves away to the face recording server 3.

Next, the operation of the face recording server 3 according to the sixth embodiment will be described.
FIG. 21 is a flowchart for explaining the operation of the face recording server 3 according to the sixth embodiment.
First, the control unit 51 of the face recording server 3 determines whether or not there is a request for transmitting a moving image from the input terminal 2 (step S101). If it is determined that there is a request to transmit moving image data from the input terminal 2, the control unit 51 receives moving image data (stream data) transmitted from the input terminal 2. Are displayed on the display unit 55. At this time, for example, a screen as shown in FIG. 22 is displayed on the display unit 55. In the display example shown in FIG. 22, moving image data transmitted from each input terminal 2 is displayed together with a terminal number indicating each input terminal. Although the moving image data sent from the input terminal 2 is displayed on the display unit 55 here, the moving image data sent from the input terminal 2 is recorded on the recording device such as the face history database 4. You may make it record on.

  Thereby, the monitor of the face recording server 3 can monitor the state within the shooting range of the camera 22 in real time by looking at the moving image displayed on the display unit 55.

As described above, in the sixth embodiment, moving image data including a person photographed by the camera 22 of the input terminal 2 is transmitted to the face recording server 3 together with the operation as in the other embodiments. The face recording server displays or records moving image data transmitted from each input terminal as a moving image.
Thereby, the said monitoring person can monitor the mode in the imaging | photography range of the camera of each input terminal efficiently in real time.

The seventh embodiment will be described below.
The seventh embodiment is applied to, for example, the systems of the above-described embodiments. In the seventh embodiment, face images are registered in advance, a face pattern as a face matching template (dictionary) is generated from the registered face images, and feature points of the face are detected. In addition to the face detection process to be performed, a simple face image matching process is added.

  The seventh embodiment can be realized as an additional function to the first to sixth embodiments. In the seventh embodiment, personal authentication using a face image is executed as a post-processing of face detection processing at the input terminal 2. Furthermore, the difference between the personal authentication in the seventh embodiment and the conventional personal authentication device is that the face dictionary of the user is constituted by history information by the operation of the administrator. Thereby, in the seventh embodiment, there is an advantage that the user's usability is not lost because the operation of registering the face image for the user does not occur and is executed as a post-processing of face detection. Furthermore, the threshold value of the collation degree in personal authentication at the input terminal is set lower than usual, and there is also an advantage that the user error rejection rate becomes almost zero and user convenience is not lost.

First, the operation of the face recording server 3 according to the seventh embodiment will be described.
FIG. 23 is a flowchart for explaining the operation of the face recording server 3 according to the seventh embodiment. Here, it is assumed that the face history information stored in the face history database 4 does not include ID information indicating the user.

  First, the administrator of the face recording server 3 instructs the clustering of the face history information accumulated in the face history database 4 by the operation unit 56. If the implementation is a stand-alone, clustering is instructed by the numeric keypad 24 of the input terminal 2. Then, the control unit 51 performs face history information clustering by collating the face patterns of all the face history information accumulated in the face history database 4 (step S111).

  As a result of such collation processing, a combination of face patterns (face history information) having a collation degree equal to or higher than a predetermined threshold is obtained. That is, the face history information of the same user is determined from the face history information accumulated in the face history database 4 by this matching process. Based on the result of this collation processing, the control unit 51 classifies the face history information accumulated in the face history database 4 for each user. When the face history information is classified for each user, the control unit 51 assigns a cluster number as an ID code to each user (cluster) (step S112).

Further, the control unit 51 integrates (merges) the face patterns of the face history information into one in each cluster, so that one template for each cluster (user) (face matching dictionary data). As a face pattern is generated (step S113). As a method for merging facial pattern (feature vector) data in a cluster into one template, for example, a conventional method is used in which only principal components are extracted by averaging or statistically performing principal component analysis. be able to. These techniques are described in Non-Patent Document 4 and the like.
When the template data of each cluster is generated by such processing, the control unit 51 downloads the template data to the input terminal 2 side (step S114). This template data includes at least a template of each cluster (user) and an ID code.

  As described above, in the seventh embodiment, the face recording server 3 collates face patterns of the face history information accumulated in the face history database 4, and based on the collation result, The history information is clustered for each user, and a template (representative face pattern) of the user is generated from the face patterns of the face history information in each clustered cluster and downloaded to the input terminal 2. .

Next, the operation of the input terminal 2 according to the seventh embodiment will be described.
First, the template data from the face recording server 3 is downloaded to the input terminal 2. The template data includes at least a face pattern (template) for face matching and an ID code assigned to the cluster (user). Therefore, when template data is downloaded from the face recording server 3, the processor 31 of the input terminal 2 associates the template of the template data with the ID code of the user and registers them in the magnetic disk 37. It is assumed that the user of the face history information clustered by the face recording server 3 is informed of his / her ID code from the administrator.

FIG. 24 is a flowchart for explaining the operation of the input terminal 2 according to the seventh embodiment.
Steps S11 to S17 in the flowchart shown in FIG. 24 are the same as steps S11 to S17 in the flowchart shown in FIG. That is, when the user's face pattern is generated from the image input from the camera 22 in step S16, the processor 31 displays the face pattern generated from the input image and the user recorded on the magnetic disk 37. Is collated with the template (face pattern) corresponding to the ID information input in step S11 (step S121).

  The result of this matching process is given as a matching degree. Therefore, the processor 31 determines whether or not the collation degree obtained by the collation process is greater than or equal to a predetermined threshold (step S122). If it is determined by this determination that the collation degree obtained in the collation process is greater than or equal to a predetermined threshold value, the processing processor 31 proceeds to step S17 and performs a process of transmitting face history information to the face recording server 3. Do. In addition, when it is determined by the above determination that the collation degree obtained by the above collation processing is not equal to or greater than a predetermined threshold, the processor 31 determines that the user is not the person himself ("impersonation"), and Reject processing.

  Note that the matching degree used in the process of step S122 may be set to a value lower than usual. In this case, the collation process by the face image becomes “light”, and the possibility of being rejected due to erroneous recognition even though the person is the person can be made close to zero. That is, by setting a threshold value lower than usual and performing “light” collation processing, there is an advantage that the false error rejection rate becomes almost zero and user convenience is not lost.

  As described above, in the seventh embodiment, the face history information accumulated in the face recording server is clustered and classified for each user, and a template as a face pattern for each user is generated. The template and the ID code are downloaded to the input terminal. As a result, a cluster or template for each user can be created from the face history information accumulated in the face history database, saving the user from creating a template.

  Furthermore, in the seventh embodiment, a template and an ID code are registered in the input terminal, and impersonation detection by face matching is performed at the input terminal when a face image is input. Thereby, it is possible to detect impersonation at the stage of face image input processing at the input terminal.

  As described above in detail, according to each embodiment, when a user accesses an object with high security, registration of the front face is obligated and unauthorized access is suppressed. In addition, as a first model, a system that performs ID authentication based on ID information input by a user and then performs collation processing and recording of the face image of the user's face image, and input of ID information as a second model Are omitted, and two systems are assumed: a system that performs only a face image matching process and a face image recording.

  Furthermore, each model has a face image accumulated as a history and an automatic tracking (search) function for the history information. In the first model, “spoofing detection” can be performed to improve the security level. In the model, it is possible to perform “user classification” to accumulate efficient face images and their history information. Furthermore, the function of searching for a specific person can be implemented in either model and can be combined with other functions. Moreover, in the implementation form of each model, there can be both an integrated apparatus (stand-alone type) and a network type system. Furthermore, face tracking (collation processing using face images) can be performed in two ways: real-time processing or batch processing.

  As an effect of each of these embodiments, since the user is obliged to record a face facing forward, unauthorized access is compared to a conventional system for monitoring a person or a system for recording a monitoring image. The deterrent effect becomes higher. Further, convenience is improved as compared with the conventional face matching device. As a result, it is possible to provide an efficient system that balances convenience and security levels that cannot be realized by conventional apparatuses and systems, and it is possible to widen the application range of the system.

  DESCRIPTION OF SYMBOLS 1 ... ID authentication part, 2 ... Face image input device, 3 ... Face recording server, 4 ... Face history database, 4a ... Specific person list, 4 ... Face image history database, 11 ... Operation part, 12 ... Processing part, 21 ... Display unit, 22 ... Camera, 24 ... Numpad, 31 ... Processing processor, 37 ... LAN interface, 37 ... Magnetic disk, 38 ... LAN interface, 51 ... Control unit, 55 ... Display unit, 56 ... Operating unit, 57 ... LAN interface

Claims (3)

In a face image recording system having a plurality of input terminals installed at a plurality of points and a server device capable of receiving data from these input terminals,
The input terminal is
Image input means for inputting a user's face image at each installation point;
Transmission means for transmitting the face image data input by the image input means and history information including the shooting location and shooting time to the server device,
The server device
Image recording means for recording face image data and history information transmitted from the input terminal;
When face image data from the input terminal is received, collation means for collating face image data received from the input terminal and face image data already recorded in the image recording means;
When the collation means succeeds in collating the face image data received from the input terminal with the face image data already recorded in the image recording means, the face image data collated with the face image data received from the input terminal Display means for reading history information corresponding to the image recording means, and displaying a trajectory of the user's action based on the shooting location and the shooting time included in the history information,
A face image recording system characterized by that. The image recording means classifies and records the face image data and its history information for each user,
The collation means collates the user's face image data input by the image input means with the face image data of each user already recorded in the image recording means,
The display unit collates with the face image data received from the input terminal when the collation between the face image data received from the input terminal and the user face image data already recorded in the image recording unit is successful. Read the user's history information from the image recording means, and display the user's action trajectory based on the shooting location and shooting time included in the history information.
The face image recording system according to claim 1, wherein: A face image recording method of a face image recording system having a plurality of input terminals installed at a plurality of points and a server device connected to these input terminals,
An image input step of inputting a user's face image at each installation point of the input terminal;
A transmission step of transmitting the face image input by the image input step and history information including the shooting point and the shooting time to the server device;
An image recording step of recording the face image data transmitted from the input terminal and its history information in an image recording means;
A collation step of collating the face image data received from the input terminal with the face image data already recorded in the image recording means when receiving the face image data from the input terminal;
When the collation between the face image data received from the input terminal and the face image data already recorded in the image recording means is successful, the face image data collated with the face image data received from the input terminal. A display step of reading the history information corresponding to the image recording means, and displaying the locus of the user's action based on the shooting location and the shooting time included in the history information;
A face image recording method characterized by comprising:

Images