JP3999983B2 - Operator monitoring device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an operator monitoring apparatus that monitors an operator (driver) of, for example, an aircraft, a traveling vehicle, a ship, etc. by applying a personal authentication technique based on biometric information (biometrics) such as an operator's face image. About.
[0002]
[Prior art]
Conventionally, fingerprints, handprints, voiceprints, signatures, retinas, irises, etc. have been put to practical use as biometrics. These can be classified into a “contact type” that directly touches the apparatus and a “non-contact type” that does not need to be touched.
That is, among the above biometrics examples, fingerprints, handprints, and signatures belong to “contact forms”. However, signatures are somewhat different in nature from fingerprints in that they must act to sign themselves.
[0003]
On the other hand, the same biometric is voiceprint (speech) collation in that the person needs to be conscious, but this belongs to the “non-contact type”. In addition, the retina and iris can be said to belong to the “non-contact type” in that the image of the target part is captured from the camera and processed, but the retina is located behind the eyeball and the eye needs to be almost in contact with the lens. Therefore, it is close to the contact type.
Since the iris appears on the surface of the eyeball, it can be input with a camera from a distant position, but since it is a fine pattern, there is a limit to the distance that can be separated. On the other hand, non-contact biometrics that have recently attracted attention are personal identification based on face patterns obtained from face images. Since the face pattern has a larger scale than the iris pattern, it can be input from a relatively distant position, and it is also used for personal identification between humans. is there.
[0004]
[Problems to be solved by the invention]
Conventionally, there has been no effective general means for managing / preventing the change of an illegal operator (driver) of public transportation, such as an aircraft, a traveling vehicle, and a ship.
As a possible means, it is conceivable that an identification card using an IC card or the like is carried and read by an IC card reader installed in the vicinity of the driver / operator seat to automatically identify the person. The security level is not necessarily good, and the security level is not necessarily high, such as theft or the possibility of deliberate lending.
[0005]
On the other hand, in the conventional personal authentication system using a face image, a normal video camera is installed at an appropriate position, the face image is taken in front or near the front, and registered in advance under the same conditions. And pattern matching.
For example, Reference 1 (“Face Recognition System Using Moving Images”, IEICE Technical Report PRMU97-50, June 1997; Yamaguchi, Fukui, Maeda) and Reference 2 (“Face by Combination of Shape Extraction and Pattern Matching”) Feature point extraction "Science theory D-II.Vol.j-80-D-II No.8 August 1997; Fukui, Yamaguchi) A system that takes in a face image looked up and performs registration and verification has been proposed. The identity verification using the face image has an advantage that the face image of the subject person can be captured without contact, but no example of effective application to the operator confirmation has been found.
[0006]
Therefore, the present invention takes full advantage of the naturalness and convenience of the face image matching method, and greatly improves the overall security level compared to the prior art. An object of the present invention is to provide an operator monitoring device capable of performing the above.
[0007]
[Means for Solving the Problems]
  The operator monitoring device according to the present invention includes an image input unit that captures and inputs a face image of an operator who operates the operated object, and a point in time when the operation of the operated object is started by the image input unit. A face image registration unit that captures the input face image of the operator and registers the captured face image as an initial face image, and captures the face image of the operator input by the image input unit at a predetermined timing. The determination unit determines whether or not the operator has changed by collating the captured face image with the face image registered by the face image registration unit, and the determination unit has changed the operator. Is determined,The operator's face image captured at the time of determination is registered in the face image registration means as a new registered face image.Control means.
[0008]
Further, the operator monitoring device of the present invention includes an image input means for capturing and inputting a face image of an operator who operates the operated object, and the image input at the time when the operation of the operated object is started. The face image registration means for capturing the face image of the operator input by the means and registering the captured face image as an initial face image, and the operator captured when the face image is registered by the face image registration means The first confirmation that confirms whether or not the operator is a legitimate operator by comparing the face image with a face image in a legitimate person list in which the face images of legitimate operators are registered in advance. And when the first confirmation means recognizes that the operator is a valid operator, the operator's face image input by the image input means is captured at a predetermined timing, and the captured face image and the Registered by face image registration means A determination unit that determines whether or not the operator is changed by collating the face image, and an operation that is taken in at the time of determination by the determination unit when the determination unit determines that the operator has changed. A second confirmation means for confirming whether the operator is a legitimate operator by comparing the face image of the person with the face image in the right person list, and the first and second confirmation means If it is determined that the operator is not a valid operator, the control unit performs a predetermined operation.
[0009]
Further, the operator monitoring device of the present invention includes an image input means for capturing and inputting a face image of an operator who operates the operated object, and the image input at the time when the operation of the operated object is started. The face image registration means for capturing the face image of the operator input by the means and registering the captured face image as an initial face image, and the operator captured when the face image is registered by the face image registration means By comparing the face image with the face image of the unauthorized person list in which the face image of a specific unauthorized operator is registered in advance, it is confirmed whether or not the operator is a specific unauthorized operator. When the first confirmation means and the first confirmation means recognize that it is not a specific unauthorized operator, the operator's face image input by the image input means is captured at a predetermined timing, and this capture is performed. Face image and face image registration A determination unit that determines whether or not the operator has changed by collating the face image registered by the stage, and when the determination unit determines that the operator has changed, when the determination unit determines A second confirmation means for confirming whether or not the operator is a specific unauthorized operator by comparing the captured operator's face image with a face image of the unauthorized person list; And a control means for performing a predetermined operation when it is recognized by the first and second confirmation means that the operator is a specific unauthorized person.
[0010]
  Also,The operator monitoring device according to the present invention includes an image input unit that captures and inputs a face image of an operator who operates the operated object, and a point in time when the operation of the operated object is started by the image input unit. By capturing the input face image of the operator and comparing the captured face image with the face images in the valid person list in which the valid operator's face images are registered in advance, A first confirmation unit for confirming whether or not the user is an operator, and an operator input by the image input unit at a predetermined timing when the first confirmation unit recognizes that the operator is a valid operator A second confirmation means for confirming whether or not the operator is a legitimate operator by comparing the captured face image with a face image of the legitimate person list, Legitimate operation by the first and second confirmation means If it accepted without a person, and a control means for performing a predetermined operation with a predeterminedAnd when at least one of the first and second confirmation means is recognized as a valid operator as a result of collation with the face image of the valid person list, Whether the operator is a legitimate operator again by using identity verification means based on whether or not the order in which the pressure of the hand of the operator's hand is applied to the left and right of the handle of the operation target is a predetermined order. It is characterized by confirming whether or not.
  Furthermore, the operator monitoring device of the present invention includes an image input means for capturing and inputting a face image of an operator who operates the operated object, and the image input at the time when the operation of the operated object is started. A face image registration means for capturing an operator's face image input by the means and registering the captured face image as an initial face image; and an operator's face image input by the image input means at a predetermined timing. A determination unit that determines whether or not the operator has changed by collating the captured face image with the face image registered by the face image registration unit, and the determination unit allows the operator to When it is determined that the operator has changed, the operator's face image captured at the time of determination by the determination unit is registered in the face image registration unit as a new registered face image, and at least the captured operator's face image Record Or recorded in the body, or, and a control means for transmitting to the place determined in advance.
[0011]
DETAILED DESCRIPTION OF THE INVENTION
First, an outline of the present invention will be described before describing embodiments of the present invention. The present invention is roughly composed of the following five inventions (1) to (5).
(1) Detect and manage operator changes by face matching.
・ Install at least one camera (image input means) at a position and orientation where a face image can be captured when the operator is operating an object to be operated (aircraft, traveling vehicle, ship, etc.) Is also possible). In this case, the camera is provided, for example, at the center position of the handle for operating the object to be operated, or an operator console or operation panel positioned in front of the operator, and displays a face image viewed from the lower oblique direction with respect to the operator. Take an image.
[0012]
When the operation is started, the face image of the operator at that time is taken in, and the face image that has been subjected to the predetermined processing is registered as the initial face image. In this case, the operation is started when, for example, a pressure of a predetermined value or more is applied to the handle of the object to be operated, or a load of a seat on which the operator is seated is a predetermined value or more. This is when an image is taken from the camera at a fixed period and a change is detected by image analysis.
[0013]
-The operator's face image is taken in at a predetermined timing and collated with the registered face image to determine whether the operator has changed. In this case, for example, the predetermined timing is set to a constant cycle when the load on the seat on which the operator is seated once becomes zero after the load is applied again or at all times.
[0014]
・ If an operator change is detected, a predetermined operation is performed. In this case, as the predetermined operation, for example, the face image captured at that time is registered as a new face image. Alternatively, the face image captured at that time and attribute information such as time are recorded by the recording device or transmitted to a predetermined place.
(2) Operator change detection and correctness confirmation are performed by face matching.
At least one camera (image input means) is installed at a position and orientation in which a face image can be taken when the operator is operating an object to be operated (a plurality of cameras can be installed).
[0015]
When the operation is started, the face image of the operator at that time is taken in, and the face image that has been subjected to the predetermined processing is registered as the initial face image. At the same time, collation with the face images in the valid person list in which the valid operator's face images are registered in advance is performed. As a result of the confirmation, if the operator is not recognized as a valid operator, a predetermined operation is executed.
[0016]
-The operator's face image is taken in at a predetermined timing and collated with the registered face image to determine whether the operator has changed. When the change of the operator is detected, collation with the face images in the authorized person list in which the authorized operator's facial images are registered in advance is performed. As a result of the confirmation, if the operator is not recognized as a valid operator, a predetermined operation is executed.
-As an operation when the operator is not authorized, the operation of the operation target is disabled. Alternatively, notification is made to a predetermined place (for example, a monitoring center of a company that operates the apparatus).
(3) Modification type 1 of (2) above
-Combine face image verification with other identity verification means at the start of operation or at the time of operator change. In this case, as another identity verification means, the order in which the pressure of the hand of the operator's hand is applied to the left and right of the handle of the operation target (for example, the order from right to left) is used instead of the password. That is, the identity verification is performed based on whether or not the order in which the hand gripping pressure to the left and right is applied to the handle is a predetermined order. Alternatively, identity verification means based on operator's voice verification may be used.
(4) Modification type 2 of (2) above
・ An unauthorized person list in which a face image of a specific unauthorized operator is registered in advance when it is recognized that the operator is not an authorized operator instead of collating with the authorized person list or by comparing with the authorized person list Checking with a face image in a so-called black list is performed. As a result of the confirmation, if it is recognized as an unauthorized operator, a predetermined operation is executed.
(5) Deformation type 3 of (2) above
-After detecting the change of operator, it does not collate with the right person list or the fraudster list (black list) but immediately collates with the right person list or the fraudster list.
[0017]
In addition, it cannot be overemphasized that the invention of said (1)-(5) can be implemented even if it combines independently and mutually.
[0018]
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
First, the first embodiment will be described. The first embodiment corresponds to the above-described invention (1).
FIG. 1 schematically shows a configuration of an operator monitoring apparatus according to the first embodiment. In FIG. 1, a plurality of video cameras as image input means (hereinafter simply referred to as “cameras”, which may be one instead of a plurality) 1-1, 1-2,. In this embodiment, a video camera capable of monochrome photography is employed. The face image of an operator who operates an aircraft, a traveling vehicle, a ship, and the like is captured and input.
[0019]
The outputs of the cameras 1-1, 1-2,..., 1-N are connected to the inputs of the capture boards 2-1, 2-2,. That is, the video signal (analog data) 1 from the camera 1-1 is converted into digital data by the A / D conversion circuit 2-1a and then temporarily stored in the image memory 2-1b. Here, for simplification of explanation, although not shown, an A / D conversion circuit and an image memory are similarly arranged in the capture boards 2-2,. The same processing is performed for the video signals 2 to N in the capture boards 2-2,.
If the cameras 1-1, 1-2,..., 1-N have a USB (universal serial bus) interface, the A / D conversion circuit 2-1a is not necessary, and the USB interface Will be provided separately.
[0020]
The capture boards 2-1, 2-2,..., 2 -N, the recording interface 3, the processor 4, the work memory 5, the display board 6, and the dictionary 7 as registration means can communicate freely via the system bus 8. It is connected to the. A display 9 is connected to the display board 6. A plurality of dictionary data (reference face feature data) is registered (stored) in the dictionary 7.
[0021]
The processor 4 transmits a control signal for synchronization to the cameras 1-1, 1-2,..., 1-N, and the cameras 1-1, 1-2,. The digital data related to the face image sent from is acquired, and the registration and collation processing of the face image as described later is performed on the data using the work memory 5 and the dictionary 7.
[0022]
The recording interface 3 is connected to a recording device 10 such as a hard disk device as recording means. The recording device 10 records an operator's face image, face feature data, and the like on a recording medium such as a hard disk.
The system bus 8 is connected to a seat load detection device 11 as a seat load detection means. The seat load detection device 11 detects the load on the seat of the operation object on which the operator is seated. Further, a communication control unit 12 is connected to the system bus 8. The communication control unit 12 is for communicating with an external device, for example, a monitoring center of a company that operates this device.
[0023]
2 and 3 show examples of camera arrangement. FIG. 2A shows a case where the number of cameras is one. For example, the camera 1-1 is installed at the center of the handle 21 for operating the object to be operated. The position and orientation are set so as to capture the seen face image. 2B is a view of the operator's head 22 viewed from directly above, and FIG. 2C is a view of the operator's head 22 viewed from the side.
FIG. 3 shows a case where there are a plurality of cameras. For example, the cameras 1-1 and 1-2 are installed on the operation panel 23 located in front of the operator, and the camera is viewed from the lower oblique direction with respect to the operator's head. It captures the face image it sees.
FIG. 4 shows an example of an operator's face image captured by the camera 1-1 of FIG. 2, for example, and it can be seen that a face image obtained by viewing the operator's face from an obliquely downward direction is captured.
[0024]
Hereinafter, with reference to the flowchart shown in FIG. 5, a face image registration process performed by the processing processor 4 of the operator monitoring apparatus according to the first embodiment will be described. In the following description, for example, one camera is installed as shown in FIG.
First, the processor 4 issues a management number (step S1). Next, the input image from the camera 1-1 is taken in (step S2). Next, a face area is detected from the entire input image (step S3). Next, in the detected face area, a pupil area and a nostril area that can be regarded as a substantially circular area are detected, and the center position of the area is set as a feature point of the face image (step S4). In the processes of steps S3 and S4, for example, the conventional method described in the above-mentioned document 2 can be adopted.
[0025]
Next, the collation area is cut out based on the position of the detected feature point (step S5), the size is normalized by geometric correction (step S6), and the density distribution is normalized by density correction (step S6). Step S7). Next, vector pattern expression by feature extraction is performed and registered in the dictionary 7 as face feature data (hereinafter also referred to as dictionary data) (step 8). In this case, face feature data is registered together with attribute information such as a management number issued previously.
[0026]
Next, with reference to the flowchart shown in FIG. 6, the face image matching process by the processing processor 4 of the operator monitoring apparatus according to the first embodiment will be described. In the following description, for example, one camera is installed as shown in FIG.
First, the processor 4 captures an input image from the camera 1-1 (step S11). Next, a face area is detected from the entire input image (step S12). Next, it is determined whether or not the face area has been successfully detected (step S13). If the face area is successfully detected, a pupil area and a nostril area that can be regarded as a substantially circular area are detected in the detected face area. The center position is set as a feature point of the face image (step S14).
[0027]
Next, the matching region is cut out based on the detected feature point position (step S15), the size is normalized by geometric correction (step S16), and the density distribution is normalized by density correction (step S16). Step S17). Next, vector pattern expression by feature extraction is performed and collated with the dictionary data of the dictionary 7 as input face feature data (step 18). That is, the similarity is calculated by collating the input facial feature data with a plurality of facial feature data registered in the dictionary 7, and the obtained similarity is larger than a predetermined threshold value set in advance. In this case, collation is successful (legitimate operator), and other cases are collation failure (invalid operator).
[0028]
Next, the flow of the operator change detection process using the registration and verification process described above will be described with reference to the flowchart shown in FIG. In the following description, for example, one camera 1-1 is installed at the center of the handle 21 as shown in FIG. The processing described below is mainly performed by an operation based on the processing program of the processing processor 4.
[0029]
First, by referring to the output of the seat load detection device 11, whether or not the seat load of the operation object on which the operator is seated has exceeded a predetermined value (that is, whether or not the operation of the operation object has started). Is determined (step S21). As a result of this determination, if the seat load is greater than or equal to a predetermined value, it is considered that the operator is seated and the operation of the operated object is started, and input from the camera 1-1 installed at the center portion of the handle 21 An image is captured, and facial feature data is extracted from the captured input image by the same operation as described above, and the extracted facial feature data is registered in the dictionary 7 as an initial registered facial image (step S22).
[0030]
Next, the input image (face image) at this time and the facial feature data extracted from the input image are recorded together with attribute information such as the date / time data by the recording device 10 or predetermined by the communication control unit 12. (For example, a monitoring center of a company that operates this apparatus) is transmitted (step S23).
[0031]
Next, by referring to the output of the seat load detection device 11, it is determined whether or not the seat load has become equal to or less than a predetermined value (that is, whether or not the operation of the operation target has ended) (step). S24). As a result of this determination, when the load on the seat becomes equal to or less than a predetermined value, it is considered that the operator has finished the operation and left the seat, and the process ends here.
[0032]
As a result of the determination in step S24, if the seat load is not less than or equal to the predetermined value, it is assumed that the operator is still seated and the operation has not been completed, and the output of the seat load detection device 11 is referred to. Then, it is determined whether or not the seat load is once zeroed again (that is, whether or not the operator has changed) (step S25). If the result of this determination is that the seat load is once zero and it is not in a state where it has been reapplied, it is assumed that the operator has not changed, returning to step S24, and repeating the same operation as above.
[0033]
As a result of the determination in step S25, when the seat load is once zero and then reapplied, the operator assumes that the operator is changing, and takes the input image from the camera 1-1 again. Face feature data is extracted from the captured input image by the same operation as described above, and the extracted face feature data and a plurality of dictionary data (face feature data) registered in the dictionary 7 are collated. Is calculated (step S26).
[0034]
Next, it is determined whether or not the degree of similarity calculated in step S26 is greater than a predetermined threshold value set in advance (step S27). Therefore, the process returns to step S24 to repeat the same operation as described above.
[0035]
As a result of the determination in step S27, if the calculated similarity is smaller than the set threshold value, it is considered that the operator has finally changed. For example, the facial feature obtained from the face image captured at the time of collation in step S26 The data is registered in the dictionary 7 as new dictionary data, and the input image (face image) and face feature data are recorded together with attribute information such as the date / time data by the recording device 10 or by the communication control unit 12. The data is transmitted to a predetermined place (for example, a monitoring center of a company that operates this apparatus) (step S28), the process returns to step S24, and the same operation as described above is repeated.
As described above, according to the first embodiment, it is possible to detect and manage the change of the operator by collating the face of the operator.
[0036]
Next, a second embodiment will be described. The second embodiment corresponds to the above-described invention (2).
In the second embodiment, operator change detection and correctness confirmation are performed by face matching similar to the first embodiment, and the apparatus configuration may be the same as in the first embodiment. The following describes this with reference to the flowcharts shown in FIGS. 8 and 9.
[0037]
In the following description, for example, one camera 1-1 is installed at the center of the handle 21 as shown in FIG. In the present embodiment, it is assumed that all legitimate operators are registered (stored) in the work memory 5 in the form of a legitimate person list. Further, the processing described below is mainly performed by an operation based on the processing program of the processing processor 4.
[0038]
First, by referring to the output of the seat load detection device 11, whether or not the seat load of the operation object on which the operator is seated has exceeded a predetermined value (that is, whether or not the operation of the operation object has started). Is determined (step S31). As a result of this determination, if the seat load is greater than or equal to a predetermined value, it is considered that the operator is seated and the operation of the operated object is started, and input from the camera 1-1 installed at the center portion of the handle 21 An image is captured, and facial feature data is extracted from the captured input image by the same operation as described above, and the extracted facial feature data is registered in the dictionary 7 as an initial registered facial image (step S32).
[0039]
Next, the input image (face image) at this time and the facial feature data extracted from the input image are recorded together with attribute information such as the date / time data by the recording device 10 or predetermined by the communication control unit 12. (For example, it is transmitted to the monitoring center of a company that operates this apparatus) (step S33).
[0040]
Next, the face feature data obtained in step S32 is collated with each face feature data in the legitimate person list in which the face feature data of all legitimate operators are registered, and the similarity between them is calculated ( Step S34). Next, it is determined whether or not face feature data whose calculated similarity is larger than a predetermined threshold value set in advance exists in the valid person list (that is, whether the operator is a valid operator). (Step S35).
[0041]
If the result of this determination is that there is no such operator, the operator is deemed not to be a legitimate operator, and control is performed so that the operation (operation) of the object to be operated is disabled. Is notified to the monitoring center of the company that operates the system (step S36), and the process ends here.
[0042]
As a result of the determination in step S35, when face feature data having a calculated similarity greater than a predetermined threshold is present in the legal person list, the operator is regarded as an authorized operator, and the seat load detection device is again detected. 11 is referred to, it is determined whether or not the load on the seat has become equal to or less than a predetermined value (that is, whether or not the operation of the operation target has ended) (step S37). As a result of this determination, when the load on the seat becomes equal to or less than a predetermined value, it is considered that the operator has finished the operation and has left the seat, and the process ends here.
[0043]
As a result of the determination in step S37, if the seat load is not less than or equal to the predetermined value, it is assumed that the operator is still seated and the operation is not finished, and the output of the seat load detection device 11 is referred to. Then, it is determined whether or not the seat load is once zeroed again (that is, whether or not the operator has changed) (step S38). If the result of this determination is that the seat load is once zero and it is not in a state where it has been reapplied, the operator assumes that the seat has not been replaced, and returns to step S37 to repeat the same operation as above.
[0044]
If the result of determination in step S38 is that the seat load has once been zero and then reapplied, the operator assumes that the operator is changing, and takes the input image from the camera 1-1 again. Face feature data is extracted from the captured input image by the same operation as described above, and the extracted face feature data and a plurality of dictionary data (face feature data) registered in the dictionary 7 are collated. Is calculated (step S39).
[0045]
Next, it is determined whether or not the similarity calculated in step S39 is greater than a predetermined threshold value set in advance (step S40). Assuming that there is, return to step S37 and repeat the same operation as above.
[0046]
As a result of the determination in step S40, if the calculated similarity is smaller than the set threshold value, it is considered that the operator has finally changed, for example, the facial feature obtained from the face image captured at the time of collation in step S39. The data is registered in the dictionary 7 as new dictionary data, and the input image (face image) and face feature data are recorded together with attribute information such as the date / time data by the recording device 10 or by the communication control unit 12. The data is transmitted to a predetermined place (for example, a monitoring center of a company that operates this apparatus) (step S41).
[0047]
Next, the face feature data obtained in step S39 is collated with each face feature data in the valid person list, and the similarity between the two is calculated (step S42). Next, it is determined whether or not face feature data whose calculated similarity is larger than a predetermined threshold value set in advance exists in the valid person list (that is, whether the operator is a valid operator). (Step S43).
[0048]
If the result of this determination is that there is no such operator, the operator is deemed not to be a legitimate operator, and control is performed so that the operation (operation) of the operation target is disabled, and the monitoring center of the company that operates the apparatus is controlled. The fact that an unauthorized operator has been detected is notified (step S44), and the process ends here.
[0049]
As a result of the determination in step S43, when face feature data whose calculated similarity is larger than a predetermined threshold exists in the right person list, the operator is regarded as a right operator, and the process returns to step S37. The same operation as above is repeated.
As described above, according to the second embodiment, it is possible to detect the change of the operator and confirm the validity by comparing the face of the operator.
[0050]
Next, a third embodiment will be described. The third embodiment corresponds to the invention of (3) described above.
In the third embodiment, the identity verification function is further enhanced with respect to the second embodiment, and the apparatus configuration may be basically the same as that of the first embodiment. The point that the handle is provided with a pressure detector and the flow of processing are different, which will be described below.
[0051]
In the third embodiment, for example, as shown in FIG. 10, pressure detectors 24a and 24b are respectively embedded in left and right portions that are held by the operator of the handle 21 of the object to be operated. The left and right pressures when the handle 21 is grasped with both hands are detected.
[0052]
The processing flow will be described below with reference to the flowcharts shown in FIGS.
First, by referring to the output of the seat load detection device 11, whether or not the seat load of the operation object on which the operator is seated has exceeded a predetermined value (that is, whether or not the operation of the operation object has started). Is determined (step S51). As a result of this determination, if the seat load is greater than or equal to a predetermined value, it is considered that the operator is seated and the operation of the operated object is started, and input from the camera 1-1 installed at the center portion of the handle 21 An image is captured, face feature data is extracted from the captured input image by the same operation as described above, and the extracted face feature data is registered in the dictionary 7 as an initial registered face image (step S52).
[0053]
Next, the input image (face image) at this time and the facial feature data extracted from the input image are recorded together with attribute information such as the date / time data by the recording device 10 or predetermined by the communication control unit 12. (For example, it is transmitted to a monitoring center of a company that operates this apparatus) (step S53).
[0054]
Next, the face feature data obtained in step S32 is collated with each face feature data in the legitimate person list in which the face feature data of all legitimate operators are registered, and the similarity between them is calculated ( Step S54). Next, it is determined whether or not face feature data whose calculated similarity is larger than a predetermined threshold value set in advance exists in the valid person list (that is, whether the operator is a valid operator). (Step S55).
[0055]
If the result of this determination is that there is no such operator, the operator is deemed not to be a legitimate operator, and control is performed so that the operation (operation) of the object to be operated is disabled. Is notified to the monitoring center of the company that operates the system (step S56), and the process ends here.
[0056]
As a result of the determination in step S55, when face feature data whose calculated similarity is larger than a predetermined threshold exists in the valid person list, for example, a predetermined operation is performed on the handle 21 by voice guidance or guidance message display. (Step S57). By this guidance, if it is a legitimate operator, a predetermined operation taught in advance, for example, first strongly squeezes the part of the pressure detector 24a with the right hand and then strongly squeezes the part of the pressure detector 24b with the left hand. Perform the operation.
[0057]
Next, by referring to the outputs of the pressure detectors 24a and 24b, it is determined whether or not the addition order of the pressure held by the operator's hand to the left and right with respect to the handle 21 is a predetermined correct order (step). S58). As a result of this determination, if the order of adding pressure to grip is not the correct order, the operator is regarded as not a valid operator, and control is performed so that the operation (operation) of the object to be operated is disabled. The fact that an unauthorized operator has been detected is notified to the monitoring center of the company that operates the apparatus (step S56), and the process ends here.
[0058]
As a result of the determination in step S58, if the order in which the gripping pressure is applied is the correct order, the operator is regarded as a valid operator, and by referring to the output of the seat load detection device 11 again, It is determined whether or not the load has become equal to or less than a predetermined value (that is, whether or not the operation of the operation target has ended) (step S59). As a result of this determination, when the load on the seat becomes equal to or less than a predetermined value, it is considered that the operator has finished the operation and left the seat, and the process ends here.
[0059]
As a result of the determination in step S59, if the seat load is not less than or equal to the predetermined value, it is assumed that the operator is still seated and the operation is not finished, and the output of the seat load detection device 11 is referred to. Then, it is determined whether or not the seat load is once added again after being zero (that is, whether or not the operator has changed) (step S60). If the result of this determination is that the seat load is once zero and it is not in a state where it has been reapplied, it is assumed that the operator has not changed, and the flow returns to step S59 to repeat the same operation as described above.
[0060]
If the result of the determination in step S60 is that the seat load has once been zero and then reapplied, the operator assumes that the seat has been replaced and re-captures the input image from the camera 1-1. Face feature data is extracted from the captured input image by the same operation as described above, and the extracted face feature data and a plurality of dictionary data (face feature data) registered in the dictionary 7 are collated. Is calculated (step S61).
[0061]
Next, it is determined whether or not the similarity calculated in step S61 is greater than a predetermined threshold value set in advance (step S62). Assuming that there is, return to step S59 and repeat the same operation as above.
[0062]
As a result of the determination in step S62, if the calculated similarity is smaller than the set threshold value, it is considered that the operator has finally changed. For example, the facial feature obtained from the face image captured at the time of collation in step S61 The data is registered in the dictionary 7 as new dictionary data, and the input image (face image) and face feature data are recorded together with attribute information such as the date / time data by the recording device 10 or by the communication control unit 12. The data is transmitted to a predetermined place (for example, a monitoring center of a company that operates this apparatus) (step S63).
[0063]
Next, the face feature data obtained in step S61 is collated with each face feature data in the legal person list, and the similarity between the two is calculated (step S64). Next, it is determined whether or not face feature data whose calculated similarity is larger than a predetermined threshold value set in advance exists in the valid person list (that is, whether the operator is a valid operator). (Step S65).
[0064]
If the result of this determination is that there is no such operator, the operator is deemed not to be a legitimate operator, and control is performed so that the operation (operation) of the operation target is disabled, and the monitoring center of the company that operates the apparatus is controlled. The fact that an unauthorized operator has been detected is notified (step S66), and the process ends here.
[0065]
As a result of the determination in step S65, when face feature data whose calculated similarity is larger than a predetermined threshold is present in the legal person list, a predetermined operation is performed on the handle 21 by displaying voice guidance or a guidance message. Guidance is made (step S67).
[0066]
Next, by referring to the outputs of the pressure detectors 24a and 24b, it is determined whether or not the addition order of the pressure held by the operator's hand to the left and right with respect to the handle 21 is a predetermined correct order (step). S68). As a result of this determination, if the order of adding pressure to grip is not the correct order, the operator is regarded as not a valid operator, and control is performed so that the operation (operation) of the object to be operated is disabled. The fact that an unauthorized operator has been detected is notified to the monitoring center of the company that operates the apparatus (step S66), and the process ends here.
[0067]
As a result of the determination in step S68, if the order in which the gripping pressure is applied is the correct order, the operator is regarded as a valid operator, and the process returns to step S59 to repeat the same operation as described above.
As described above, according to the third embodiment, at the start of the operation and when the operator is changed, the identification function can be further enhanced by combining the collation of the face image with other identification means. It becomes possible.
[0068]
The other person confirmation means is not limited to the order in which the pressure of the hand of the operator's hand is applied to the left and right with respect to the handle of the object to be operated as described above. For example, as shown in FIG. A microphone 25 as a voice input means is embedded in the center of the voice, and guidance for inputting a predetermined code is made by voice by displaying voice guidance or a guidance message. Voice identification) may be a means for verifying the identity by collation.
[0069]
Next, a fourth embodiment will be described. The fourth embodiment corresponds to the invention of (4) described above.
In the fourth embodiment, verification with a fraudulent list (so-called black list) is performed instead of verification with a legitimate list in the second embodiment. Although it may be the same as that of the first embodiment, the flow of processing is different, which will be described below with reference to the flowcharts shown in FIGS.
In this embodiment, it is assumed that all unauthorized operators are registered (stored) in the work memory 5 in the form of an unauthorized person list.
[0070]
First, by referring to the output of the seat load detection device 11, whether or not the seat load of the operation object on which the operator is seated has exceeded a predetermined value (that is, whether or not the operation of the operation object has started). Is determined (step S71). As a result of this determination, if the seat load is greater than or equal to a predetermined value, it is considered that the operator is seated and the operation of the operated object is started, and input from the camera 1-1 installed at the center portion of the handle 21 An image is fetched, face feature data is extracted from the fetched input image by the same operation as described above, and the extracted face feature data is registered in the dictionary 7 as an initial registered face image (step S72).
[0071]
Next, the input image (face image) at this time and the facial feature data extracted from the input image are recorded together with attribute information such as the date / time data by the recording device 10 or predetermined by the communication control unit 12. (For example, the monitoring center of a company that operates the apparatus) is transmitted (step S73).
[0072]
Next, the facial feature data obtained in step S72 is collated with each facial feature data in the unauthorized person list in which the facial feature data of a specific unauthorized operator is registered, and the similarity between them is calculated ( Step S74). Next, it is determined whether or not face feature data whose calculated similarity is larger than a predetermined threshold set in advance exists in the unauthorized person list (that is, whether or not the operator is an unauthorized operator). (Step S75).
[0073]
If the result of this determination is that the operator is present, the operator is regarded as an unauthorized operator and is controlled so that the operation (operation) of the operation target is disabled. The fact that an unauthorized operator has been detected is notified to the monitoring center of the company that operates the apparatus (step S76), and the process ends here.
[0074]
As a result of the determination in step S75, when face feature data whose calculated similarity is larger than a predetermined threshold value does not exist in the legal person list, the operator is regarded as an authorized operator, and the seat load detection device is again detected. 11 is referred to, it is determined whether or not the seat load has become equal to or less than a predetermined value (that is, whether or not the operation of the operation target has ended) (step S77). As a result of this determination, when the load on the seat becomes equal to or less than a predetermined value, it is considered that the operator has finished the operation and left the seat, and the process ends here.
[0075]
As a result of the determination in step S77, if the seat load is not less than or equal to the predetermined value, it is assumed that the operator is still seated and the operation is not completed, and the output of the seat load detection device 11 is referred to. Then, it is determined whether or not the seat load is once added again after the seat load becomes zero (that is, whether or not the operator has changed) (step S78). If the result of this determination is that the seat load is once zero and it is not in a state where it has been reapplied, it is assumed that the operator has not changed, returning to step S77 and repeating the same operation as above.
[0076]
As a result of the determination in step S78, if the seat load is once zero and then reapplied, the operator assumes that the operator is changing and takes in the input image from the camera 1-1 again. Face feature data is extracted from the captured input image by the same operation as described above, and the extracted face feature data and a plurality of dictionary data (face feature data) registered in the dictionary 7 are collated. Is calculated (step S79).
[0077]
Next, it is determined whether or not the degree of similarity calculated in step S79 is greater than a predetermined threshold value set in advance (step S80). Assuming that there is, return to step S77 to repeat the same operation as above.
[0078]
As a result of the determination in step S80, if the calculated similarity is smaller than the set threshold value, it is considered that the operator has finally changed. For example, the facial feature obtained from the face image captured at the time of collation in step S79. The data is registered in the dictionary 7 as new dictionary data, and the input image (face image) and face feature data are recorded together with attribute information such as the date / time data by the recording device 10 or by the communication control unit 12. The data is transmitted to a predetermined place (for example, a monitoring center of a company that operates this apparatus) (step S81).
[0079]
Next, the face feature data obtained in step S79 is collated with each face feature data in the unauthorized person list, and the similarity between both is calculated (step S82). Next, it is determined whether or not face feature data whose calculated similarity is larger than a predetermined threshold set in advance exists in the unauthorized person list (that is, whether or not the operator is an unauthorized operator). (Step S83).
[0080]
If the result of this determination is that the operator is present, the operator is regarded as an unauthorized operator, and is controlled so that the operation (operation) of the operation target is disabled, and the monitoring center of the company that operates the apparatus Is notified that an unauthorized operator has been detected (step S84), and the process ends here.
[0081]
As a result of the determination in step S83, if face feature data having a calculated similarity greater than a predetermined threshold does not exist in the unauthorized person list, the operator is regarded as a valid operator, and the process returns to step S77. The same operation as above is repeated.
As described above, according to the fourth embodiment, it is possible to detect and eliminate unauthorized operators by collating with the black list.
[0082]
Next, a fifth embodiment will be described. The fifth embodiment corresponds to the invention of (5) described above.
The fifth embodiment, as in the second embodiment, does not collate with the rightful person list after detecting the change of the operator, but immediately collates with the rightful person list. The configuration may be the same as that of the first embodiment, but the flow of processing is different, which will be described below with reference to the flowchart shown in FIG.
[0083]
First, by referring to the output of the seat load detection device 11, whether or not the seat load of the operation object on which the operator is seated has exceeded a predetermined value (that is, whether or not the operation of the operation object has started). Is determined (step S91). As a result of this determination, if the seat load is greater than or equal to a predetermined value, it is considered that the operator is seated and the operation of the operated object is started, and input from the camera 1-1 installed at the center portion of the handle 21 An image is captured, and facial feature data is extracted from the captured input image by the same operation as described above, and the extracted facial feature data is stored in the authorized person list in which facial feature data of all authorized operators are registered. Each face feature data is collated, and the similarity between the two is calculated (step S92).
[0084]
Next, it is determined whether or not face feature data whose calculated similarity is larger than a predetermined threshold value set in advance exists in the valid person list (that is, whether the operator is a valid operator). (Step S93). If the result of this determination is that there is no such operator, the operator is deemed not to be a legitimate operator, and control is performed so that the operation (operation) of the object to be operated is disabled. Is notified to the monitoring center of the company that operates the system (step S94), and the process ends here.
[0085]
As a result of the determination in step S93, when face feature data whose calculated similarity is larger than a predetermined threshold is present in the right person list, the operator is regarded as a right operator, and the seat load detection device is again detected. 11 is referred to, it is determined whether or not the load on the seat has become equal to or less than a predetermined value (that is, whether or not the operation of the operation target has ended) (step S95). As a result of this determination, when the load on the seat becomes equal to or less than a predetermined value, it is considered that the operator has finished the operation and left the seat, and the process ends here.
[0086]
As a result of the determination in step S95, if the seat load is not less than or equal to the predetermined value, it is assumed that the operator is still seated and the operation is not finished, and the output of the seat load detection device 11 is referred to. Then, it is determined whether or not the seat load is once zeroed again (that is, whether or not the operator has changed) (step S96). If the result of this determination is that the seat load is once zero and it is not in a state where it has been reapplied, it is assumed that the operator has not changed, returning to step S95 and repeating the same operation as described above.
[0087]
If the result of determination in step S96 is that the seat load has once been zero and then reapplied, the operator assumes that the operator is changing and takes the input image from the camera 1-1 again. Face feature data is extracted from the captured input image by the same operation as described above, and the extracted face feature data and a plurality of dictionary data (face feature data) registered in the dictionary 7 are collated. Is calculated (step S97).
[0088]
Next, it is determined whether or not the degree of similarity calculated in step S97 is greater than a predetermined threshold value set in advance (step S98). Assuming that there is, return to step S95 and repeat the same operation as above.
[0089]
If the result of determination in step S98 is that the calculated similarity is smaller than the set threshold value, it is considered that the operator has been changed to an unauthorized operator, and the process proceeds to step S94. After performing the same process as described above, the process ends. .
As described above, according to the fifth embodiment, it is possible to immediately collate with the legitimate list instead of collating with the legitimate list after detecting the change of the operator.
[0090]
Next, a sixth embodiment will be described. The sixth embodiment corresponds to the invention of (4) described above.
In the sixth embodiment, in the fourth embodiment, when it is determined that the operator is not a legitimate operator by collation with the legitimate person list, the collation confirmation with the fraudster list is performed. The apparatus configuration may be the same as that of the first embodiment, but a verification process with a legitimate list is added, which will be described below with reference to the flowcharts shown in FIGS. 16 and 17.
FIGS. 16 and 17 are the same as FIGS. 13 and 14 except that collation processing with the legitimate list is added to FIGS. 13 and 14. Only the matching process) will be described.
[0091]
When the process of step S73 is completed, the face feature data obtained in step S72 is collated with each face feature data in the legitimate person list in which the face feature data of all legitimate operators are registered. The degree is calculated (step S85). Next, it is determined whether or not face feature data whose calculated similarity is larger than a predetermined threshold value set in advance exists in the valid person list (that is, whether the operator is a valid operator). (Step S86).
[0092]
As a result of this determination, when face feature data whose calculated similarity is larger than a predetermined threshold is present in the valid person list, the operator is regarded as a valid operator, and the process returns to step S77 and the same as above. Repeat the operation.
As a result of the determination in step S86, if face feature data having a calculated similarity greater than a predetermined threshold value does not exist in the valid person list, the operator is regarded as not a valid operator, and the process proceeds to step S74. Check against the user list.
[0093]
When the processing in step S81 is completed, the face feature data obtained in step S79 is collated with each face feature data in the legitimate person list in which the face feature data of all legitimate operators are registered. Is calculated (step S87). Next, it is determined whether or not face feature data whose calculated similarity is larger than a predetermined threshold value set in advance exists in the valid person list (that is, whether the operator is a valid operator). (Step S88).
[0094]
As a result of this determination, when face feature data whose calculated similarity is larger than a predetermined threshold is present in the valid person list, the operator is regarded as a valid operator, and the process returns to step S77 and the same as above. Repeat the operation.
As a result of the determination in step S88, if face feature data having a calculated similarity greater than a predetermined threshold does not exist in the valid person list, the operator is regarded as not a valid operator, and the process proceeds to step S82. Check against the user list.
[0095]
As described above, according to the sixth embodiment, when it is determined that the operator is not a legitimate operator by collation with the legitimate person list, the collation with the black list is performed to further increase the identity verification function. Strengthened.
[0096]
Next, a seventh embodiment will be described. The seventh embodiment corresponds to the invention of (5) described above.
In the seventh embodiment, verification with a fraud list is performed instead of verification with a legitimate list in the fifth embodiment. The apparatus configuration is the same as that of the first embodiment. The process flow is the same as that shown in FIG. 18, except that step S92 for performing the matching process with the rightful person list in FIG. 15 is changed to step S99 for performing the matching process with the unauthorized person list. Other than that, since it is the same as that of FIG.
As described above, according to the seventh embodiment, it is possible to immediately collate with the unauthorized person list instead of collating with the unauthorized person list after detecting the change of the operator.
[0097]
As described above, according to the above-described embodiment, the security level as a whole is improved by making the most of the advantages of the face image pattern matching method (the burden on the user is very small) and the convenience. It becomes possible to greatly improve. Then, by taking advantage of the non-contact property of the face image recognition method, it is possible to perform operator change management, validity confirmation, fraud detection, and the like.
[0098]
In the above embodiment, a monochrome camera is used as an image input means for inputting a face image. However, a color camera may be used. In this case, a process for converting the face image into a monochrome image is added. That's fine.
In addition, although one face image (still image) is a processing target, a plurality of continuous face images (moving images) are captured at regular time intervals in consideration of fluctuations in input conditions, and the capture shown in FIG. It may be buffered in the board image memory.
[0099]
When using a moving image, there are two possible cases: a case where it is used only during registration and a case where it is used during both registration and verification. As a processing flow when processing a moving image, it is only necessary to make changes in dictionary registration and dictionary collation. Statistical processing such as principal component analysis is performed from a plurality of feature vectors, and pattern registration and collation processing is performed. To do. This processing may be a conventional method, and for example, the processing method shown in Document 2 can be applied.
[0100]
【The invention's effect】
As described above in detail, according to the present invention, the advantage of the naturalness and convenience of the face image matching method is utilized to the maximum, the overall security level is greatly improved as compared with the prior art, and the change management and validity of the operator are improved. It is possible to provide provision of an operator monitoring device that can perform confirmation, detection of unauthorized persons, and the like.
[Brief description of the drawings]
FIG. 1 is a block diagram schematically showing a configuration of an operator monitoring apparatus according to a first embodiment.
FIG. 2 is a schematic diagram illustrating an arrangement example in the case of one camera.
FIG. 3 is a schematic diagram illustrating an arrangement example when there are a plurality of cameras.
FIG. 4 is a diagram showing an example of an operator's face image captured by a camera.
FIG. 5 is a flowchart for explaining face image registration processing by a processing processor;
FIG. 6 is a flowchart for explaining face image collation processing by a processing processor;
FIG. 7 is a flowchart for explaining the flow of operator change detection processing according to the first embodiment;
FIG. 8 is a flowchart for explaining a flow of processing for detecting an operator change and checking a validity according to the second embodiment;
FIG. 9 is a flowchart for explaining a flow of processing for detecting an operator change and confirming validity according to the second embodiment;
FIG. 10 is a schematic diagram showing an arrangement example of pressure detectors in the third embodiment.
FIG. 11 is a flowchart for explaining a flow of processing for detecting an operator change and checking a validity according to the third embodiment;
FIG. 12 is a flowchart for explaining a flow of processing for performing an operator change detection and a validity check according to the third embodiment.
FIG. 13 is a flowchart for explaining the flow of processing for performing operator change detection and correctness confirmation according to the fourth embodiment;
FIG. 14 is a flowchart for explaining a flow of processing for detecting an operator change and confirming validity according to the fourth embodiment;
FIG. 15 is a flowchart for explaining the flow of processing for detecting an operator change and confirming validity according to a fifth embodiment;
FIG. 16 is a flowchart for explaining a flow of processing for detecting an operator change and checking a validity according to the sixth embodiment;
FIG. 17 is a flowchart for explaining a flow of processing for detecting an operator change and a validity check according to the sixth embodiment;
FIG. 18 is a flowchart for explaining a flow of processing for detecting an operator change and confirming validity according to the seventh embodiment;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1,1-1 to 1-N ... Camera (image input means), 2, 2-1 to 2-N ... Capture board, 4 ... Processing processor, 5 ... Work memory, 7 ... Dictionary (registration means), 9 ... Display 10, recording device (recording means) 11 seat load detection device (seat load detection means) 12 communication control unit 21 handle 22 operator head 23 operation panel 24a, 24b ... pressure detector (pressure detection means), 25 ... microphone (voice input means).

Claims (5)

An image input means for capturing and inputting a face image of an operator who operates the object;
At the time when the operation of the object to be operated is started, a face image registration unit that captures the face image of the operator input by the image input unit at that time, and registers the captured face image as an initial face image;
An operator's face image input by the image input means is captured at a predetermined timing, and the operator is changed by comparing the captured face image with the face image registered by the face image registration means. Determination means for determining whether or not,
When it is determined by the determination means that the operator has changed, a control means for registering the face image of the operator captured at the time of determination in the face image registration means as a new registered face image ;
An operator monitoring device comprising: An image input means for capturing and inputting a face image of an operator who operates the object;
At the time when the operation of the object to be operated is started, a face image registration unit that captures the face image of the operator input by the image input unit at that time, and registers the captured face image as an initial face image;
The operator's face image captured at the time of registration of the face image by the face image registration means is collated with the face images in the legitimate person list in which the face images of the legitimate operators are registered in advance. Is a first confirmation means for confirming whether or not the user is a valid operator;
When it is recognized by the first confirmation means that the operator is a legitimate operator, the operator's face image input by the image input means is captured at a predetermined timing, and the captured face image and the face image registration are captured. Determining means for determining whether or not the operator has changed by collating the face image registered by the means;
When it is determined by the determination means that the operator has been changed, the operator is authorized by comparing the face image of the operator captured at the time of determination by the determination means with the face image of the valid person list. A second confirmation means for confirming whether or not the user is an operator;
Control means for performing a predetermined operation when it is determined by the first and second confirmation means that the operator is not a valid operator;
An operator monitoring device comprising: An image input means for capturing and inputting a face image of an operator who operates the object;
At the time when the operation of the object to be operated is started, a face image registration unit that captures the face image of the operator input by the image input unit at that time, and registers the captured face image as an initial face image;
By comparing the face image of the operator captured at the time of registration of the face image by the face image registration unit with the face image of the unauthorized person list in which the face image of a specific unauthorized operator is registered in advance, First confirmation means for confirming whether or not the operator is a specific unauthorized operator;
When it is determined by the first confirmation means that the operator is not a specific unauthorized operator, the face image of the operator input by the image input means is captured at a predetermined timing, and the captured face image and the face image are captured. A determination unit that determines whether or not the operator has changed by collating the face image registered by the registration unit;
When it is determined by the determination means that the operator has been changed, the operator is identified by comparing the face image of the operator captured at the time of determination by the determination means with the face image of the unauthorized person list. A second confirmation means for confirming whether or not the user is an unauthorized operator;
Control means for performing a predetermined predetermined operation when the first and second confirmation means are recognized as a specific unauthorized operator; and
An operator monitoring device comprising: An image input means for capturing and inputting a face image of an operator who operates the object;
When the operation of the object to be operated is started, the operator's face image input by the image input means is captured, and the captured face image is registered in advance as a legitimate operator's face image. A first confirmation means for confirming whether or not the operator is a valid operator by collating with a face image of a valid person list,
When it is recognized by the first confirmation means that the operator is a valid operator, the face image of the operator input by the image input means is taken at a predetermined timing, and the taken face image is taken as the right person list. A second confirmation means for confirming whether or not the operator is a legitimate operator by collating with the face image of
Control means for performing a predetermined operation when it is determined by the first and second confirmation means that the operator is not a legitimate operator ;
If at least one of the first confirmation means and the second confirmation means is recognized as a valid operator as a result of collation with the face image in the valid person list, the confirmation means Confirm again whether or not the operator is a legitimate operator by using identity verification means based on whether or not the order of pressure applied by the operator's hand to the left and right of the handle is a predetermined order. An operator monitoring device characterized by: An image input means for capturing and inputting a face image of an operator who operates the object;
At the time when the operation of the object to be operated is started, a face image registration unit that captures the face image of the operator input by the image input unit at that time, and registers the captured face image as an initial face image;
An operator's face image input by the image input means is captured at a predetermined timing, and the operator is changed by comparing the captured face image with the face image registered by the face image registration means. Determination means for determining whether or not,
When it is determined by the determination means that the operator has changed, the operator's face image captured at the time of determination by the determination means is registered in the face image registration means as a new registered face image, and at least the capture is performed. Control means for recording the face image of a rare operator on a recording medium or transmitting it to a predetermined place;
An operator monitoring device comprising:

Images