JP2008112222A - Operation display device, operation display program, and recording medium with the same recorded - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an operation display device that easily saves operators' face images as well as the operators' operation history. <P>SOLUTION: When an operator makes a predetermined switch operation on a screen displayed on a programmable display 1, an HMI (Human Machine Interface) processing part 8 instructs a face log processing part 9 to log the operator's face image. In the instructed face log processing part 9, a face detection part 92b detects a full face image from the operator's image captured by a camera 15 and stores it in a file in a predetermined face image database 98. The face detection part 92 further writes to a shared memory 94 the pathname of the file in the face image database 98 as well as information on an operation history such as the switch operation, the face detection date/time and the screen number of the switch operation screen. A face logging part 92c saves the data written to the shared memory 94 in a face log file 95 when instructed by a face log control part 92a. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an operation display device that accepts input operations and displays a screen, such as a programmable display, and more particularly to an operation display device that can store an operator's history.

  An HMI (Human Machine Interface) device such as a programmable display has an interface with an operation target device (PLC, inverter, temperature controller, etc.), displays the operation status of the operation target device, or operates the operation target device. This is an operation display device having a function of accepting an operation input for giving a control instruction to the screen from the screen. In general, since such an operation display device has a graphic display function, it can display an operation panel, a switch, an indicator lamp, and the like, and serves as an operation terminal in the control system.

  Problems may occur depending on the operation of the operation display device by the operator, but in order to analyze the problem later, it is necessary to record the process from the operation that caused it to the result. For example, in Patent Document 1, when it is detected that the key is operated by the operator, the key code data of the key and the time stamp data of the time when the key operation occurs are written together in the history memory, It describes that an operator's image taken with a video camera is written in an image memory, and an address in the image memory in which the image is stored is written in a history memory and associated with both of the above data.

As another history recording method for leaving the operator's image in this way, there is a method using a personal authentication technique as described in Patent Document 2, for example. In Patent Document 2, facial features unique to a person to be authenticated are stored in advance as dictionary information, and the event information is collated with facial feature information obtained from a person's input facial image using the apparatus. Thus, there is described a person authentication device that recognizes whether or not the person of the input face image is the person himself / herself based on the similarity between the two information. This person authentication device stores a face image of a person using the device as a history image.
Japanese Unexamined Patent Publication No. 2000-171515 (released on June 14, 2002) JP-A-2005-84985 (published on March 31, 2005)

  However, in the history recording method disclosed in Patent Document 1, there is a possibility that an image of the operator is recorded even when the operator is not facing the front with respect to the camera. For this reason, there is an inconvenience that a person who has performed an operation causing a problem cannot be identified from an image recorded including an image in which the operator is not facing the front. In addition, since an image that cannot be used to specify a person and cannot be used or an image including a portion other than a face is stored even if the person can be specified, the memory capacity for storing the image is significantly increased. .

  In the history recording method disclosed in Patent Document 2, when the face image of the first frame after the start of collation is saved as a history image, the highest similarity is obtained in order to eliminate the inconvenience that the face is not facing the front. The input face image is saved as a history image. Thereby, the probability that the history image is arranged in the center of the image and the face image faces the front increases.

  However, since the history image saving described in Patent Document 2 uses personal authentication technology, it is necessary to previously register the facial feature information of the person from the face image of the person to be authenticated. In addition to the time and effort required for this process, a maintenance operation is required to update the database for registering the feature information every time a registered person is updated, which is costly. Therefore, applying such a history recording method to the operation display device to save a person's face image is time-consuming and costly for analyzing problems that do not occur frequently. This is not preferable from the viewpoint of cost reduction.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide an operation display device that saves an operator's face image together with a history of operations performed by the operator without being troublesome. There is.

  An operation display device according to the present invention accepts an operation by an operator on a screen showing operation contents, and stores an image of the photographed operator in association with history information of a predetermined operation by the operator In the device, in order to solve the above problem, whether or not the photographed image of the operator includes a front face image facing the front with respect to the operation surface of the operation display device in response to a predetermined operation by the operator A front image determination unit that determines whether the image includes the front face image, and a face image storage unit that cuts out a part of the front face image from the image and stores it in a first storage area; History information storage means for storing the history information in a second storage area in a state associated with the front face image so that the front face image can be read based on the history information. Characteristic It is.

  In the above configuration, an operator's image previously captured by a camera or the like is input to the operation display device. In addition, when the operator performs a predetermined operation on the screen (for example, operation of a login switch on the login screen), if the operator is facing the front with respect to the operation surface of the operation display device, The front image determination unit determines that the photographed operator image includes a front face image. Then, the face image storage means cuts out the front face image from the image and stores it in the first storage area, and the history information storage means stores the history information in association with the face image in the second storage area. Saved in.

  Assuming that the person recognized by the three parties operates the operation display device, in this way, by storing the front face image of the operator who performed a specific operation on the operation display device, A person who knows the operator can specify who operated the operation display device. Therefore, the front image determination means determines that the operator's image includes a front face image so that a third person can easily identify a person from the stored front face image. Further, when an outsider operates the operation display device, this can be specified from the stored front face image. Thereby, it is possible to take measures to prevent outsiders from operating the operation display device.

  Thus, since the operation display device does not require personal authentication as described in Patent Document 2, it is not necessary to register the characteristics of the front face image of the person in charge who operates the operation display device in advance. Therefore, the front face image of the operator can be saved together with the history of operations by the operator, easily and without taking time and effort. Further, since the front face image is cut out from the image including the part other than the face and stored in the first storage area, the capacity of the storage medium (for example, memory) constituting the first storage area can be reduced.

  The operation display device reads the front face image stored in the first storage area based on the history information read from the second storage area, and displays the history information together with the history information. It is preferable to provide display means. Thereby, the front face image and the history information can be confirmed together.

  Further, in the operation display device, the front image determination unit includes an operator image captured during a predetermined period before and after the predetermined operation among images of an operator captured continuously during a preset period. By determining whether the front face image is included or not, more appropriate front face images can be acquired, and the possibility of omission of recording of the front face image can be reduced.

  In any one of the operation display devices described above, the front image determination unit determines that the image is the front face when a predetermined facial part is detected from the image among eyes, nose, mouth, eyebrows, and ears. It is preferable to determine that an image is included. Thereby, even when the entire face is not shown in the video of the camera, the front image determination unit can objectively determine whether or not the image includes a front face image.

  In any one of the operation display devices described above, or the front image determination unit, a facial component detection unit that detects two or more facial components among eyes, nose, mouth, eyebrows, and contour from the image; In the combination of two face parts, the inclination degree calculating means for calculating the inclination degree indicating the deviation of the relative position of the face parts of each combination from the relative position of the face parts in the front face in the combination of the two face parts And when the inclination of at least one of the selected combinations is within a predetermined range, and the front index preset for the combination is equal to or greater than a reference value, the image is It is preferable to have a front face determination unit that determines that a front face image is included. In this configuration, the front image determination unit can detect the degree of inclination of the face from the front by the determination of the front face determination unit, and can more accurately determine whether the image includes a front face image. .

  In this operation display device, the front face determination means sums the front indices of each selected combination when the front index of any combination of the selected combinations is less than the reference value, and the total value If is more than the reference value, it is preferable to determine that the image includes a front face image. Thereby, even if the front index for each combination is less than the reference value, it can be determined that the image includes a front face image if the sum of the front indexes satisfies the condition.

  The operation display program of the present invention is an operation display program for operating any one of the operation display devices described above, and is a program that causes a computer to function as each of the means. The operation display program is recorded on a computer-readable recording medium. Thereby, the said operation display apparatus is realizable by making a computer read and run a program.

  In the operation display device according to the present invention, as described above, whether or not the photographed image of the operator includes a face image facing the front with respect to the operation surface of the operation display device is triggered by a predetermined operation by the operator. A front image determination unit that determines whether the image includes the face image, a face image storage unit that stores the face image in a first storage area, and the face based on the history information. History information storage means for storing the history information in a second storage area in a state associated with the face image so that an image can be read out. Thus, the operator's face image can be saved together with the history of operations performed by the operator, easily and without any effort. Therefore, there is an effect that error analysis with high reliability can be performed.

  An embodiment of the present invention will be described with reference to FIGS. 1 to 11 as follows.

  As shown in FIG. 1, a programmable display 1 as an operation display device includes an arithmetic processing unit such as a CPU, and displays an input operation created by a user and screen data for display. It is a dedicated computer that realizes the operation function and display function.

  The programmable display 1 acquires the state of each device 11 displayed on the display screen via the PLC 13 through communication with the PLC 13 via the serial cable 12. It has a function to display the status. The programmable display 1 has a function of instructing the PLC 13 to control the state of the device 11 in accordance with an operation on the touch panel 3 described later. The programmable display 1 also has a function of recording, storing, and displaying an operator's front face image together with operation information.

  In order to realize the above functions, the programmable display 1 includes a display 2, a touch panel 3, interface units (I / F in the figure) 4, 5, a data memory 6, a work memory 7, and an HMI process. Unit 8 and a face detection processing unit 9. Hereinafter, each main part of the programmable display device 1 will be described in detail.

  As the display 2, a flat panel display such as a liquid crystal display, an EL display, or a plasma display is preferably used in order to configure the programmable display 1 to be thin. The touch panel 3 is an input device provided for performing touch input on the display image of the display 2.

  The interface unit 4 is a communication control unit for the programmable display 1 to perform serial communication with the PLC 13, and each is connected by a serial cable 12. The interface unit 4 performs communication using a communication protocol according to the manufacturer and model of the PLC 13 based on the connected device data. The connected device data and the communication protocol are stored in, for example, a data memory 6 described later.

  Note that the connection between the programmable display 1 and the PLC 13 is not limited to the serial cable 12, and may be made by a network such as Ethernet (registered trademark) or other communication means.

  Here, the PLC 13 will be described.

  The PLC 13 captures the state of the input device 11 via the input unit and outputs it at every predetermined scan time such as several tens of ms, for example, according to a sequence program (ladder program or the like) created by the user. A control instruction is given to the device 11 for changing the state.

  As the input device 11, devices such as sensors (temperature sensors, optical sensors, etc.) and switches (push button switches, limit switches, pressure switches, etc.) are used. As the output device 11, an actuator, a relay, a solenoid valve, a display, or the like is used. These devices 11 are arranged in required parts such as a production line.

  The device 11 may be a specific memory area in the programmable display device 1 (for example, a user area in a system area described later), or may be a specific memory area in a memory in the PLC 13. These memory areas are set as internal devices.

  The memory (device memory) in the PLC 13 stores data (word data or bit data) indicating the state of the device 11 (output value from the device 11 or set value to the device 11) in an area specified by the device address. is doing. In the memory, a word device and a bit device are set. The word device is set as an area for storing word data such as numeric values as input / output data, and is designated by a word address (device address). The bit device is set as an area for storing bit data such as an on / off state, and is set by a bit address (device address). With such a setting, the device 11 can be controlled by simply designating and accessing an arbitrary word device or bit device in the memory as a device address, or information regarding its state can be individually retrieved.

  A variable is associated with a device address called by a unique name for each manufacturer of the PLC 13. The variable is a name that can be arbitrarily set by the user, and a comment described later can be handled as a variable.

  In the following description, the device address is appropriately referred to as an address.

  The interface unit 5 is a communication control unit for the programmable display device 1 to communicate with the editor computer 21. The interface unit 5 is connected to a transfer cable 14 for transferring screen data and the like from the editor computer 21 to the programmable display 1. For example, a USB cable is used as the transfer cable 14.

  The data memory 6 is a memory for storing screen data, the above-described communication protocol, etc., and FEPROM (Flash Erasable and Programmable ROM) is used. Since FEPROM is a rewritable read-only flash memory, it plays the role of a hard disk drive in a general personal computer. A flash memory does not have a movable part and is resistant to impacts, and thus operates stably even in a poor ambient environment.

  The screen data is a data file of a user screen (unit screen configured by screen data for one frame) displayed on the programmable display 1. The screen data includes data of objects such as a base screen and parts to be displayed on the display 2, processing definition information assigned to each object, and the like. The process definition information defines various processes performed in relation to the user screen, and is created for each event to be executed on the base screen. This processing specification information basically includes the file number (screen number) of the base screen on which the display process is to be executed, the event name that specifies the operation content to be executed on this base screen, and each execution event. A set of reference information including one or more data to be referred to is provided.

  The work memory 7 is composed of, for example, a DRAM, and is used for work during arithmetic processing such as display control. The work memory 7 is used for temporary storage of the acquired data of the device 11. The working memory 7 has a state memory area for transferring the state of the device 11 (the contents of the device address) stored in the device memory of the PLC 13 to and from the device memory.

  The memory of the programmable display 1 is provided with a system area. The system area includes a system data area that stores data necessary for the operation of the programmable display 1, a user area that can be used as an internal device by a user, a special relay area that is used as a special relay, and the like. This system area is provided in the work memory 7, for example.

  The HMI processing unit 8 is provided as the center of control in order to realize the HMI function. The HMI processing unit 8 is a functional block realized by executing a program.

  The HMI processing unit 8 acquires the state of the device 11 as state data from the memory of the PLC 13 by communicating with the PLC 13 at predetermined time intervals or at predetermined events, and is used in the programmable display 1. The data is written in a temporary memory space, for example, a specific area (state memory area) of the working memory 7. Further, the HMI processing unit 8 rewrites the state data of the device 11 in the state memory area so as to update the state data in the internal memory of the PLC 3 and transmits it to the PLC 13. Based on the screen data created by combining the above-mentioned objects, the HMI processing unit 8 acquires the state data of the device 11 from the PLC 13 and draws and displays it on the programmable display 1 and performs operations on the screen. Accordingly, the operation when the change of the state of the device 11 is instructed is specified by the process definition information.

  In addition, when an event for operating the face log processing unit 9 to be described later occurs, the HMI processing unit 8 adds a parameter to a command for starting execution of the face log processing by the face log processing unit 9 and adds a parameter to the face log processing unit 9 to send. As parameters, for example, an event ID, a screen number, and a state are prepared. When using these parameters, the HMI processing unit 8 generates and sends a command with parameters in the format of $ SendEvent.exe <event ID> <screen number> <state>.

  The event ID and the screen number in the parameter become operation history information by the operator.

  The event ID is an ID number that identifies the event. The screen number is the screen number of the screen where the event has occurred. An event is an operation of an operator on the screen, and a typical operation is a switch operation. Such switch operations include login and logout to the programmable display 1, screen switching, setting value input, ON / OFF operation, and the like. For each event of login, logout, screen switching, setting value input, and ON / OFF switching, for example, event IDs “1”, “2”, “3”, “10”, “11” are defined, respectively. ing.

  The state is state data stored in the user area of the above-described system area that the user wants to add as a parameter. The user can arbitrarily set this state.

  Further, the HMI processing unit 8 operates a parameter sending unit 91 and a face log execution unit 92, which will be described later, based on processing definition information defined for a switch operated to generate an event. Specifically, the switch includes a word switch for rewriting data at a specified address (word address) in the user area of the programmable display 1 and a specified address (bit address) in the user area. The function as a bit switch for switching bits is set in combination.

  In the above bit address, a bit for starting a program for realizing the parameter sending unit 91 is set when turned ON. As a function of the bit switch, the switch is set in advance to switch the bit to ON when a touch operation is performed. In addition, as a function of the word switch, the switch is preset to write the value of each parameter to a designated word address when a touch operation is performed. These settings are performed by a switch attribute setting unit 24a of the screen editor unit 24 described later.

  Thereby, the HMI processing unit 8 switches the bit specified by the word address to ON by touching the switch, generates the command, and writes the parameter to the word address. Generate a command with parameters.

  Further, the HMI processing unit 8 operates a log display execution unit 96 to be described later based on the processing definition information specified for the switch operated to instruct the display of the stored front face image of the operator. Specifically, the switch has a function as a bit switch for switching a bit at a specified address (bit address) of the user area of the programmable display 1.

  In the above bit address, a bit for starting a program for realizing the log display execution unit 96 is set when turned ON. As a function of the bit switch, the switch is set in advance to switch the bit to ON when a touch operation is performed. These settings are also made by a switch attribute setting unit 24a of the screen editor unit 24 described later.

  As a result, the HMI processing unit 8 generates a command for activating the log display execution unit 96 by switching the bit specified by the word address to ON by touching the switch.

  The program for realizing the HMI processing unit 8 can be recorded on a recording medium configured to be separable from the programmable display device 1 and may be installed in the programmable display device 1 from the recording medium.

  The above recording media include tape systems such as magnetic tapes and cassette tapes, magnetic disk systems such as flexible disks and hard disks, optical disk systems such as CD-ROM, MO, MD, and DVD, IC cards (including memory cards), optical A card system such as a card is preferred. In addition, the program medium may be a medium that carries a fixed program including a semiconductor memory such as a mask ROM, EPROM, EEPROM, flash ROM, or the like.

  Further, if the programmable display 1 is configured to be connectable to a communication network including the Internet, the recording medium is a medium that carries the program in a fluid manner so as to download the program from the communication network. Also good. However, when the program is downloaded from the communication network in this way, the download program may be stored in advance in the programmable display 1 or may be installed from another recording medium.

  The face log processing unit 9 records and displays the front face image of the operator together with operation information (face image detection date and time, operated screen number, etc.) in accordance with an instruction from the HMI processing unit 8. I do.

  In order to realize this function, the face detection log processing unit 9 includes a parameter transmission unit 91, a face log execution unit 92, a shared memory 93, a face log initial setting file 94, a face log file 95, a log display execution unit 96, a log A display initial setting file 97 and a face image database (face image DB in the figure) 98 are provided.

  The parameter sending unit 91 is activated by the command with parameters sent from the HMI processing unit 8 and sends the parameter added to the command to the face log execution unit 92 by socket communication.

  The face log execution unit 92 is a part that executes face log processing for detecting and recording a human face from the operator's image taken by the camera 15, and includes a face log control unit 92a, a face detection unit 92b, and a face logging. A portion 92c is included. The camera 15 is a small camera such as a USB camera, for example, and the programmable display 1 so that the operator can take a front image of the display 2 facing the front when the programmable display 1 is operated. It is attached to the case. As the mounting position of the camera 15, for example, the central portion on the upper end surface of the housing is suitable.

  The face log control unit 92a controls operations of the face detection unit 92b and the face logging unit 92c. The face log control unit 92a is activated upon receiving the parameter sent from the parameter sending unit 91, writes the parameter to the shared memory 93, and reads the face log setting data from the face log initial setting file 94 to the shared memory 93. . When the face log control unit 92a receives a log storage request from the HMI processing unit 8 via the parameter transmission unit 91, the face log control unit 92a causes the face logging unit 92c to execute a storage process. As an event for giving the above-mentioned log storage request, a switch operation by the user on the screen displayed on the programmable display 1, ON or OFF of a specific bit in the memory of the PLC 12, the above-mentioned user area in the programmable display 1 For example, ON or OFF of a specific bit that can be detected.

  The face log setting data includes face detection accuracy (%), face detection time difference (seconds), number of detected images, shooting period (seconds), face detection interval (seconds), maximum detection time (seconds), storage directory name, etc. It is out. The detection accuracy is a degree of certainty that the face detection unit 92b determines that a human face is included in the input image data. The face detection time difference is a time difference when the face detection unit 92b detects a face with respect to an event occurrence, and is represented with a sign of ±. The number of detected images is the number of images for which face detection is performed from input images sequentially input from the camera 15. The face detection interval is a time interval at which the face detection unit 92b performs face detection processing. The maximum detection time is the maximum time for the face detection unit 92b to detect a face from input image data. The storage directory name is a name of a directory in which a face image database 98 for storing detected face image data (described later) as a JPEG file (detected face file) is provided. The face image database 98 is set in the data memory 6, for example.

  The face log setting data also includes logging conditions. The logging condition is a condition when the face logging unit 92c stores face detection state data described later. This condition includes information for specifying data to be stored together with the detected front face image.

  The face detection unit 92b uses the operation surface (display 2 on the display 2) of the programmable display 1 based on the face log setting data based on the image data (input image data) of the figure including the face of the operator photographed by the camera 15. The face (front face image) of the person facing the front with respect to the face) is detected. The face detection unit 92b detects a face from the input image data when the detection accuracy is equal to or higher than a preset detection accuracy (for example, 80%). Specifically, as shown in FIG. 2, the face detection unit 92b detects a face from input images input before and after the event occurrence (in FIG. 2, only five input images of ± seconds from the event occurrence are shown). When detecting, if the face detection interval is set to 1 second, the face detection time difference is set to -1 second, and the maximum detection time is set to 3 seconds, the second to fourth from the left in FIG. Face detection is performed on the input image. In FIG. 2, the input image shown in white is an input image in which a face is detected by determination with an accuracy of 80% or more, and the input image shown in gray is an input image in which the face is not detected with an accuracy of less than 80%. It is. In this example, faces are detected in the second and third input images, indicating that the face detection facing the front is successful.

  Thus, by performing face detection for a plurality of input images in a predetermined period, the success rate of face detection can be increased as compared to the case where face detection is performed with one input image.

  In addition, the face detection unit 92b cuts out a face portion (detected face image data) from the input image data in which a face is detected, and stores it in the face image database 98 as a face detection file. At the same time, the face detection unit 92b includes the path name indicating the location of the face detection file together with the detection date and time (may include up to minutes and seconds), the detection accuracy, and the parameters stored in the shared memory 93. Write to the shared memory 93 as data.

  The detection of the front face image by the face detection unit 92b will be described in detail later.

  The face logging unit 92c reads the face detection state data from the shared memory 93 at a timing instructed by the face log control unit 92a, refers to the logging condition of the face log setting data in the shared memory 93, and is necessary among the face detection state data Various data is recorded in the face log file 95. The face logging unit 92c writes the face detection state data in the face log file 95 in the CSV (Comma Separated Values) format in the order of detection during recording. Thereby, for example, by developing the face detection state data from the face log file 95 into a table using spreadsheet software or the like, a list of face detection state data can be easily confirmed.

  The face log initial setting file 94 stores the aforementioned face log setting data. The face log initial setting file 94 is set in the data memory 6, for example.

  The face log file 95 is set in, for example, a data log memory (not shown) provided in the programmable display device 1. The data log memory is composed of SRAM, but may be composed of a memory card that is detachably connected to the programmable display 1. Since the memory card generally has a larger storage capacity than the SRAM, it is effective when logging a large amount of data.

  The log display execution unit 96 reads the face detection state data from the face log file in accordance with the log request from the face log control unit 92 and superimposes it on the screen displayed on the display 2, for example as shown in FIG. It is displayed on the face log window 201. At this time, the log display execution unit 96 passes the display data of the face log window 201 to the HMI processing unit 8 and causes the HMI processing unit 8 to display the face log window 201. When displaying the face log window 201, the log display execution unit 96 displays the correspondence between the event code (“event ID” described above) and the event name (character string) stored in the log display initial setting file 97. The event ID included in the face detection state data is converted into an event name and displayed. Further, the log display execution unit 96 opens the face detection file stored in the face image database 98 based on the path name included in the face detection state data, and the “face image” area in the face log file 95 is displayed. To display.

  In the face log window 201 shown in FIG. 3, the event occurrence date / time, event ID, screen number, state, face detection time difference, and detection accuracy are displayed as face log information in a line following the face image. Each face log information is displayed side by side in the order of event occurrence (face detection order). In the face log file 95, the state, face detection time difference, and detection accuracy may be displayed as necessary. In addition, when a plurality of cameras 15 are installed, the camera number assigned to each camera 15 is logged as face detection state data as information for identifying the camera 15 that captured the face-detected input image. You may make it display on the face log window 201. FIG.

  In the face detection processing unit 9, the parameter sending unit 91, the face log execution unit 92, and the log display execution unit 96 are functional blocks that are realized by executing a program. This program can also be recorded on the above-described recording medium configured to be separable from the programmable display 1 in the same manner as the above-described HMI processing unit 8, and may be installed in the programmable display 1 from the recording medium. .

  Next, the editor computer 21 will be described.

  As shown in FIG. 1, the editor computer 21 has a function as an editor device that creates a screen displayed on the programmable display 1.

  The editor computer 21 is constituted by a personal computer having a CPU, a memory (RAM, ROM, etc.), an external storage device (hard disk drive, MO drive, etc.), a display device and an input device (keyboard, mouse, etc.). The editor computer 21 includes a control unit 23, an interface unit (I / F in the figure) 24, a screen editor unit 24, and a data storage unit 25.

  The interface unit 23 is a communication control unit for performing communication with the programmable display 1 and is connected to the transfer cable 14.

  The control unit 22 is a functional block realized by hardware (CPU, memory, etc.) and software (operating system, etc.), and controls the execution of application programs and the operation of peripheral devices. The control unit 11 has a work memory 22a to fulfill its function. The work memory 22a is constituted by a DRAM or the like, and is used for developing a screen file or a ladder file.

  The control unit 22 implements the editor unit 24 by executing editor software that is an application program. The editor unit 24 is recorded on a recording medium configured to be separable from the editor computer 21 and can be installed in the editor computer 21 from this storage medium.

  The above-mentioned recording medium (program medium) is a computer-readable recording medium, such as a tape system such as a magnetic tape or a cassette tape, a magnetic disk system such as a flexible disk or a hard disk, CD-ROM, MO, MD, DVD, etc. A card system such as an optical disk system, an IC card (including a memory card), or an optical card is suitable. In addition, the program medium may be a medium that carries a fixed program including a semiconductor memory such as a mask ROM, EPROM, EEPROM, flash ROM, or the like.

  Further, if the editor computer 21 is placed in an environment that can be connected to a communication network including the Internet, the editor computer 21 may be a medium that fluidly carries the program so as to download the program from the communication network. However, when the program is downloaded from the communication network in this way, the download program may be stored in the editor computer 21 in advance, or may be installed from another recording medium.

  The editor unit 24 supports components (objects) such as switches, lamps, numeric keys, and various displays (for example, a numerical display, a meter display, and a graph display) so as to support the creation of a user screen that is a user-specific screen. ), Drawing function, text input function, etc. Objects are registered in a library format so that the user can easily select them.

  The screen data created by the editor computer 21 is stored in the data storage unit 25 configured by the above external storage device. The screen data is read from the data storage unit 25 and transferred to the programmable display device 1 via the transfer cable 14.

  The screen editor unit 24 has a switch attribute setting unit 24a.

  The switch attribute setting unit 24a is provided to set various attributes as processing definition information in the switch object. The switch attribute setting unit 24a displays a dialog box for setting attributes as a user interface and allows the user to set attributes. Thereby, the user can set the attribute for generating the above-mentioned event to a switch using switch attribute setting part 24a.

  Here, face log and log display processing in the programmable display device 1 will be described with reference to the flowcharts of FIGS. 5 and 6.

  In the case of face log processing, first, as shown in FIG. 5, when an event occurs by operating a specific switch on the displayed screen (step S1), the HMI processing unit 8 A command to which parameters such as an event ID and a screen number of the screen are added is transmitted (step S2). The parameter sending unit 91 of the face log processing unit 9 is activated in response to this command and sends the above parameters to the face log control unit 92a (step S3). The face log control unit 92a is activated upon receiving the above parameters, writes the parameters to the shared memory 93, reads the face log setting data from the face log initial setting file 94, and sets it in the shared memory 93 (step S4). .

  Upon receiving an instruction from the log control unit 92a, the face detection unit 92b starts detecting a human face (step S5). The face detection unit 92b detects the face based on the face log setting data from the input image data of the operator input from the camera 15. Here, the face detection unit 92b determines whether or not the face detection is successful (step S6). If the face detection unit 92b is successful, the face portion is cut out from the input image data as detected face image data, and the face image database. 98 (step S7), and the face detection state data including the path name, detection date and time, detection accuracy, and parameters of the face detection file is written to the shared memory 93 (step S8).

  If the face detection unit 92b fails to detect the face in step S5, the face detection unit 92b does not cut out and store the face part as face image data from the input image data (step S9), and the path name of the face detection file. The face detection state data excluding is written in the shared memory 93 (step S11).

  Subsequent to step S8, upon receiving an instruction from the face log control unit 92a, the face logging unit 92c reads the face detection state data from the shared memory 93 and stores it in the face log file 95 (step S9). However, if the detected face image data is not saved in step S10, the detected face image data is in a blank state, and only the face detection state data is saved in the face log file 95.

  Further, the series of processes described above are continuously performed until the operation of the programmable display device 1 is stopped (step S10).

    In the case of face log processing, first, as shown in FIG. 5, when a log display instruction is generated by operating a switch for instructing log display on the displayed screen (step S11), HMI processing is performed. The unit 8 receives the instruction and sends a log display command (step S12).

  The log display execution unit 96 is activated in response to this command, reads the face detection state data from the face log file 95, and displays the face log in the face log window 201 based on the data (step S13). When the user performs an operation to close the face log window 201 (step S14), the log display process is finished.

  Here, the front face image detection function of the face detection unit 92b will be described in detail. The determination of whether or not the input image includes a front face image by the face detection unit 92b is divided into determination based on face component detection and determination based on the positional relationship of the detected components, which are used depending on the application.

  First, the determination based on face part detection will be described. The front face image detection function of the face detection unit 92b is specifically configured as the face detection unit 100 shown in FIG. The face detection unit 100 includes a face part detection unit 101 and a front determination unit 102.

  The face part detection unit 101 detects each part of the face (eyes, nose, mouth, eyebrows, ears, etc.) using a known method.

  The front determination unit 102 refers to the front index table 103 for the facial parts detected by the facial part detection unit 101, and sums the front indices used for the determination among the front indices set in advance for each part of the face. Then, by comparing the total value with the front face determination reference value, it is determined whether or not it is a front face. Table 1 shows an example of the front index table 103. The front index table 103 sets face parts, detection criteria, front indices, and used / unused in association with each other. For example, in the programmable display 1, the front index table 103 is developed in the work memory 7.

  The detection criterion is a criterion for determining whether or not each part of the face has been detected. For example, in the case of detecting the eyes, it is determined that the eyes are detected only when both eyes are detected, and only one eye is detected. In this case, it is not determined that eye detection has been performed. The nose is detected only when the central part of the nose is detected. When only a part other than the central part of the nose is detected, it is not determined that the nose is detected. .

  The front index is a numerical value of the weight of the detected face part with respect to the front face determination criterion.

  Use / unused indicates whether or not the front index of the detected face part is used for front face determination.

  The front determination unit 102 determines whether or not each part of the face is detected according to the above detection criterion, and sums up the front indices of the detected parts. For example, when eyes and eyebrows are detected, the total value of the front index is calculated as 90. The total value is compared with a predetermined determination reference value, and if it is equal to or greater than the determination reference value, it is determined that the face is a front face.

  The determination reference value is set to 80, for example, and when eyes and eyebrows are detected, the total value of the front index is equal to or greater than the determination reference value, so that it is determined to be a front face. On the other hand, when only the nose and the mouth are detected, the total value of the front index is 40, which is less than the determination reference value, and thus is not determined to be a front face.

  The determination reference value is variously set according to the environment in which it is used. When the front face determination is used to determine whether or not the display 2 is being viewed, the determination reference value is set high.

  Further, in an environment such as a factory where the programmable display 1 is used, since a mask and a hat are worn, the mouth, nose, and ears are often hidden. Therefore, even if the mouth, nose, and ear are detected, the front index of those facial parts can be set not to be used.

  Next, the determination based on the positional relationship of the detection components will be described with reference to FIGS. This determination can detect the degree of inclination of the face from right to left, which cannot be determined by the above-described determination based on face component detection, and enables more accurate determination of whether or not the face is a front face.

  The front face image detection function of the face detection unit 92b is specifically configured as the face detection unit 110 shown in FIG. The face detection unit 110 includes a face part detection unit 111, an inclination calculation unit 112, and a front determination unit 113.

  The face part detection unit 111 detects each part of the face (eyes, nose, mouth, eyebrows, ears, contours, etc.) using a known method as described above.

  The inclination calculation unit 112 determines whether or not each part of the face has been detected, and calculates an inclination described later from the relative position in the detected combination of face parts. A method of calculating the inclination by the inclination calculation unit 112 will be described in detail later with reference to FIGS. 9 to 11.

  The front determination unit 113 refers to the front index table 114, and the front index when the inclination calculated by the inclination calculation unit 112 is within the detection reference range satisfies the front face determination criterion (the aforementioned determination accuracy). Whether or not the face is the front is determined by whether or not it is present. The front face determination by the front determination unit 113 will be described in detail later with reference to the flowchart shown in FIG.

  As shown in Table 2, the front index table 114 sets two types of face parts for calculating a relative position, an inclination, a detection reference, a front index, and used / unused in association with each other. . For example, in the programmable display 1, the front index table 114 is developed in the work memory 7.

  The degree of inclination is a numerical value indicating the degree of inclination of the face from the front to the left and right, and a specific calculation method will be described later.

  The detection criterion is a criterion for determining whether or not a combination of each part of the face has been detected. For example, the detection of eyebrows and nose is performed when the slope Sbn calculated by the calculation formula described later is within ± 10%. Determines that eyebrows and nose have been detected, and does not determine that eyebrows and nose have been detected if the slope Sbn is less than −10% or exceeds 10%.

  The front index is a numerical value of the weight of the detected combination of face parts for the front face determination criterion.

  Use / unused indicates whether or not the front index of the detected face part is used for front face determination.

  Here, the determination procedure based on the positional relationship of the detection components will be described with reference to FIG.

  FIG. 8 is a flowchart of determination based on the positional relationship of detection components. First, as in the case of the above-described determination by facial component detection, the facial component detection unit 111 detects facial components (eyes, nose, mouth, eyebrows, ears, contours) (step S21). Subsequently, the inclination calculating unit 112 calculates the inclination for each combination of detected components (step S22). Subsequently, the front face determination unit 113 determines whether or not the inclination of the combination of face parts set to be used in the front index table 114 satisfies the detection criterion (step S23). If there is a combination that satisfies the detection criterion (“YES” in step S23), the front face determination unit 113 further determines whether or not the front index to be used satisfies a preset front face determination criterion (step S23). S24). If the front index satisfies the front face determination criterion (“YES” in step S24), the front face determination unit 113 determines that it is a front face (step S25), and the front index does not satisfy the front face determination criterion. In the case (“NO” in step S24), the sum of the front index to be used is further calculated (step S26).

  Then, the front face determination unit 113 determines whether or not the total value of the front index satisfies the front face determination criterion (step S27), and when the front face determination criterion is satisfied (“YES” in step S27). If the total value of the front index does not satisfy the front face determination criterion (“NO” in step S27), it is determined that the face is not a front face (step S28). Also, the front face determination unit 113 determines that the face is not a front face even if there is no combination that satisfies the detection criterion in Step S27 ("NO") (Step S28).

  Finally, the face detection unit 110 determines whether or not to perform the front face detection process for the next face image (step S29). If the detection process is performed, the process returns to step S21, and the detection process is not performed. If this is the case, the process ends.

  Note that the determination reference value is arbitrarily set. As the determination reference value is higher, the face is not determined to be a front face unless the face is directed in a direction closer to the front. In an environment such as a factory where the programmable display 2 is used, the mouth and nose are often hidden by wearing a mask or a hat. In this case, it may be set so that the front ratio is not calculated for each combination of eyebrows and nose, eyebrows and mouth, contour and nose, and contour and mouth.

  Subsequently, a method of calculating the inclination performed by the inclination calculating unit 112 will be specifically described with reference to FIGS. 9 to 11.

FIG. 9 shows a face directly in front. In the following description, a direction along a straight line connecting the centers of both eyes is a horizontal direction, and a direction along a straight line connecting the centers of both eyes is a vertical direction. Here, the slope Sen is calculated for the combination of eyes and nose. First, let Le1 be a straight line extending in the vertical direction from the center of the right eye, and let Le2 be a straight line extending in the vertical direction from the center of the left eye. Similarly, let Ln be a straight line extending in the vertical direction from the center of the nose. Here, when the distance between the straight line Le1 and the straight line Ln is Den1, and the distance between the straight line Le2 and the straight line Ln is Den2, the slope Sen is
Sen = ((Den2-Den1) / (Den1 + Den2)) × 100
Is calculated. Since Den1 = Den2 when the face is in front, the slope Sen = 0 (%).

  FIG. 10A shows a face slightly facing right, and FIG. 10B shows a face slightly facing left. In FIGS. 10A and 10B, eyes, nose, eyebrows, and the like can be detected. Therefore, in the determination based on the above-described face component detection, a front face is determined. Here, the straight lines Le1, Le2, and Ln and the distances Den1 and Den2 are set as in the case of the face in front of FIG. When the face is facing right (FIG. 10 (a)), Den1 <Den2, and therefore, the slope Sen> 0 (%), and when the face is facing left (FIG. 10 (b)), Den1> Den2, so the slope is Sen <0 (%). The inclination Sen approaches 100 (%) as the face turns to the right, and the inclination Sen approaches -100 (%) as the face turns to the left. That is, the slope Sen indicates the deviation of the relative position between the eyes and the nose from the relative position between the eyes and the nose in the true front face.

  In addition to the eyes and nose, the inclination is also calculated for each combination of eyebrows and nose, eyebrows and mouth, eyes and mouth, nose and contour, eyebrows and contour, eyes and contour, and mouth and contour.

For example, in FIG. 9, the slope Sno in the combination of the nose and the contour is the horizontal distance between the center of the nose and the contour of the right side of the face in the horizontal direction Dno1, and the horizontal distance between the center of the nose and the contour of the left of the face If the distance in the direction is Dno2, the slope Sno is
Sno = ((Dno1-Dno2) / (Dno1 + Dno2)) × 100
Is calculated. In FIG. 9, since Dno1 = Dno2, Fno = 0 (%). On the other hand, when the face is facing right, Dno1 <Dno2 and Sno> 0 (%), and when the face is facing left, Dno1> Dno2 and Sno <0 (%). Note that Sen approaches 100 (%) as the face turns to the right, and Sen approaches -100 (%) as the face turns to the left.

Similarly, in FIG. 9, the inclination Seo in the combination of the eyes and the contours is represented by the horizontal distance between the center of the right eye and the contour of the right side of the face as Deo1, and the horizontal direction between the center of the left eye and the contour of the left side of the face. If the distance is Deo2, the slope Seo is
Seo = ((Deo1-Deo2) / (Deo1 + Deo2)) × 100
Is calculated.

  Further, a method for calculating the inclination in the combination of other components will be described with reference to FIG. The face shown in FIG. 11 and the face shown in FIG. 9 are the same. In FIG. 11, straight lines Le1, Le2, and Ln are set in the same manner as in FIG. Further, a straight line extending in the vertical direction from the inner end of the right eyebrow is set as Lb1, a straight line extending in the vertical direction from the inner end of the left eyebrow is set as Lb2, and a straight line extending in the vertical direction from the center of the mouth is set as Lm. In FIG. 11, the straight line Ln and the straight line Lm are the same straight line.

First, the slope Sbn in the combination of the eyebrows and the nose is given by assuming that the horizontal distance between the straight line Lb1 and the straight line Ln is Dbn1, and the horizontal distance between the straight line Lb2 and the straight line Ln is Dbn2.
Sbn = ((Dbn1−Dbn2) / (Dbn1 + Dbn2)) × 100
Is calculated.

The inclination Sbm in the combination of the eyebrows and the mouth is expressed as follows. When the horizontal distance between the straight line Lb1 and the straight line Lm is Dbm1, and the horizontal distance between the straight line Lb2 and the straight line Lm is Dbm2, the inclination Sbm is
Sbm = ((Dbm1−Dbm2) / (Dbm1 + Dbm2)) × 100
Is calculated. In FIG. 11, Dbn1 = Dbm1 and Dbn2 = Dbm2.

The slope Sem in the combination of the eyes and the mouth is expressed as follows. When the horizontal distance between the straight line Le1 and the straight line Lm is Dem1, and the horizontal distance between the straight line Le2 and the straight line Lm is Dem2, the slope Sem is
Sem = ((Dem1−Dem2) / (Dem1 + Dem2)) × 100
Is calculated.

The inclination Sbo in the combination of the eyebrow and the contour is the horizontal distance between the inner edge of the right eyebrow and the right contour Dbo1, and the horizontal distance between the inner edge of the left eyebrow and the left contour. Assuming Dbo2, the slope Sbo is
Sbo = ((Dbo1-Dbo2) / (Dbo1 + Dbo2)) × 100
Is calculated.

The slope Smo in the combination of the mouth and the contour is the slope when the horizontal distance between the mouth center and the right contour is Dmo1, and the horizontal distance between the mouth center and the left contour is Dmo2. Smo is
Smo = ((Dmo1-Dmo2) / (Dmo1 + Dmo2)) × 100
Is calculated.

  As described above, when the execution of face log processing is instructed by a predetermined switch operation or the like, the programmable display device 1 of the present embodiment displays the face detection unit 92b of the face log processing unit 9 in a programmable display by the operator. A face is detected in an image of which the detection accuracy is determined to be a predetermined value or more from an image of an operator photographed by the camera 15 while facing the display 2 when the device 1 is operated. As a result, the face of the operator facing the front with respect to the display 2 of the programmable display 1 is detected and stored in the face log file 95 in association with the operation related information by the face logging unit 92c. Therefore, the front face image of the operator who performed a specific operation on the programmable display 1 can be stored. Further, since the front face image is cut out from the input image and saved in the face image database 98, the size of the saved image can be reduced. As a result, the area used by the face image database 98 in the memory is reduced, so that the memory can be used effectively.

  If it is assumed that a person recognized by a third party operates the programmable display 1, it is possible to specify who operated the programmable display 1 by storing the face image. For example, as shown in FIG. 2, it is possible to know who logged in, what operation was performed, and when the user logged out with respect to the programmable display 1, and an error that occurred during that time is an operation of the operator. Can be almost specified.

  For this reason, in the programmable display device 1, the camera is displayed with the operator facing the display 2 when operating the programmable display device 1 so that a third party can easily identify a person from the stored face image. 15 is taken. Moreover, when an outsider operates the programmable display 1, it can identify from the preserve | saved face image. Thereby, it is possible to take measures to prevent outsiders from operating the programmable display 1.

  Thus, since the programmable display 1 does not require personal authentication as described in Patent Document 2, it is not necessary to register the feature of the face image of the person in charge who operates the programmable display 1 in advance. Therefore, the operator's face image can be saved together with the history of operations performed by the operator, easily and without any effort. In addition, since the operator's own face image is recorded, he / she tries to be careful with the operation, and as a result, the effect of suppressing the occurrence of an erroneous operation can be expected.

  Further, when the suspicious person who knows that the photograph is taken tries to operate the programmable display device 1 in an unnatural posture so that the face is not photographed, the face detection unit 92b correctly displays the face from the photographed image of the suspicious person. The face image of the suspicious person cannot be saved without being detected. For such suspicious person countermeasures, the face detection unit 92b cuts out a partial image to be saved from an input image with a predetermined detection accuracy and stores it in the face image database 98, and detects the partial image and the detected face. Set logging conditions to log images separately. As a result, it is possible to collectively display the partial images for which the face could not be detected, and to confirm the presence of the suspicious person.

  Such suspicious person countermeasures are effective for devices requiring high security, for example, ATM (Automated Teller Machine). Thereby, for example, it is possible to save an image of a suspicious person who operates while hiding his face or operating his face away.

  In the present embodiment, when the face is not correctly detected by the face detection unit 92b, the switch operation is enabled as it is, but the switch operation may be disabled. In order to realize this, the face detection unit 92b notifies the HMI processing unit 8 that the face detection has failed, and the HMI processing unit 8 responds to the notification by performing a switch operation that is an event for executing the face log. Invalid. As a result, if the face image is not correctly detected, the switch operation is invalidated. Therefore, when it is necessary to specify the operator, such as an important operation, the HMI processing unit 8 is operated with the front facing the programmable display 1, for example. By displaying such a message, it is possible to prompt the operator to redo the switch operation. However, in this case, if the aforementioned face detection interval or maximum detection time is set long, it takes time until the operator determines whether the operation is valid or invalid after the switch operation, and the operability is reduced. Therefore, a restriction may be provided so that the face detection interval and the maximum detection time cannot be set long.

  In addition, in this Embodiment, although the programmable display 1 was demonstrated as an operation display apparatus, this invention is not limited to the programmable display 1 if it is an apparatus which can accept the operation input on the displayed screen. For example, the operation display device may be a device such as the above ATM as well as a panel computer or an operation panel.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims. That is, embodiments obtained by combining technical means appropriately changed within the scope of the claims are also included in the technical scope of the present invention.

  The operation display device of the present invention can identify an operator as long as he / she knows the operator by detecting and storing the face from the photographed operator's image and displaying it as necessary. Therefore, since the face image of the operator can be saved together with the operation history information easily and without taking time and effort, it can be suitably used for error countermeasures of the operation display device.

It is a block diagram which shows the structure of the programmable display and editor computer which concern on embodiment of this invention. It is a figure which shows the operation | movement which detects the operator's face by the face detection part of the face log process part in the said programmable display. It is a figure which shows the face log window displayed by the log execution part containing the operator's face logged by the face log process part in the said programmable display. It is a flowchart which shows the procedure of the log process of the operator's face image by the said programmable display. It is a flowchart which shows the procedure of the display process by the said programmable display of the logged operator's face image. It is a block diagram which shows one structural example of the front face image detection function by the face detection part in the said face log execution part. It is a block diagram which shows the other structural example of the front face image detection function by the face detection part in the said face log execution part. It is a flowchart which shows the procedure of the determination process of the front face by the face detection part of the structure of FIG. It is a figure which shows the face image which faced the front detected by the face detection part of FIG. (A) is a diagram showing a face image slightly facing right detected by the face detection unit of FIG. 7, and (b) is a face image slightly facing left detected by the face detection unit of FIG. FIG. It is a figure which shows the other face image which faced the front detected by the face detection part of FIG.

Explanation of symbols

1 Programmable display (operation display device)
2 Display 3 Touch Panel 8 HMI Processing Unit 92 Face Log Execution Unit 92b Face Detection Unit (Front Image Determination Unit, Face Image Storage Unit)
92c face logging unit (history information storage means)
95 Face log file (second storage area)
96 Log display execution part (history information display means)
98 face image database (first storage area)
100 face detection unit (front image determination means)
110 Face detection unit (front image determination means)
111 Facial component detection unit (facial component detection means)
112 Inclination degree calculation part (Inclination degree calculating means)
113 Front degree determination unit (front determination means)
201 Face log display window

Claims (8)

In the operation display device that accepts an operation by the operator on the screen showing the operation content, and stores the captured image of the operator in association with the history information of the predetermined operation by the operator,
Front image determination means for determining whether or not a photographed image of the operator includes a front face image facing the front with respect to the operation surface of the operation display device in response to a predetermined operation by the operator;
When it is determined that the image includes the front face image, face image storage means for cutting out a portion of the front face image from the image and storing it in a first storage area;
History information storage means for storing the history information in a second storage area in a state associated with the front face image so that the front face image can be read based on the history information. A characteristic operation display device.   There is provided history information display means for reading the front face image stored in the first storage area based on the history information read from the second storage area and displaying it together with the history information. The operation display device according to claim 1.   The front image determination unit includes the front face image of an operator image captured during a predetermined period before and after the predetermined operation among operator images continuously captured during a preset period. The operation display device according to claim 1, wherein the operation display device determines whether or not.   The front image determination unit determines that the image includes the front face image when a predetermined facial part is detected from the image among eyes, nose, mouth, eyebrows, and ears. The operation display device according to any one of claims 1 to 3. The front image determination means includes a face part detection means for detecting two or more face parts among eyes, nose, mouth, eyebrows and contours from the image;
Of the detected face parts, in the combination of two face parts, the slope calculation for calculating the slope indicating the deviation of the relative positions of the face parts of each combination from the relative positions of the face parts in the frontal face. Means,
When the slope of at least one of the selected combinations is within a predetermined range, and the front index set in advance for the combination is greater than or equal to a reference value, the image is the front face The operation display device according to any one of claims 1 to 3, further comprising a front face determination unit that determines that an image is included.   When the front index of any combination among the selected combinations is less than the reference value, the front face determination unit sums up the front indexes of each selected combination, and the total value is equal to or greater than the reference value. 6. The operation display device according to claim 5, wherein if there is, it is determined that the image includes a front face image.   An operation display program for operating the operation display device according to any one of claims 1 to 6, wherein the operation display program causes a computer to function as each means.   A computer-readable recording medium on which the operation display program according to claim 7 is recorded.

Images