JP4201644B2 - Terminal device and control program for terminal device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a terminal device including an input unit and a display unit and a control program for the terminal device, and more particularly to a terminal device and a control program for the terminal device that can improve operability for an operator.
[0002]
[Prior art]
In general, a terminal device such as a personal computer / portable information terminal device / terminal device including a display unit such as an ATM in a financial institution can arbitrarily change a display screen by a user or an operator. For example, the personal computer can arbitrarily set the screen resolution / character size / background image or the like according to the preference of the operator.
[0003]
On the other hand, in recent computer systems, in order to improve the user interface, a technique for detecting the movement of the line of sight of an operator instead of an input device such as a keyboard or a mouse has been proposed, and as a document describing these techniques, for example, JP-A-2003-15816 and JP-A-11-345073 are exemplified.
[0004]
[Problems to be solved by the invention]
Although the terminal device according to the above-described prior art can arbitrarily change the display screen setting by the operator's operation, consideration has been given to automatically changing the display screen according to the operator's fatigue level, for example. Therefore, there is a problem that it is difficult to improve the operability of the user-friendly terminal device.
[0005]
On the other hand, the prior art described in the above publication can detect the operator's line of sight, for example, determine whether the driver of the automobile is facing the front, perform personal authentication, or use the line of sight instead of the mouse pointer. Although it was possible, there was no consideration such as automatically changing the display screen according to the fatigue level of the operator.
[0006]
An object of the present invention is to eliminate the above-described problems caused by the prior art, and it is possible to detect a natural operator action (referred to as an event) due to operator fatigue and improve operability in accordance with this event. It is to provide a terminal device and a control program for the terminal device.
[0007]
[Means for Solving the Problems]
In order to achieve the above object, the present invention provides an input means for inputting data and the like, a display means for displaying character data and an image, and a photographing means for photographing an operator's face image arranged at a basic position in front of the display means. And a storage means for storing the photographed face image, and a control means for controlling the input means, display means, photographing means and storage means , wherein the control means stores the operator stored in the storage means. A first step of obtaining the operator's face length dimension value and lip width dimension value from the face image, and obtaining a reference lip width ratio obtained by dividing the lip width dimension value by the face length dimension value, and photographing the operator's face image A second step of photographing by means, and a face longitudinal dimension value and a lip width dimension value of the operator obtained from the face image photographed by the second step, and the lip width dimension value is divided by the face longitudinal dimension value at the time of photographing. Third work to obtain lip width ratio And a fourth step for comparing the reference lip width ratio obtained in the first step with the photographing amplitude ratio obtained in the third step, and the photographing lip amplitude ratio in the fourth step has a large reference lip width ratio. When it is determined , the first feature is to execute a fifth step of generating a yawning event of the operator and a sixth step of changing the display form of the display means according to the yawning event of the fifth step. .
[0008]
The present invention, in the terminal apparatus of the first aspect, wherein, when it is detected a predetermined number of times or more in succession an operator of yawning event of the fifth step, the display unit, the second characterized by changing the output of the audio unit And
[0009]
Furthermore, the present invention provides an input means for inputting data and the like, a display means for displaying character data and an image, a photographing means for photographing a face image of an operator arranged at a basic position on the front surface of the display means, and the photographed face A control program for a terminal device comprising storage means for storing an image, control means for controlling the input means, display means, photographing means, and storage means, wherein the control means stores an operator's program stored in the storage means. A first function for obtaining an operator's face length dimension value and lip width dimension value from the face image, and obtaining a reference lip width ratio obtained by dividing the lip width dimension value by the face length dimension value; The second function for photographing by the above-mentioned method, and the operator's face vertical dimension value and lip width dimension value are obtained from the face image photographed by the second function, and the lip at the time of photographing is obtained by dividing the lip width dimension value by the face vertical dimension value The third function to get the width ratio A fourth function for comparing the reference lip width ratio obtained by the first function and the shooting amplitude ratio obtained by the third function, and in the fourth function, the shooting amplitude ratio is determined to be larger than the reference lip width ratio. Then, the third feature is to realize a fifth function for generating a yawning event by the operator and a sixth function for changing the display form of the display unit in accordance with the yawning event by the fifth function.
[0010]
According to the present invention, in the control program of the third feature, when the control means detects the yawning event of the operator by the fifth function continuously more than a predetermined number of times, the output of the display means and the sound means is changed. Implementing the seventh function is a fourth feature.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment of a terminal device and a control program according to the present invention will be described in detail with reference to the drawings. 1 is an external view of a terminal device according to the present embodiment, FIG. 2 is a diagram for explaining a hardware configuration of the terminal device according to the present embodiment, and FIG. 3 is a software configuration of the terminal device according to the present embodiment. FIG. 4 is a diagram for explaining face data according to the present embodiment, FIG. 5 is a diagram showing an event determination table according to the present embodiment, and FIG. 6 is a diagram for explaining an event determination algorithm according to the present embodiment. FIG. 7 is a diagram for explaining the overall operation of event processing according to the present embodiment.
[0013]
<Description of configuration>
First, the event described in the present embodiment is an operation that an operator instinctively unconsciously due to factors such as fatigue. For example, the screen is difficult to see due to eye fatigue, so “squinting eyes” / “face on screen” “Move closer”, “Keep your face away from the screen” due to presbyopia and presbyopia, “Yawning” / “Hold your eyes with hands” / “Stretch out” / “Close your eyes (sleep)” / “Neck your neck” It means unconscious actions such as "turn", "crush the neck with hand", "crush or hit the shoulder with hand", but if it is caused by human fatigue / weakness / stiff shoulders, etc. However, the present invention is not limited to this, and includes, for example, personal wrinkles, such as wrinkles that rotate a pencil when frustrated / poor shaking wrinkles (requires a camera for photographing the foot portion).
[0014]
Next, as shown in FIG. 1, the terminal device 2 according to the present embodiment includes a camera (photographing means) 3 for photographing a face image in front of the operator 1 and a display (display means) 4 for displaying various information. And an input device (not shown) such as a keyboard, the hardware configuration temporarily stores display information to be displayed on the display 4 in addition to the camera 3 and the display 4 as shown in FIG. The video memory 107, the input means 103 that is an input device such as a keyboard, the storage unit 104 that stores the operating system, various application software, and the control program / face data table that is a feature of the present embodiment, and the operating system, etc. A memory 106 for temporarily storing the data and a central processing unit (control unit) 105 for controlling them. Obtain.
[0015]
Also, the software configuration of the terminal device according to the present embodiment includes the operating system 201 and a display form control unit that controls the display form of display information displayed on the display 4 under the control of the operating system 201, as shown in FIG. 205, a display driver 206, a screen saver 207 that temporarily switches the display screen to another screen, a word processor / spreadsheet and other various application software 208, and information input from the input means 103 to the operating system 201. An input driver 214 for inputting, and an event input processing application 2 for performing input processing of events (: operation commands such as screen enlargement / reduction / screen saver activation / music playback, etc.) input via the input driver 214 3, a camera driver 216 that controls image capturing by the camera 3, an image recognition application 217 that recognizes an image captured by the driver 216, and an event recognition processing application that recognizes an event based on the image recognized by the application 217 218, an image transfer application 215 that transfers a photographed image taken by the camera driver 216 to the operating system 201, and a face feature extraction application 211 that extracts facial image features based on the image information transferred by the application 215. A basic face image registration application 212 for registering face image information of the operator in advance.
[0016]
As shown in FIG. 4, the face data table stored in the storage unit has a face width (width) dimension W, a face length dimension h, an eye length (width) dimension e, and a lip length. When the (thickness) dimension is m, in addition to the dimension W / h / e / m, the eye width ratio e / h, which is the ratio of the face vertical dimension to the eye vertical dimension, and the face vertical dimension And the lip width ratio m / h, which is the ratio of the lip length to the vertical length (width as thickness) dimension. These dimensions and ratios are reference values for the operator.
[0017]
As shown in FIG. 3, this face data is obtained when the camera driver 216 drives the camera 3 and the operator sits on the front of the display in a normal state and sits at a position (called a basic position) when performing work. A front face image is taken and stored in the storage unit 104 of FIG. 2 by the basic face image registration application 212 via the operating system 201 by the image transfer processing application 215, and the face feature extraction application 211 of FIG. Each dimension is extracted or calculated from image information and stored in the storage unit 104 of FIG. In addition, the terminal device and the control program according to the present embodiment capture not only the face image but also the upper body when the operator is in a standard position when operating the terminal device, and the face position / shoulder position / hand position, etc. Can be stored in the storage unit 104 after image analysis. In this embodiment, an image (face image, upper body image, etc.) taken at the basic position of the operator is called a basic image in the present embodiment, and the face image, upper body image, and other images taken by the camera during the operation of the operator Called an image.
[0018]
The reason why the eye width ratio e / h and the lip width ratio m / h of the face data are stored is that the measurement width of the imaging changes depending on the face up / down direction / approach / distance. This is to prevent the change in the eye width and the lip width from being accurately measured in comparison with the absolute value.
[0019]
In particular, in the terminal device and the control program according to the present embodiment, the natural operation of the operator, for example, when the eye width ratio e / h photographed by the camera is smaller than the reference eye width ratio e / h stored in the storage unit 104. When the operator detects that the eyes are narrowed and the eyes are closed, and the photographed lip width ratio m / h is larger than the reference lip width ratio m / h, the operator opens his mouth. When the horizontal W and the vertical h of the photographed face are both larger than the reference value, it is detected that the operator is approaching the screen. Conversely, when the face is small, the operator removes the face from the screen. It is possible to detect being away.
[0020]
Further, the terminal device and the control program according to the present embodiment, for example, have narrowed eyes / closed eyes / open mouth / closed face / closed face based on the captured face image. And the occurrence of the operator's event can be determined using the detection result. For example, if the number of times that it is detected that the eyes are narrowed within a predetermined time exceeds the predetermined number of times, it is determined that an event of narrowing the eyes for a long time due to fatigue has occurred, and the number of times the eyes are closed If it is detected that the second predetermined number of times has been exceeded, it is determined that a dozing event may have occurred, and if the mouth has been opened for a predetermined time greater than a predetermined ratio, a yawning event has occurred. It is judged that there is a possibility that it is, and other things such as “stretch out” / “turn the neck” / “crush the neck with hands” / “crush or hit the shoulders with hands” / “hold the eyes with hands” etc. An event can be determined by comparing a basic action with a basic image registered in advance.
[0021]
Further, as shown in FIG. 5, the event determination table described in the present embodiment has three image change conditions (face vertical / horizontal enlargement and eye vertical / height ratio reduction) to determine an occurrence event (difficult to see / dissolve difficult to see / yawn event). Each of the operation counters is provided in accordance with the detection of (face vertical / horizontal reduction and eye vertical / height ratio expansion / face vertical / horizontal reduction and mouth vertical / longitudinal ratio expansion), and the counter value is added or subtracted and compared with a predetermined threshold The detailed operation will be described later.
[0022]
<Description of operation>
Next, operations of the terminal device and the control program according to the present embodiment configured as described above will be described below.
First, as shown in FIG. 6, the event determination algorithm by the event recognition application 218 shown in FIG. 3 is an event in which the operator approaches the screen and the vertical and horizontal dimensions of the face image increase and the eyes are narrowed. It is determined in step 61 whether or not it has occurred. The determination at step 61 is performed by determining whether or not the image change condition 1 of the conditional expression “the face width w is enlarged and the face length h is enlarged and the eye length ratio e / h is reduced” is satisfied. When an event satisfying the image change condition 1 in step 61 is detected, the present algorithm adds 1 to the counter value of “difficulty event” (step 62) and determines whether or not the counter value n exceeds 100 times. If it is determined in step 63 that the counter value n has exceeded 100, the counter value n is reset to zero (step 64), and it is determined that an “unobservable event” has occurred ( In step 65), if the counter value n does not exceed 100 in step 63, the image change condition 2 is determined in the next step 66.
[0023]
In other words, when an event that causes the operator to approach the screen and narrow his / her eyes occurs 100 times or more according to the image change condition 1 according to the present embodiment, it is determined that an “unobservable event” has occurred due to fatigue. The determination of the event occurrence is to count the number of occurrences of “difficulty event” within a unit time (for example, 30 seconds to 10 minutes). In the determination of the counter value of the image change condition in the example described later. Are also set in the same manner although the threshold values are different.
[0024]
In step 66 for determining the image change condition 2, whether or not an event has occurred in which the vertical and horizontal dimensions of the face image are reduced and the eyes are widened as the operator moves away from the screen as compared with the basic position. This is performed by determining whether or not the image change condition 2 of the conditional expression “the face width w is reduced and the face length h is reduced and the eye length ratio e / h is increased” is satisfied.
[0025]
When an event satisfying the image change condition 2 in step 66 is detected, the present algorithm subtracts 1 from the counter value of “difficulty event” (step 67), and whether or not the counter value n is smaller than 100 times. If the counter value m is determined to be smaller than −100 in step 68, the counter value m is reset to zero (step 69), and an “invisibility cancellation event” occurs. (Step 70). If the counter value n is larger than −100 in Step 68, the image change condition 3 is determined in the next Step 71.
[0026]
Next, the present algorithm determines whether or not an event that satisfies the image change condition 3 in step 71 has occurred. In step 71 for determining the image change condition 3, it is determined whether or not an event has occurred in which the vertical and horizontal dimensions of the face image are reduced and the mouth is wide open due to the operator moving away from the screen (display). Do. The determination in step 71 is performed by determining whether or not the image change condition 3 of the conditional expression “the face width w is reduced and the face length h is reduced and the mouth length ratio m / h is increased” is satisfied. When an event satisfying the image change condition 3 in step 71 is detected, the present algorithm subtracts 1 from the counter value of “yawn event” (step 72), and determines whether or not the counter value m is smaller than 100. When the counter value m is determined to be smaller than 100 in step 73, the counter value m is reset to zero (step 74) and it is determined that the “yawn event” has occurred (step 75). . That is, this algorithm counts the number of times per unit time (in this example, for example, 10 seconds) that is far away from the screen due to the operator's back stretching, etc., and is 100 times or more within the unit time (10 seconds). If it is detected that the mouth is far away and the mouth is open, it is determined that a “yawn event” has occurred.
[0027]
The event determination algorithm shown in FIG. 6 is an example of an “event that closes the screen and narrows the eyes” and “yawn event” by the operator, and the above-mentioned presbyopia causes “the face is moved away from the screen”. / "Hold your eyes with your hands" / "Extend your back" / "Close your eyes (sleep)" / "Turn your neck" / "Pinch your neck with your hand" / "Pinch or hit your shoulder with your hand" / "Personal The determination of “行 う” can be performed, and this determination can be performed by comparison with image information captured and registered in advance.
[0028]
Next, as shown in FIG. 7, the terminal device and control program according to the present embodiment using the above-described event determination algorithm is a movement in which the CPU 105 executes various event determination algorithms as described in FIG. 6 at regular intervals. The recognition event 301 and the input event 302 by the operator's own operation are executed alternately, and the contents of the detected or input event are analyzed by the event analysis processing 303. When the analysis result of the event analysis process 303 is an event 308 that moves away from the face, the system performs a display reduction process 304 that reduces and displays the display image in order to cope with hyperopia. When the event 310 is close (motion), a display enlargement process 305 is performed to enlarge and display the display image in order to deal with myopia. When the yawning event 311 occurs, In the case of the event 312 or 313 of direct instruction by the mouse or key input from the operator, the normal processing 307 (screen enlargement / reduction / screen / increase / reduction / screen reduction) is performed. Screen saver / music playback etc.)
[0029]
The display enlargement process 304 / display enlargement process 305 does not change the reduction / enlargement ratio abruptly, but changes the reduction / enlargement ratio gently, so that, for example, if the characters on the screen are small and difficult to recognize, the character size Is gradually enlarged, and when the character size suitable for the operator is reached, the operator returns to the reference position, thereby stopping the character enlargement and preventing the operator from being aware of a sudden screen change.
[0030]
As described above, the terminal device and the control program according to the present embodiment capture and store in advance an operator's standard face image and other basic images, and detect events that the operator instinctively unconsciously due to factors such as fatigue. And by changing the display form of an image according to this event, for example, a display screen can be changed automatically according to an operator's fatigue degree, and the operativity of a user-friendly terminal device can be improved.
[0031]
More specifically, the terminal device and the control program according to the present embodiment detect this event when the operator approaches the screen because, for example, the characters are small and difficult to see, and display the display screen so that the display screen is gradually enlarged. To provide a user-friendly terminal device that performs control, for example, when the operator yawns, can activate a screen saver or play music to encourage the operator to rest or change mood it can.
[0032]
In the above-described embodiment, an example in which the display screen magnification is changed or the screen saver is activated for various events has been described. However, other processing may be executed depending on the type of event. For example, when the operator closes his eyes for a long time and is likely to fall asleep, the entire screen is flashed and a warning sound is generated, or the operator “turns his neck” / “rubs his neck with his hand” / When a stiff shoulder event such as “grabbing or hitting the shoulder by hand” is detected, a video image of a light exercise manner while being present can be displayed and various other corresponding processes can be executed.
[0033]
Furthermore, the present invention, for example, in the case where the operator's face-down state continues for a predetermined time and the operator does not change even if the flash is executed a predetermined number of times, in view of the possibility of an emergency due to illness, etc. Processing such as reporting to an emergency center can also be performed, and this can be implemented as a function of an emergency call in a terminal device used by an elderly person living alone, for example.
[0034]
Furthermore, the present invention is not limited to a specific operator image information to be registered in advance, it is also possible to register the image information of a plurality of operators, in this case, when the operator is present in the terminal device It is necessary to make a determination by inputting a logon ID or image recognition. The present invention is not limited to a terminal device such as a personal computer, but can also be applied to, for example, ATMs of financial institutions. When applying to this ATM, since the operator is unspecified, for example, the area within the contour is calculated from the contour of the entire image of the face or upper body where the operator has entered the camera field of view and stopped, and this memory is stored. When the operator approaches the contour area and the area of the contour of the captured image of the face or upper body increases, it is determined that a hard-to-see event has occurred and the guidance image is enlarged or characters are enlarged. It may be configured.
[0035]
Furthermore, in an unspecified operator, for example, the eyes are the upper and horizontal lines arranged on the left and right sides, the nose is the vertical line arranged on the center of the face, and the mouth is the lower and central lines of the face. By extracting the operator's eyes, nose, and mouth from features such as minutes, these changes in position, size, etc. are determined, and an event of operator movement is generated, thereby enlarging the displayed image. Alternatively, reduction control may be performed.
[0036]
【The invention's effect】
As described above, the terminal device and the control program according to the present invention are provided with the photographing means for photographing the basic image of the operator arranged at the basic position on the front surface of the display means, and the storage means for storing the photographed basic image, The control means takes an image of the operator by the photographing means, compares the photographed operator image with the basic image stored in the storage means, determines the occurrence of an operator event based on the comparison result, and determines By changing the display form of the display means according to the event, when an event occurs, for example, an event where the operator brings his face close to the screen, the enlargement ratio of the screen is increased to make it easier to read characters, etc. When an event occurs, a screen saver can be activated to prompt a change of mood, providing a user-friendly terminal device. Can.
[Brief description of the drawings]
FIG. 1 is an external view of a terminal device according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining a hardware configuration of a terminal device according to the present embodiment.
FIG. 3 is a diagram for explaining a software configuration of the terminal device according to the embodiment;
FIG. 4 is a view for explaining face data according to the present embodiment.
FIG. 5 is a diagram showing an event determination table according to the present embodiment.
FIG. 6 is a diagram for explaining an event determination algorithm according to the present embodiment.
FIG. 7 is a diagram for explaining the overall operation of event processing according to the present embodiment;
[Explanation of symbols]
1: operator, 2: terminal device, 3: camera, 4: display, 103: input means, 104: storage unit (magnetic disk storage device), 105: CPU, 106: memory, 107: video memory, 201: operating system 205: Display form control unit, 207: Screen saver, 208: Application program, 211: Face feature extraction application, 212: Basic face image registration application, 213: Event input processing application, 214: Input driver, 215: Image transfer processing application 216: Camera driver, 217: Image recognition processing application, 218: Event recognition processing application.

Claims (4)

Input means for inputting data and the like, display means for displaying character data and images, photographing means for photographing an operator's face image arranged at a basic position on the front surface of the display means, and storage for storing the photographed face image A terminal device comprising: means; and control means for controlling the input means, display means, photographing means, and storage means ,
The control means is
A first step of obtaining an operator's face length dimension value and lip width dimension value from the operator's face image stored in the storage means, and obtaining a reference lip width ratio obtained by dividing the lip width dimension value by the face length dimension value; ,
A second step of photographing an operator's face image by photographing means;
A third step of obtaining a face length dimension value and a lip width dimension value of an operator from the face image photographed in the second step, and obtaining a lip width ratio at the time of photographing obtained by dividing the lip width dimension value by the face length dimension value; ,
A fourth step of comparing the reference lip width ratio obtained in the first step and the shooting amplitude ratio obtained in the third step;
A fifth step of generating an operator yawning event when the shooting amplitude ratio is determined to be large in the fourth step in the fourth step;
A terminal device that executes a sixth step of changing a display form of the display means in response to a yawning event in the fifth step . The terminal device according to claim 1, wherein when the yawning event of the operator in the fifth step is continuously detected a predetermined number of times or more, the output of the display means and the voice means is changed . Input means for inputting data and the like, display means for displaying character data and images, photographing means for photographing an operator's face image arranged at a basic position on the front surface of the display means, and storage for storing the photographed face image A terminal device control program comprising: means; and input means, display means, photographing means, and control means for controlling storage means,
  In the control means,
  A first function for obtaining an operator's face length dimension value and lip width dimension value from the operator's face image stored in the storage means, and obtaining a reference lip width ratio obtained by dividing the lip width dimension value by the face length dimension value; ,
  A second function for photographing an operator's face image by photographing means;
  A third function for obtaining a face length dimension value and a lip width dimension value of an operator from a face image photographed by the second function, and obtaining a lip width ratio at the time of photographing by dividing the lip width dimension value by the face length dimension value; ,
  A fourth function for comparing the reference lip width ratio obtained by the first function and the shooting amplitude ratio obtained by the third function;
  A fifth function for generating an operator yawn event when it is determined in the fourth function that the shooting amplitude ratio is large in the reference lip width ratio;
  A control program for a terminal device for realizing a sixth function for changing a display form of a display unit in response to a yawn event by the fifth function. In the control means,
The control program for a terminal device according to claim 3, wherein a seventh function for changing the output of the display means and the voice means is realized when the yawning event of the operator by the fifth function is continuously detected a predetermined number of times or more .

Images