JP2010152798A - Automatic processing apparatus, automatic processing method and automatic processing program corresponding to visual line recognition - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To recognize the destination of a user's visual line and to call for the user's attention based on the recognized visual line. <P>SOLUTION: An automatic teller machine 10 has: a face image acquiring section 205 for acquiring a face image of the user U; an eye detector 210 for detecting the state of the eyes of the user U based on the facial image; a recognition section 220 for recognizing the direction of the visual line based on the detected state of the eyes; a face detector 215 for detecting the state of the user's face based on the face image; a calculating section 310 for calculating whether the destination of the user U's visual line is directed to the automatic teller machine 10 based on the detected state of the face and the recognized direction of the visual line; a storing section 315 for storing a present step which the automatic teller machine 10 carries out and a next step which is to be carried out next; and a determination section 320 for determining whether or not to call for the user U's attention based on the destination of the visual line of the user. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an automatic processing device, an automatic processing method, and an automatic processing program according to line-of-sight recognition. In particular, the present invention relates to an automatic processing device, an automatic processing method, and an automatic processing program that perform desired processing while alerting the user according to the line of sight of the user who uses the automatic processing device.

  Conventionally, various techniques relating to gaze detection and gaze recognition have been devised (see, for example, Patent Documents 1 and 2). Patent Document 1 discloses a system that detects the direction of the line of sight and controls the cursor on the display according to the detected direction of the line of sight. Patent Document 2 discloses a system that detects the line of sight of an operator gazing at display means and performs predetermined editing processing control based on the detected movement of the line of sight.

  Techniques have also been proposed in which this line-of-sight detection and line-of-sight recognition technology is applied to ATM (Automatic Teller Machine) (see, for example, Patent Documents 3 to 6). For example, Patent Document 3 discloses a system that detects a customer's line-of-sight direction and displays a three-dimensional image on an ATM screen in the detected line-of-sight direction. Patent Document 4 discloses an ATM cash forgetting alarm system that tells a user who has forgotten to withdraw cash or the like and leaves the place to the user. Patent Document 5 discloses a system that learns a pattern of transaction contents that a customer repeatedly operates, predicts transaction contents for each customer based on the learning result, and displays an operation guidance screen for transaction contents that matches the predicted contents. ing. Patent Document 6 discloses a technique for detecting whether a user's eyes are open from an image around the user's eyes and detecting the user's line-of-sight direction when the user's eyes are open. Yes.

JP 05-298015 A Japanese Patent Application Laid-Open No. 07-191796 JP 2005-301484 A JP 2002-117429 A Japanese Patent Laid-Open No. 09-062747 JP 2003-015816 A

  However, when calling attention to an ATM user, in the above technique, an operation to call attention from the ATM side is executed only when the user forgets a medium such as cash or a passbook. There was a problem that it was not possible to perform an action to call attention.

  Moreover, regarding the technique of changing the operation guidance screen based on the learning result of the transaction pattern for each customer, there is a problem that the customer does not always perform the same operation periodically.

  Therefore, the present invention provides an automatic processing device that detects and recognizes the direction of the user's line of sight and alerts the user in a predetermined case based on the recognized line of sight.

  In order to solve the above problems, according to an aspect of the present invention, an automatic processing device that recognizes a user's line of sight and executes desired automatic processing based on the recognized line of sight, A face image acquisition unit to acquire; an eye detection unit that detects a user's eye state based on the face image acquired by the face image acquisition unit; and a line of sight based on the eye state detected by the eye detection unit. A recognition unit for recognizing a direction; a face detection unit for detecting a face state of a user based on the face image acquired by the face image acquisition unit; and a state of the face detected by the face detection unit and the recognition unit Based on the direction of the line of sight recognized by the calculation unit that calculates whether the user's line of sight is directed to the automatic processing device, the current process in which the automatic processing device is operating, and the next operation schedule A storage unit for storing the next process of A determination section for determining attention to the user based on the relationship between the calculated user's line of sight before said automatic processing apparatus by parts, the automatic processing apparatus in accordance with the line of sight recognition with proposed.

  According to this configuration, it is calculated whether the user's line of sight is directed to the automatic processing device based on the face state of the user and the direction of the line of sight. In addition, as a result of the calculation, it is determined whether to alert the user depending on whether the user's line of sight is directed toward the automatic processing device or other than the device. This not only executes an action to call attention from the automatic processing device to things that happened in the past, such as forgetting the medium such as cash and bankbook, but also, for example, paying attention to the user during the operation. It is possible to assist the user in advance, such as prompting, alerting the user proactively so as not to forget it in the future, or substituting for guidance for the next operation. As a result, effects such as improved operability with respect to the automatic processing device and shortened transaction time can be achieved.

  The recognizing unit may determine the presence or absence of blinking from the state of the eyes detected by the eye detecting unit, and may recognize the direction of the line of sight when the eyes are open as a result of the determination.

  When the calculation unit recognizes that the blink is not blinking, the calculation unit detects the user's line of sight based on the direction of the line of sight and the distance from the face position detected by the face detection unit to the automatic processing device. It may be calculated which part of the automatic processing device is pointed to.

  According to the calculation, the determination unit is configured such that the user's line of sight is not directed to a part of the automatic processing apparatus that the user wants to operate in the current process or the next process stored in the storage unit. In this case, it may be determined that the user should be alerted.

  According to the calculation, the determination unit is configured such that the user's line of sight is directed to a part of the automatic processing device that the user wants to operate in the current process or the next process stored in the storage unit. In this case, it may be determined that the current process or the next process stored by the storage unit is processed as usual.

  The determination unit may specify a type of attention to be urged to the user according to a point of the user's line of sight.

  You may further have a warning part which warns according to the kind of attention specified by the said determination part.

  The warning unit may warn by at least one of display of a screen attached to the automatic processing device, output of sound, lighting of a warning light, or blinking of a warning light.

  The warning unit may warn by at least one of outputting sound or blinking the screen when the user's line of sight is not directed to the automatic processing device.

  The warning unit may enlarge the display of the user's line of sight when the user's line of sight is directed to the automatic processing device.

  The automatic processing apparatus may be an automatic deposit / withdrawal machine.

  The face image acquisition unit may acquire a user's face image captured using a plurality of cameras installed on an upper portion of the automatic depositing and dispensing machine.

  In order to solve the above problem, according to another aspect of the present invention, there is provided an automatic processing method for recognizing a user's line of sight and executing desired automatic processing based on the recognized line of sight. Acquiring a face image; detecting a user's eye state based on the acquired face image; recognizing a gaze direction based on the detected eye state; and the acquired The step of detecting the face state of the user based on the face image, and the point of the user's line of sight is directed to the automatic processing device based on the detected face state and the recognized line-of-sight direction. A step of calculating whether the current process in which the automatic processing apparatus is operating and a next process scheduled to be operated next are stored in the storage unit, the calculated line of sight of the user, and the automatic processing apparatus Based on the relationship with Automatic processing method and determining whether alert the stored current process or the next step, or the user is processed normally, depending on the line of sight recognition including the serial storage unit is provided.

  According to another aspect of the present invention, there is provided an automatic processing program for recognizing a user's line of sight and causing a computer to execute desired automatic processing based on the recognized line of sight. A process of acquiring a user's face image, a process of detecting a user's eye state based on the acquired face image, and a process of recognizing the direction of the line of sight based on the detected eye state; Based on the process of detecting the face state of the user based on the acquired face image and the detected face state and the direction of the recognized line of sight, the point of the user's line of sight is the automatic processing device. A process of calculating whether the automatic processing apparatus is operating, a process of storing the next process scheduled to be operated next and a next process scheduled to be operated in the storage unit, and the calculated user's line of sight Based on the relationship with the automatic processor , Automatic processing program for executing a process of determining whether alert the current step or next step stored by the storage unit in or user to process normally the computer is provided.

  As described above, according to the present invention, the direction of the user's line of sight can be detected and recognized, and the user can be alerted in a predetermined case based on the recognized line of sight.

BEST MODE FOR CARRYING OUT THE INVENTION

  Hereinafter, an embodiment for carrying out the invention will be described with reference to the accompanying drawings. However, in the present specification and drawings, components having substantially the same function are denoted by the same reference numerals, and redundant description is omitted.

In addition, according to the order shown below, this embodiment is described taking an automatic depositing and dispensing machine as an example of an automatic processing device.
[1] Overall configuration of automatic deposit / withdrawal machine [2] Functional configuration of automatic deposit / withdrawal machine [3] Gaze recognition / automatic deposit / withdrawal processing

[1] Overall Configuration of Automatic Deposit / Withdrawal Machine First, an automatic deposit / withdrawal machine (ATM) according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a diagram schematically showing the overall configuration of the automatic depositing and dispensing machine 10. The automatic depositing / dispensing machine 10 is an example of an automatic processing device that recognizes the line of sight of the user U and executes desired automatic processing based on the recognized line of sight.

  The apparatus main body 100 of the automatic depositing / dispensing machine 10 includes a card reader unit 105, a passbook printer unit 110, a banknote depositing / dispensing unit 115, a coin depositing / dispensing unit 120, flicker lighting units 125a, 125b, 125c, 125d, a screen display unit 130 ( Display) and an audio output unit 135 (speaker). In addition, two cameras 140 a and 140 b are attached to the apparatus main body 100.

  The card reader unit 105 reads the cash card inserted by the user U. The bankbook printer unit 110 prints transaction contents on the bankbook inserted by the user U. The banknote depositing / withdrawing unit 115 deposits and withdraws the banknote of the user U. The coin deposit / withdrawal unit 120 deposits / withdraws user U's coins. The flicker lighting units 125a, 125b, 125c, and 125d are provided on the card reader unit 105, the passbook printer unit 110, the banknote deposit / withdrawal unit 115, and the coin deposit / withdrawal unit 120, respectively, depending on the function of each unit. In order to call attention to the user U, the flicker lamp is turned on or blinked at an appropriate color or blinking speed. The screen display unit 130 displays an operation guidance screen for transaction details. The screen display unit 130 displays a message (word) or an image (image) that calls attention to the user U in a predetermined case. The voice output unit 135 outputs a voice to alert the user U in a predetermined case. The cameras 140a and 140b capture the user U in order to detect the position of the user U's face and the eyeballs and line of sight. The cameras 140a and 140b are attached to the upper part of the apparatus main body 100 so as to easily capture the user U's face image.

[2] Functional configuration of automatic deposit / withdrawal machine Next, the functional configuration of the automatic deposit / withdrawal machine 10 will be described with reference to FIG. FIG. 2 is a block diagram showing a functional configuration of the automatic depositing and dispensing machine 10. The automatic deposit / withdrawal machine 10 includes a line-of-sight information acquisition unit 200 and an automatic deposit / withdrawal unit 300. The line-of-sight information acquisition unit 200 detects and recognizes the direction of the line of sight of the user U from images captured by the cameras 140a and 140b. The automatic deposit / withdrawal unit 300 acquires information on the line of sight from the line-of-sight information acquisition unit 200, performs normal deposit / withdrawal processing on the user U, and, in a predetermined case based on the acquired information on the line of sight, the user U Process to call attention to.

  The line-of-sight information acquisition unit 200 includes a face image acquisition unit 205, an eye detection unit 210, a face detection unit 215, and a recognition unit 220. The recognition unit 220 includes an eyeball recognition unit 220a and a line-of-sight recognition unit 220b.

  The face image acquisition unit 205 acquires the face image of the user U. The eye detection unit 210 detects the eye state of the user based on the face image acquired by the face image acquisition unit 205. The eye detection unit 210 detects, for example, the position and orientation of the user U's eyes and whether the eyes are open from the image area around the eyes of the acquired face image. The face detection unit 215 detects the face state of the user based on the face image acquired by the face image acquisition unit 205. For example, the face detection unit 215 detects the orientation and position of the face from the acquired face image.

  The recognition unit 220 recognizes the direction of the line of sight based on the eye state detected by the eye detection unit 210. Specifically, the eyeball recognition unit 220a acquires the characteristics of the eyeball (eye) so that erroneous recognition due to blinking or the like does not occur. The line-of-sight recognition unit 220b determines the presence or absence of blinking from the state of the eyes detected by the eye detection unit 210, and recognizes the direction of the line of sight when the eyes are open as a result of the determination. For example, the line-of-sight recognition unit 220b detects the position of the pupil from the brightness and darkness of the acquired image around the eye (eyeball), and detects the line-of-sight direction from the detected center position of the pupil and the center position of the eyeball.

  The line-of-sight information cooperation unit 305 determines and executes processing according to the direction of the line of sight of the user U on the automatic deposit / withdrawal unit 300 side, so that information on the line of sight acquired from the line-of-sight information acquisition unit 200 It is an interface for passing to 300.

  In the automatic deposit / withdrawal unit 300, the information obtained through the line-of-sight information linkage unit 305 is compared with the processing operation (current process) of the automatic depositing / dispensing machine 10 at that time, and the processing operation of the automatic depositing / dispensing machine 10 performed next time. It is determined whether the line of sight of the user U and the (next process) match. If they do not match, the automatic depositing / dispensing machine 10 is controlled to perform an operation for prompting the line of sight of the user U. .

  The automatic deposit / withdrawal unit 300 includes a calculation unit 310, a storage unit 315, a determination unit 320, a process execution unit 325, and a warning unit 330. Each function of the automatic deposit / withdrawal unit 300 will be described below.

  The calculation unit 310 determines whether the line of sight of the user U is directed to the automatic depositing / dispensing machine 10 based on the face state detected by the face detection unit 215 and the direction of the line of sight recognized by the recognition unit 220. calculate. More specifically, the calculation unit 310 calculates the position and orientation of the face detected by the face detection unit 215, the presence or absence of blinks recognized by the recognition unit 220, the direction of the line of sight, and the position of the face detected by the face detection unit 215. Based on the distance to the automatic depositing / dispensing machine 10, it is calculated to which part of the automatic depositing / dispensing machine 10 the point of view of the user U is directed.

  The storage unit 315 stores all processes in which the automatic depositing / dispensing machine 10 operates and stores and manages the current process in which the automatic depositing / dispensing machine 10 is operating and the next process scheduled to be operated next.

  The determination unit 320 determines whether to alert the user U based on the relationship between the line of sight of the user U calculated by the calculation unit 310 and the automatic depositing / dispensing machine 10. For example, the determination unit 320 urges the user to pay attention when the point of view of the user U is not directed to the part of the automatic depositing / dispensing machine 10 that the user U wants to operate in the current process or the next process. judge. On the other hand, when the point of view of the user U is directed to the part of the automatic depositing / dispensing machine 10 that the user U wants to operate in the current process or the next process, the determination unit 320 determines the current process or the next process. Determine to process as usual. The determination unit 320 also identifies the type of attention to be urged to the user U according to the point of view of the user U.

  The process execution unit 325 performs a normal automatic deposit / withdrawal process. For example, when the user U requests deposit, the user U is prompted to insert cash into the banknote deposit / withdrawal unit 115 or the coin deposit / withdrawal unit 120, and the depositing process of the inserted cash is executed. For example, when the user U requests entry of a passbook, the user U is prompted to insert the passbook into the passbook printer unit 110, and print processing of the inserted passbook is executed.

  The warning unit 330 warns according to the type of attention specified by the determination unit 320. The warning unit displays a message (word) or an image (image) on a screen display unit (display) 130 attached to the automatic depositing / dispensing machine 10, outputs a voice to a voice output unit (speaker) 135, and flickers. The user U is warned by at least one of lighting or flashing the lighting units 125a to 125d (warning lights).

  As a specific method of warning, for example, when prompting the user U to input necessary data in the screen display, the warning unit 330 outputs voice or flashes the screen when the user U is looking out of the screen. Thus, the line of sight of the user U may be directed to the screen, and when the user U is looking on the screen, the display of the line of sight may be enlarged. At that time, the word is enlarged when looking at the word, and the button is enlarged when looking at the button.

  In addition, for example, when the user U inserts or discharges any of a card, a passbook, and cash, the user U's line of sight is not directed to either the card, the passbook, or cash. The unit 330 may direct the user U's line of sight to the screen by outputting sound or turning on or blinking flicker. For example, when the user U's line of sight is not directed to the automatic depositing / dispensing machine 10, the warning unit 330 performs the operation of the next step (for example, inhaling) stored in the storage unit 315 and the above-described operation. The user U may be informed of the location of any of the card, bankbook, and cash inlets by outputting sound or turning on or blinking flicker.

  Further, for example, when the user U is looking at the screen, the warning unit 330 displays the wording on the screen to guide the location of the suction port and the guidance by the image video, and also performs the guidance by the voice at the same time. You may do it. The warning unit 330 can appropriately use these specific operations in combination for each process.

  In addition, the user's line of sight is recognized, and the user U is alerted in a predetermined case based on where the user U is looking, or the convenience of the user U is improved. The automatic processing apparatus that performs desired automatic processing while displaying a guide is not limited to the automatic depositing / dispensing machine 10 according to the present embodiment. For example, the automatic processing apparatus includes the line-of-sight information acquisition unit 200 and the line-of-sight information linkage unit 305 and automatic deposit / withdrawal. Any device having a desired automatic processing unit instead of the unit 300 may be used. Specifically, the automatic processing device may be a vending machine or various ticketing machines (air tickets, train tickets, etc.). In addition, the automatic processing device can be applied as a device that prevents a driver from looking away in an automobile.

[3] Gaze recognition / automatic deposit / withdrawal process Next, the gaze recognition / automatic deposit / withdrawal process executed by the automatic deposit / withdrawal machine 10 according to the present embodiment will be described with reference to FIG. FIG. 3 is a flowchart showing a procedure of line-of-sight recognition / automatic deposit / withdrawal processing.

  First, when the user U is expected to perform an operation by approaching the automatic depositing / dispensing machine 10, that is, the camera 140a, 140b always acquires the previous image of the automatic depositing / dispensing machine 10 and acquires it. When the face of the user U can be recognized in the image, the line-of-sight recognition / automatic deposit / withdrawal process is started from step S300.

  After the start, in step S305, the face image acquisition unit 205 acquires the face image of the user U. Next, the face detection unit 215 detects the position and orientation of the face from the face image acquired by the face image acquisition unit 205 in step S310. Next, in step S315, the eye detection unit 210 detects the position and orientation of the user U's eyes based on the acquired face image. The eyeball recognition unit 220a acquires the characteristics of the eyeball (eye) in step S320, and the line-of-sight recognition unit 220b acquires the face image acquired based on the detection result of the eye position and orientation detected by the eye detection unit 210. It is determined whether the eyes of the user U are open from the image area around the eyes.

  As a result of the determination, if it is determined that the eyes are not open (blinks), the process immediately proceeds to step S395, the present process is terminated, and the process is terminated so that erroneous recognition due to blinking or the like does not occur. On the other hand, if it is determined that the eyes are open (not blinking) as a result of the determination, the process advances to step S325, and the line-of-sight recognition unit 220b determines the direction of the pupil from the contrast of the acquired image around the eyes (eyeballs). And the line-of-sight direction is detected from the detected center position of the pupil and the center position of the eyeball.

  Subsequently, the process proceeds to step S330, where the calculation unit 310 calculates the distance from the user U to the automatic depositing / dispensing machine 10, and uses the calculated distance and the position of the face detected by the face detection unit 215. It is calculated whether the line of sight of the person U is directed to the automatic depositing / dispensing machine 10. When the line of sight of the user U is directed to the automatic depositing / dispensing machine 10, the calculation unit 310 further determines to which part of the automatic depositing / dispensing machine 10 the line of sight of the user U is directed. Is calculated.

  Information about the line of sight acquired by the line-of-sight information acquisition unit 200 is sent to the automatic depositing / withdrawing unit 300 via the line-of-sight information cooperation unit 305 and stored in the storage unit 315 in step S335. As a result, the storage unit 315 automatically calculates information on the user's line of sight from the face position, orientation, eye (eyeball) position, presence / absence of blinking, line-of-sight direction, and face position in the image. The distance between the depositing / dispensing machine 10 and the user U, information on which part of the automatic depositing / dispensing machine 10 the user U's line of sight is directed to, and the like are stored. The storage unit 315 stores all processes in which the automatic depositing and dispensing machine 10 operates in advance. The storage unit 315 manages the current process in which the automatic depositing / dispensing machine 10 is operating and the next process scheduled to be operated next.

  Next, the process proceeds to step S340, and the determination unit 320 makes the user perform the current step or the next step that the user U's line of sight wants to perform based on various information related to the user's line of sight stored in the storage unit 315. It is determined whether it is directed to the operation part. Specifically, the determination unit 320 determines whether the line of sight is directed toward the automatic depositing / dispensing machine 10 or other than the automatic depositing / dispensing machine 10. When the line of sight is directed to the automatic depositing / dispensing machine 10, the determination unit 320 operates the automatic depositing / dispensing machine 10 at that point of time in the portion in the automatic depositing / dispensing machine 10 to which the line of sight is directed. It is compared whether it corresponds to the operation part to be performed by the user U including (current process) to the next operation (next process).

  As a result, when it is determined that the user U's line of sight is directed to the portion of the automatic depositing / dispensing machine 10 that the user U wants to operate in the current process or the next process (that is, the line of sight When it is determined that the operation unit or the operation unit that the user U wants to perform matches), the process execution unit 325 performs a normal automatic deposit / withdrawal process according to the request of the user U in step S345, It progresses to step S395 and complete | finishes this process.

  On the other hand, when it is determined that the line of sight of the user U is not directed to the part of the automatic depositing / dispensing machine 10 that the user U wants to operate in the current process or the next process, the warning unit 330 performs step S350. After calling attention to the user U, the process proceeds to step S395 to end the present process. In addition, after alerting the user U, it may progress to step S345 and may perform a normal automatic deposit / withdrawal process.

  An example of a specific method of alerting executed in step S350 is shown in FIGS. For example, as shown in FIG. 4, a case where the current operation (current process) of the automatic depositing / dispensing machine 10 is “transaction selection” and the next operation (next process) is “card inhalation” will be described. When the line of sight is directed to something other than the automatic depositing / dispensing machine, the warning unit 330 outputs the screen and outputs the sound, and the flicker is not turned on or blinked. Specifically, the warning unit 330 blinks the background of the screen and outputs a voice message “Please look at the screen and select a transaction”. Thereby, the line of sight of the user U is turned on the screen.

  On the other hand, when the line of sight is directed to the screen of the automatic depositing / dispensing machine or the suction / discharge port, the warning unit 330 changes the warning method depending on the line of sight. For example, when the line of sight is directed to the background of the screen, the warning unit 330 expands all the buttons on the screen and outputs a voice message “Please select a transaction from the displayed buttons”. When the line of sight is in the wording of the screen, the warning unit 330 expands the buttons on the screen and expands all the buttons on the screen, and outputs a voice message “Please select a transaction from the displayed buttons” To do. When the line of sight is on a button on the screen, the warning unit 330 expands the line of sight on the screen, outputs a voice saying "Please insert a card after pressing the" draw "button", and the card The flicker 125a in the vicinity of the reader unit 105 is turned on. When the line of sight is in the moving image on the screen, the warning unit 330 expands all buttons on the screen and outputs a voice message “Please select a transaction from the displayed buttons”.

  When the line of sight is in the automatic deposit / withdrawal machine and is at the suction / discharge port (any of the card reader unit 105, the passbook printer unit 110, the banknote deposit / withdrawal unit 115, and the coin deposit / withdrawal unit 120), the warning unit 330 Flashes the background of, expands all buttons on the screen, and outputs a voice message "Please check the screen and select a transaction". Even when the line of sight is in the automatic depositing / dispensing machine and in the rear confirmation mirror, the warning unit 330 blinks the background of the screen and expands all buttons on the screen. Please select ".

  In FIG. 4, as a method of calling attention to the user U, (1) In addition to the case where the current process is selecting a transaction and the next process is card inhalation, (2) the current process is inhaling a card and the next process is The case of password input is specifically shown. FIG. 5 also shows that (3) the current process is a password input and the next process is an amount input, (4) the current process is an amount input and the next process is a communication process, and (5) the current process is The case where the next process is card ejection during the communication process is shown. Furthermore, FIG. 6 shows the case where (6) the current process is card receipt and the next process is banknote / detail slip discharge, (7) the current process is banknote / detail slip receipt, and the next process is a thank you screen. Yes.

  In this manner, the warning unit 330 alerts the user U using the screen display unit 130, the audio output unit 135, and the flicker lighting units 125a to 125d according to the type of attention specified by the determination unit 320. Or display guidance so that the next process can be easily performed.

  As described above, according to the automatic depositing / dispensing machine 10 according to the present embodiment, the point of the user U's line of sight is directed to the automatic depositing / dispensing machine 10 based on the state of the user U's face and the direction of the line of sight. Is calculated. As a result of the calculation, whether the current process or the next process is processed normally or whether the user U should be alerted depending on whether the line of sight of the user U is directed to the automatic depositing / dispensing machine 10 or other than the device Is determined. As a result, the user U not only performs an action to call attention from the automatic processing device to things that have occurred in the past, such as forgetting the medium such as cash and passbook, but also, for example, pay attention to the user during the operation. It is possible to assist the user's operation in advance, such as prompting the user to prevent it from being forgotten in the future, proactively alerting the user, or substituting for guidance for the next operation. As a result, effects such as improvement in operability and reduction in transaction time can be achieved.

  In the above embodiment, the operations of the respective units are related to each other, and can be replaced as a series of operations in consideration of the relationship between each other. Thereby, the embodiment of the automatic processing device can be made the embodiment of the automatic processing method and the embodiment of the automatic processing program for causing the computer to realize the function of the automatic processing device.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to the example which concerns. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

It is a whole block diagram of the automatic depositing / withdrawing machine which concerns on one Embodiment of this invention. It is a functional block diagram of the automatic depositing / withdrawing machine which concerns on the same embodiment. It is the flowchart which showed the procedure of the gaze recognition / automatic deposit / withdrawal process which concerns on the same embodiment. It is the figure which showed an example of the combination of operation | movement of the automatic depositing / withdrawing machine which concerns on the same embodiment, the point of a gaze, and the alerting method. It is the figure which showed an example of the combination of operation | movement of the automatic depositing / withdrawing machine which concerns on the same embodiment, the point of a gaze, and the alerting method. It is the figure which showed an example of the combination of operation | movement of the automatic depositing / withdrawing machine which concerns on the same embodiment, the point of a gaze, and the alerting method.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Automatic depositing / withdrawing machine 100 Apparatus main body 105 Card reader part 110 Passbook printer part 115 Banknote depositing / withdrawing part 120 Coin depositing / withdrawing part 125a, 125b, 125c, 125d Flicker lighting part 130 Screen display part 135 Audio | voice output part 140a, 140b Camera 200 Line of sight Information acquisition unit 205 Face image acquisition unit 210 Eye detection unit 215 Face detection unit 220 Recognition unit 220a Eyeball recognition unit 220b Line of sight recognition unit 300 Automatic deposit / withdrawal unit 310 Calculation unit 315 Storage unit 320 Determination unit 325 Process execution unit 330 Warning unit

Claims (14)

An automatic processing device that recognizes a user's line of sight and executes desired automatic processing based on the recognized line of sight,
A face image acquisition unit for acquiring a user's face image;
An eye detection unit that detects a user's eye state based on the face image acquired by the face image acquisition unit;
A recognition unit for recognizing the direction of the line of sight based on the eye state detected by the eye detection unit;
A face detection unit that detects a face state of the user based on the face image acquired by the face image acquisition unit;
A calculation unit that calculates whether a user's line of sight is directed to the automatic processing device based on a face state detected by the face detection unit and a line-of-sight direction recognized by the recognition unit;
A storage unit for storing a current process in which the automatic processing apparatus is operating and a next process scheduled to be operated next;
An automatic processing apparatus according to line-of-sight recognition, comprising: a determination unit that determines whether to alert a user based on a relationship between a user's line-of-sight tip calculated by the calculation unit and the automatic processing apparatus.   The line-of-sight recognition according to claim 1, wherein the recognition unit determines the presence or absence of blinking from the state of the eyes detected by the eye detection unit, and recognizes the direction of the line of sight when the eyes are open as a result of the determination. Automatic processing device according to   When the calculation unit recognizes that the blink is not blinking, the calculation unit detects the user's line of sight based on the direction of the line of sight and the distance from the face position detected by the face detection unit to the automatic processing device. The automatic processing device according to line-of-sight recognition according to claim 2, wherein a portion of the automatic processing device that is pointed at is pointed to.   According to the calculation, the determination unit is configured such that the user's line of sight is not directed to a part of the automatic processing apparatus that the user wants to operate in the current process or the next process stored in the storage unit. The automatic processing device according to the eye-gaze recognition according to claim 3, wherein it is determined that the user is warned.   According to the calculation, the determination unit is configured such that the user's line of sight is directed to a part of the automatic processing device that the user wants to operate in the current process or the next process stored in the storage unit. In this case, the automatic processing device according to line-of-sight recognition according to claim 3, wherein the current process or the next process stored in the storage unit is determined to be processed as usual.   The automatic processing device according to line-of-sight recognition according to claim 1, wherein the determination unit specifies a type of attention to be urged to the user according to a point of the user's line of sight.   The automatic processing device according to line-of-sight recognition according to claim 6, further comprising a warning unit that warns according to the type of attention specified by the determination unit.   The automatic processing apparatus according to claim 7, wherein the warning unit issues a warning by at least one of display of a screen attached to the automatic processing apparatus, output of sound, lighting of a warning light, and blinking of a warning light.   The automatic warning according to any one of claims 7 and 8, wherein the warning unit issues a warning by at least one of an audio output and a screen blinking when the user's line of sight is not directed to the automatic processing device. Processing equipment.   The automatic processing apparatus according to claim 7, wherein the warning unit enlarges the display of the user's line of sight when the user's line of sight is directed to the automatic processing apparatus.   The automatic processing apparatus according to any one of claims 1 to 10, wherein the automatic processing apparatus is an automatic depositing and dispensing machine.   The automatic processing device according to line-of-sight recognition according to claim 11, wherein the face image acquisition unit acquires a user's face image captured using a plurality of cameras installed on an upper part of the automatic depositing and dispensing machine. . An automatic processing method for recognizing a user's line of sight and executing desired automatic processing based on the recognized line of sight,
Obtaining a face image of the user;
Detecting the eye state of the user based on the acquired face image;
Recognizing the direction of the line of sight based on the detected eye state;
Detecting a state of a user's face based on the acquired face image;
Calculating whether a user's line of sight is directed to the automatic processing device based on the detected face state and the recognized line-of-sight direction;
Storing the current process in which the automatic processing apparatus operates and the next process scheduled to be operated next in a storage unit;
Based on the relationship between the calculated line of sight of the user and the automatic processing device, it is determined whether the current process or the next process stored in the storage unit is processed normally or the user is alerted. And an automatic processing method according to line-of-sight recognition. An automatic processing program for recognizing a user's line of sight and causing a computer to execute desired automatic processing based on the recognized line of sight,
Processing to acquire the user's face image;
A process of detecting the eye state of the user based on the acquired face image;
Processing for recognizing the direction of the line of sight based on the detected eye state;
A process of detecting the face state of the user based on the acquired face image;
A process of calculating whether the user's line of sight is directed to the automatic processing device based on the detected face state and the recognized line-of-sight direction;
A process of storing a current process in which the automatic processing apparatus operates and a next process scheduled to be operated next in a storage unit;
Based on the relationship between the calculated line of sight of the user and the automatic processing device, it is determined whether the current process or the next process stored in the storage unit is processed normally or the user is alerted. An automatic processing program for causing a computer to execute processing.

Images