JP6007758B2 - User operation terminal device, user operation terminal program - Google Patents

Description

  The present invention relates to a user operation terminal device and a user operation terminal program.

  The user operation terminal device is a machine that allows a user to operate a machine operation performed by a staff member. In recent years, the user operation terminal device has been introduced due to a demand for operation cost reduction. The user operation terminal device includes, for example, an ATM terminal of a financial institution, a self-check register such as a supermarket, an automatic change machine, a designated seat ticket vending machine (such as a limited express ticket or a seat designation), an automatic aircraft check-in machine, a convenience store, etc. Such as ticket sales. As described above, cases where user operation terminal devices are introduced in various scenes of daily life are increasing.

  The user operation terminal device is highly convenient for a user who is used to the user because the user does not need to line up at the counter and can achieve the purpose in a relatively short time with a simple operation. However, for an unfamiliar user, the operation takes time, and it becomes necessary for the staff to explain the operation method. For this reason, stores, financial institutions, airlines, and the like that have introduced user operation terminal devices have dealt with unfamiliar users by arranging dedicated staff to support operations. However, since the number of cases where user operation terminal devices are introduced in order to reduce operation costs, a case where the cost of arranging a full-time staff cannot be considered is assumed. Therefore, for the spread of user operation terminal devices, there is a need for a user operation terminal device that can deal with unfamiliar users without the need for dedicated staff.

  In general, in the user operation terminal device, for example, a process of instructing the type of operation from the terminal to the user and waiting for the end of the user operation is repeated. Specifically, for example, the user operation terminal device prompts the user to input numbers, and waits for the user to finish inputting numbers from the user touch panel. Alternatively, the user operation terminal device prompts the user to input a barcode of the product and waits for the user to finish reading the barcode.

  Therefore, for example, the user operation terminal device determines whether or not the user is confused by the operation according to the following method and provides an operation support guide to the user. In the first method, for example, at the start of operation, the user operation terminal device causes the user to select a detailed guide mode for an unfamiliar user and a power user mode for a familiar user (Non-Patent Document 1). In the second method, for example, the user operation terminal device provides an operation support guide when the user does not start an operation even after a predetermined time has elapsed (Patent Document 1). Specifically, for example, the user operation terminal device generates an operation support sound when no operation is performed for 5 seconds after the guide sound “please press the start button”. Further, the user operation terminal device calls a full-time staff if the user cannot complete the operation even after a longer time has elapsed.

JP 2008-204443 A

https://www.fureai-net.city.kawasaki.jp/

  However, according to the first method, the description is too detailed in the detailed guide mode, and even if the user feels confused, the mode may not be changed during the operation. On the other hand, in the power user mode, it is too difficult to perform the operation by the user. Further, according to the second method, since it is detected on the basis of time whether or not the user is at a loss, it is necessary for an unaccustomed user to always wait for a certain time in order to listen to the operation support voice. In addition, there was a case where support of a full-time staff was required because the operation support voice was not started quickly.

  Therefore, as in the first and second methods, it is required to provide operation support in accordance with the degree of hesitation with respect to each user's operation, instead of performing an operation support guide for users assuming an average user.

  An object of the present invention is to provide a user operation terminal device and a user operation terminal program that detect a user's degree of hesitation in operation and effectively provide an operation support guide.

  A first aspect is a user operation terminal device that accepts an operation by a user, and a line-of-sight detection unit that detects the line of sight of the user based on the image of the user, and the operation based on the detected line of sight Based on the line-of-sight location information and the line-of-sight location information generating means for generating line-of-sight location information for the operation target location indicating the next operable portion of the operable location, and the line-of-sight location information, the operation target location Guide output means for detecting whether or not the line of sight is wandering based on the staying state of the line of sight with respect to the operation target location until the operation is completed, and outputting the operation support guide to the user when the line of sight is wandering .

  According to the first aspect, it is possible to detect a user's degree of hesitation in operation and effectively provide an operation support guide.

It is an example figure which shows the self check cash register which is an example of a user operation terminal device. It is an example figure which shows an example of a structure of the user operation terminal apparatus shown by FIG. It is a figure showing an example of an operation possible location and an operation target location. It is a figure explaining the operation possible location and operation target location corresponding to a mesh area | region. It is a flowchart explaining the 1st detection process of the presence or absence of a gaze wandering state. It is a flowchart figure explaining the 2nd detection process of the presence or absence of a gaze wandering state. It is a flowchart figure explaining the detection process of the presence or absence of a gaze wandering state of the user operation terminal device in the second embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the technical scope of the present invention is not limited to these embodiments, but extends to the matters described in the claims and equivalents thereof.

[Example of user operation terminal device]
FIG. 1 is an example diagram showing a self-check register 100 which is an example of a user operation terminal device. In the figure, a self-check register 100 includes, for example, a camera 10, a shopping basket storage area 11, bar code readers 12 and 13, a touch panel 14, a checkout machine 15, a bagging unit 16, and the like. After instructing the user to perform an operation, the self-check register 100 detects the end of the operation performed by the user according to the instruction, and then outputs the next operation instruction.

  For example, the case of the self-check register 100 of FIG. 1 is illustrated. First, the user places a shopping cart containing a product to be settled in the shopping basket storage area 11. At this time, a transaction start button is displayed on the touch panel 14 of the self-check register 100. Therefore, the user presses the transaction start button and causes the self-check register 100 to recognize that preparation is complete. Subsequently, reading of the barcode is instructed, and the user takes out the products to be settled one by one from the shopping cart, and causes the barcode readers 12 and 13 to read the barcode printed on the products. When the bar code is read, the product information displayed on the touch panel 14 is confirmed, and the product is put in the plastic bag of the bagging unit 16. The barcode reading process is repeated for all payment objects in the shopping cart.

  When the barcodes of all the products are read, the user presses the settlement button displayed on the touch panel 14. Therefore, the self-check register 100 instructs to select on the touch panel whether to pay by money card, cash, or credit card. For example, the user selects a payment method using cash and inserts cash into the checkout machine 15. When the completion of payment is notified, the user takes out the receipt and the change and picks up the shopping bag containing the finished product from the bagging unit 16.

  Thus, the user operation terminal device instructs the user to perform an operation and waits until the end of the operation is detected. In general, the user operation terminal device has a plurality of locations to be operated, such as buttons on the touch panel 14, barcode readers 12 and 13, shopping cart storage 11, bagging unit 16, and the like. For this reason, a user discriminate | determines the location used as operation object next from several operation locations, and performs operation. An unfamiliar user cannot discriminate the location to be operated next, and the processing is delayed. For this reason, the user operation terminal device provides an operation support guide.

[Operation support guide]
The operation support guide is support provided by a voice or display related to an operation provided when the user is at a loss in operation. According to a general user operation terminal device, the operation support guide is started when, for example, the user does not start the operation even after a predetermined time has elapsed after instructing the operation to the user. For this reason, in a general user operation terminal device, an unfamiliar user cannot receive the operation support guide without waiting for a certain period of time. On the other hand, for users who are used to it, if it takes time to search for a wallet, the operation support guide is started after a certain period of time. As a result, users who are accustomed to receiving unnecessary operation support guides sometimes have unpleasant feelings.

  Therefore, the user operation terminal device 100 according to the present embodiment detects the user's line of sight based on the user's image, and based on the detected line of sight, the operable portion and the operable portion Line-of-sight location information for the operation target location indicating the next operable location is generated. Then, the user operation terminal device 100 detects the presence / absence of the line-of-sight state indicating the staying state of the line of sight with respect to the operation target part until the operation to the operation target part is completed based on the line-of-sight part information, Outputs an operation support guide to the user.

  That is, the user operation terminal device 100 according to the present embodiment detects the presence / absence of a wandering state based on the staying state of the user's line of sight in the user operation terminal device 100. Thereby, the user operation terminal device 100 can provide an operation support guide for a user who has been detected as having a wandering state. Thereby, an operation assistance guide is provided to a necessary user at a necessary timing in a necessary operation. In addition, an inexperienced user who has been detected as having a sight line wandering state can receive an operation support guide without waiting for a certain period of time. On the other hand, it is avoided that an operation support guide is erroneously provided to a user who has been used and is determined to have no sight line wandering, even though he / she is not confused.

  Next, the configuration of the user operation terminal device 100 in the present embodiment will be described.

[Configuration example of user operation terminal device]
FIG. 2 is an example diagram illustrating an example of a configuration of the user operation terminal device 100 illustrated in FIG. 1. In the figure, a user operation terminal device 100 includes, for example, a CPU (Central Processing Unit) 21, a display 22, a camera 23, a speaker 24, a memory 25, a barcode reading device 26, a touch panel 27, a bill / money receiving device 28, It has a hard disk 29.

  For example, like the camera 10 in the user operation terminal device 100 in FIG. 1, the camera 23 is installed at a position where the user's line of sight can be detected, and generates image data of the user who performs the operation. The display 22 displays, for example, operation instructions and information on products for which barcodes are read, and the touch panel 27 corresponds to the touch panel 14 of FIG. 1 and accepts user operations such as button presses. The speaker 24 outputs an operation instruction to the user, an operation support guide, and the like. The barcode reader 26 reads the barcode of the product and converts it into product information. The bill / money receiving device 28 receives payment from the user and outputs a change or a receipt. The hard disk 29 stores information related to the overall control of the user operation terminal device 100. Further, the memory 25 stores, for example, a program PR for performing user operation terminal processing in the present embodiment.

  The user operation terminal processing program PR in the present embodiment cooperates with the CPU 21 to detect the user's line of sight based on the user's image generated by the camera 23 and generate line-of-sight location information. Then, the user operation terminal processing program PR detects the presence / absence of the line-of-sight state based on the line-of-sight location information, and outputs an operation support guide via the display 22 or the speaker 24 when the line-of-sight state is present.

  Next, the detection process for the state of sight line wandering performed by the user operation terminal device 100 according to the present embodiment will be specifically described.

[Operational location, operation target location]
First, the operable locations A to G and the operation target location T in the user operation terminal device 100 of the present embodiment will be described. In the present embodiment, in the user operation terminal device 100 as shown in FIG. 1, the operation target locations A to G are the locations to be operated by the user, and the next operation target is the operable location A to G. The operation possible locations A to G are defined as operation target locations T.

  FIG. 3 is a diagram illustrating an example of the operable locations A to G and the operation target location T in the self-check register 100 illustrated in FIG. In the example shown in the figure, the operable location A corresponds to, for example, the position of the settlement machine that inserts cash, the operable location C is the payment method designation button on the touch panel 14, and the operable location D is the location of the accounting start button. Correspond. Further, the operable portion E corresponds to the position of the barcode readers 12 and 13, the operable portion F corresponds to the position of the bag filling unit 16, and the operable portion G corresponds to the position of the shopping basket storage area 11. And the operation target location T respond | corresponds to the position of the card reader used at the time of payment.

  That is, the example of FIG. 3 shows the operation possible locations A to G and the operation target location T when the payment by the card is designated by the payment method designation button. As described above, in the user operation terminal device 100 according to the present embodiment, the operable locations A to G are set in advance, and each time the operation is instructed to the user, the operable location corresponding to the operation is the operation target. Set as location T. The user operation terminal device 100 generates information on the user's line of sight directed to the operable locations A to G and the operation target location T as the sight location information. By the way, the operation possible locations A to G and the operation target location T may be set in accordance with, for example, a mesh region into which a target region for visual line detection is divided. As a result, the load of the line-of-sight location information generation process is reduced.

  FIG. 4 is an example for explaining the operable locations A to G and the operation target location T corresponding to the mesh region er. In the figure, for example, a mesh area obtained by equally dividing the target area of the visual line sensor in the x coordinate direction and the y coordinate direction is set as the unit area er. In the figure, one or a plurality of mesh regions er are set as operable portions Ae to Ge and an operation target region Te.

  The user operation terminal device 100 represents the information on the location where the user's line of sight is directed, for example, in the Cartesian coordinate type (x coordinate, y coordinate). Then, the coordinate information to which the line of sight is directed is collated with the coordinate information of the operable locations A to G and the operation target location T, and the line-of-sight location information is generated. In the example of FIG. 4, the region-by-mesh area er is set as the operable locations Ae to Ge and the operation target location Te, thereby simplifying the line-of-sight location information generation process. This is because the user operation terminal device 100 only needs to determine whether or not the coordinates to which the line of sight is directed are included in the operable locations Ae to Ge and the operation target location Te that are rectangular areas. Further, the boundaries of the regions of the operable locations Ae to Ge and the operation target location Te may be set to values such as integers that simplify the arithmetic processing. This further simplifies the line-of-sight location information generation process and reduces the load associated with the process.

[First Embodiment]
Subsequently, in the first exemplary embodiment, a process in which the user operation terminal device 100 detects the presence / absence of a sight line wandering state based on the sight line location information will be described. The user operation terminal device 100 according to the present embodiment detects the presence or absence of a state of wandering line of sight based on the staying state of the line of sight with respect to the operation target location T until the operation to the operation target location T is completed.

[Detection of sight wandering]
The user's line of sight has the characteristic of moving prior to its movement. For this reason, for example, when an operation such as pressing a button is instructed, the user turns his / her gaze on the button before extending his / her finger on the button. Therefore, the user operation terminal device 100 according to the present embodiment uses the characteristic that the line of sight is directed to the operable locations A to G and the operation target location T prior to the operation, and is the user wondering about the operation? Determine whether or not. Here, the staying state of the line of sight when the user is confused about the operation or not confused will be described based on a specific example.

[If you are not confused with the operation]
When the user is not confused by the operation and performs the operation with certainty, the line of sight is first directed to the operation target location T. For example, in the self-check register 100 shown in FIG. 3, when the barcode of the product to be paid is read, the user's line of sight is directed to the barcode reader (operation target location T) 12, 13 and then the product Directed. Then, the user carries the product in front of the barcode readers 12 and 13. Alternatively, the user checks the position of the barcode with the line of sight toward the product in advance, and then directs the product toward the barcode reader (operation target location T) 12 and 13 and in front of the barcode reader 12 and 13. In this way, when the user is not confused with the operation, the user's line of sight does not wander and is directed to the operation target location T.

[If you are confused about the operation]
On the other hand, when the user is at a loss in operation, the user's line of sight, for example, searches for the operation target location T and goes around the operable locations A to G. A person's line of sight does not move slowly and slowly, but has a characteristic of jumping and moving a point having characteristics. The operable portions A to G including the operation target portion T, which is the operation target, are given visual features such as a shape, a color, and a mark, and are represented so as to guide the line of sight. That is, the user's line of sight is easily directed to the operation target location T and the operable locations A to G. For this reason, when the user is at a loss in operation, the user's line of sight does not reach the bar code readers (operation target locations T) 12 and 13 but wanders around the operable locations A to G. Alternatively, even if the user can turn his / her line of sight toward the barcode reader (operation target location T) 12, 13, he / she thinks while looking at the barcode reader (operation target location T). In this case, although the line of sight arrives at the operation target location T, the line of sight stagnates without leaving the operation target location T and enters a gaze state. When the user is confused about the operation, such a state of wandering is seen.

  Therefore, in this embodiment, the user operation terminal device 100 determines whether or not the line of sight is wandering based on the dwell time of the line of sight at the operation target place T, the operable places A to G, or the number of stays of the line of sight. To detect. Here, the time (number of times) in which the line of sight stays in the manipulable places A to G is At to Gt, and the time (number of times) in which the line of sight stays in the operation target place T is Tt.

[First detection method in a wandering state]
In the first detection method, the user operation terminal device 100 uses the line-of-sight dwell time or line of sight to the operation target locations A to G in the accumulation of the line-of-sight dwell time or line-of-sight dwell times with respect to the operable locations A to G and the operation target location T. When the target retention ratio of the number of stays is smaller than the first reference value, it is determined that the line of sight is wandering and an operation support guide is output. That is, the user operation terminal device 100 moves to the operation target location T with respect to the total accumulated eye gaze amount for the operation target location T and the operable locations A to G after the operation instruction to the user until the operation is completed. When the ratio of the gaze retention amount is small, it is determined that there is a state in which the gaze is wandering. That is, when the ratio of the gaze retention amount to the operation target location T with respect to the total of the gaze retention amounts with respect to the operation target location T and the operable locations A to G is small, the user's gaze can be operated while searching for the operation target location T. The case of going around G is shown. Thereby, it is detected that the user is confused with the operation, and the operation support guide is output.

  Specifically, the case where the presence or absence of the state where the line of sight is wandering is detected based on the line-of-sight residence time is illustrated. First, the user operation terminal device 100 measures each time when the line of sight is directed to the operable locations A to G and the operation target location T until the current time after the operation instruction. Subsequently, the user operation terminal device 100 operates at the operation target locations T to the operation target locations T to the operation possible locations A to G in the accumulated line-of-sight dwell time “At + Ct + Dt + Et + Ft + Gt + Tt”. The ratio of the sight staying time “At + Ct + Dt + Et + Ft + Gt” to G is calculated, and when the ratio is equal to or greater than the first reference ratio (for example, 50%), the sight staying at the operation target point T is calculated. It is determined that the time is not enough, and it is considered that the line of sight is wandering.

  Note that the first reference ratio (50% in this example) in the first detection method varies depending on the environment in which the user operation terminal device 100 is used, the content, and the like. For this reason, the first reference ratio is adjusted as follows. First, the user operation terminal device 100 detects whether there is a wandering state in a state where the first reference ratio is set to an initial value. At this time, it is verified whether the guide is provided to the user who needs the operation support guide, and whether the guide is not provided to the unnecessary user. The first reference ratio is lowered when the operation support guide is not provided to a user who needs it, and is increased when the operation support guide is provided to a user who does not need it.

  The same applies to the case where the detection of the presence or absence of the sight line wandering state is based on the number of sight line stays. The number of times that the line of sight stays in the manipulable places A to G is At to Gt, and the number of times that the line of sight stays in the operation target place T is Tt. The user operation terminal device 100 is connected to the maneuverable locations A to G in the operation target location T and the cumulative number of line-of-sight stays “At + Ct + Dt + Et + Ft + Gt + Tt” at the maneuverable locations A to G. Calculate the ratio of the line-of-sight dwell count “At + Ct + Dt + Et + Ft + Gt”. When the ratio is 50% or more, for example, it is determined that the line-of-sight dwell count at the operation target location T is not sufficient. It is considered that there is a state.

[Second detection method in a wandering state]
In the second detection method, the user operation terminal device 100 uses the line-of-sight dwell time or line of sight to the operation target locations A to G in the accumulation of the line-of-sight dwell time or the number of line-of-sight dwells with respect to the operable locations A to G and the operation target location T. When the target retention ratio of the number of stays is larger than the second reference value, and when one eye-gaze residence time or the cumulative eye-gaze residence time for the operation target location exceeds the reference time, it is determined that the eye-gaze state is present and the operation Output a support guide. When the ratio of the gaze retention amount to the operation target location T with respect to the total of the gaze retention amounts for the operation target location T and the operable locations A to G is large and the gaze residence time for the operation target location T is long, the user sets the operation target location T Shows the case of staring. That is, it is a case where the line of sight is sufficiently directed to the operation target location T, but the processing is not completed while staring. Thereby, it is detected that the user is confused with the operation, and the operation support guide is output. If the user is accustomed to the operation, the line of sight does not stagnate after the line of sight captures the operation target point T.

  Specifically, the user operation terminal device 100 performs an operation at the operation target location T and the accumulated line-of-sight dwell time (number of times) “At + Ct + Dt + Et + Ft + Gt + Tt” at the operable locations A to G. The line-of-sight dwell time (number of times) “Tt” at the target location T is equal to or greater than the second reference rate (for example, 80%), and the line-of-sight stays at the operation target location T for at least the reference time (for example, 5 seconds). It is considered that the line of sight is wandering. Note that the second reference ratio and the reference time in the second detection method are also set to be adjustable according to the environment, content, and the like, as in the first detection method.

  Subsequently, processing corresponding to the first and second detection methods will be described with reference to flowcharts.

[Flowchart: First Detection Method]
FIG. 5 is a flowchart for explaining a first detection process for the presence or absence of a line-of-sight state in the user operation terminal device 100. When the self-check register 100, which is the user operation terminal device 100, detects a user, it outputs an operation instruction (S11). The operation instruction to the user is, for example, an instruction for pressing a payment start button.

  Subsequently, after a predetermined time has elapsed, the user operation terminal device 100 measures the user's line of sight with the camera 23, measures the line-of-sight dwell time (number of times) at the operable locations A to G and the operation target location T, and the cumulative total thereof. Is recorded (S12). Specifically, the user operation terminal device 100 detects the user's line of sight based on the user image data generated by the camera 23. Next, the user operation terminal device 100 generates the line-of-sight dwell time (number of times) at the operable positions A to G and the operation target position T as line-of-sight position information. Then, the user operation terminal device 100 calculates the total of the line-of-sight dwell time (number of times) at the operable positions A to G and the operation target position T, and the line-of-sight dwell time (number of times) at the operable positions A to G. Calculate the total.

  At this time, the user operation terminal device 100 according to the present embodiment instructs the user to perform the operation, and after the elapse of a certain time, the gaze dwell time (number of times) at the operable locations A to G and the operation target location T is determined. Start measurement. This is because the user is obsessed with the operation until a certain time elapses immediately after the operation instruction. The user operation terminal device 100 can detect the presence / absence of a wandering state with higher accuracy by excluding the period during which the user is busy with the operation.

  Subsequently, as a first detection method, the user operation terminal device 100 calculates the accumulated “At + Ct + Dt + Et + Ft + Gt” of the line-of-sight dwell time (number of times) at the operable locations A to G and the operation target location T. In “+ Tt”, it is determined whether or not the ratio of the cumulative “At + Ct + Dt + Et + Ft + Gt” of the line-of-sight dwell time (number of times) at the operable locations A to G exceeds 50% (S13, S14). If it exceeds 50% (YES in S14), it indicates that the ratio of the line-of-sight dwell time (number of times) at the operation target location T is small. That is, it indicates that the degree of the user looking at the operation target location T is small, and it is determined that there is a state in which the line of sight is wandering. If it is determined that the line of sight is wandering, voice guidance is started (S15). And it changes to the process ("3" of the flowchart figure of FIG. 6) shown in the following flowchart figure. The process indicated by “3” will be described later.

  On the other hand, when it does not exceed 50% (NO in S14), it is determined that there is no state in which the user is looking at the operation target location T and the line of sight is wandering. When it is determined that there is no sight line wandering state, the process proceeds to determination by the second detection method (“1” in the flowchart in FIG. 6).

[Flowchart: Second Detection Method]
FIG. 6 is a flowchart for explaining the second detection process for the presence / absence of the state of sight line wandering in the user operation terminal device 100. The flowchart in the same figure corresponds to the process when step S14 in the flowchart in FIG. 5 is NO. As a second detection method, the user operation terminal apparatus 100 accumulates “At + Ct + Dt + Et + Ft + Gt + Tt” of the line-of-sight dwell time (number of times) at the operable locations A to G and the operation target location T. It is determined whether or not the ratio of the cumulative “Tt” of the line-of-sight dwell time (number of times) at the operation target location T exceeds 80% (S21). When the ratio exceeds 80% (YES in S21), the user operation terminal device 100 further determines whether the line-of-sight dwell time Tt from the operation instruction to the operation target location T up to the present time or one line-of-sight dwell time is obtained. It is determined whether it is within 5 seconds (S22).

  When the ratio exceeds 80% (YES in S21) and the gaze residence time from the operation instruction to the operation target location T exceeds 5 seconds (NO in S22), the gaze residence time in the operation target location T Although the (number of times) is sufficient, it indicates that the line of sight is stagnant at the operation target location T. That is, it is considered that the user is staring at the operation target location T. In this case, it is determined that the line of sight is wandering and voice guidance is started (S23). When the voice guide in the flowchart of FIG. 5 is output (S15 of FIG. 5) (“3” of FIG. 5), it is then determined whether or not the user's operation is completed (S24). . Further, even when the ratio does not exceed 80% (NO in S21), it is determined whether or not the user operation is completed (S24).

  When the user operation is completed (YES in S24), it is subsequently determined whether or not all operations are completed (S25). Whether or not all the operations have been completed is determined, for example, in the example of the self-check register 100, whether or not a plastic bag containing a product that has been settled is taken up. When all the operations are completed (YES in S25), the user operation terminal device 100 ends the process. On the other hand, when all the operations are not completed (NO in S25) and when the user operation is not completed (NO in S24), the user operation terminal device 100 performs the next operation instruction (S26). The process proceeds to the first detection step (S13) in the flowchart of FIG.

  In addition, in this embodiment, the user operation terminal device 100 has a line-of-sight dwell time (number of times) at an operation impossible location (not shown) excluding the operation possible locations A to G and the operation target location T. It does not take into account the detection of the presence or absence of a state of gaze. This means that it is not detected that there is a wandering state even if the line of sight is directed to a non-operable location other than the operable locations A to G and the operation target location T for a long time or frequently. For example, a case where the user is paying attention to a matter not related to the operation when the line of sight is directed to the non-operable portion even though the operation instruction is performed is shown. For example, in the example of the self-check register 100, the action not related to the operation indicates an action such as searching for a wallet, a point card, or a change for payment. In addition, the behavior not related to the operation includes the behavior of searching for the barcode printing position on the product in order to read the barcode, and the behavior attracted by other staff members and acquaintances. When the operation support guide is provided in such a case, the user feels that an unnecessary indication has been received from the user operation terminal device 100 and is likely to feel uncomfortable.

  Therefore, the user operation terminal apparatus 100 in the present embodiment does not use the staying state of the line of sight other than the operable locations A to G and the operation target location T in detecting the presence / absence of the wandering state. Thereby, the user operation terminal device 100 in the present embodiment distinguishes between a case where the user is lost in operation and a case where attention is paid to an event or action not directly related to the operation, It makes it possible to detect the presence or absence of wandering conditions.

  As described above, the user operation terminal device 100 according to the present embodiment includes a line-of-sight detection unit that detects a user's line of sight based on the user's image, and an operable part that can be operated based on the detected line of sight. A line-of-sight location information generating means for generating line-of-sight location information for an operation target location T indicating the next operable location A to G among the A to G and the operable locations A to G is provided. Then, the user operation terminal device 100 according to the present embodiment detects the presence / absence of the line-of-sight state based on the staying state of the line of sight with respect to the operation target part T until the operation is completed at the operation target part T based on the line-of-sight part information. When there is a state where the line of sight is wandering, it has a guide output means for outputting an operation support guide to the user.

  As described above, the user operation terminal device 100 according to the present embodiment detects the presence or absence of a wobbling state for each user and for each operation by using the staying state of the line of sight with respect to the operation target location T until the operation is completed. to enable. As a result, the user operation terminal device can provide the operation support guide to the user who needs it in the required operation scene. For this reason, an operation support guide is not erroneously provided to an accustomed user, and an operation support guide is provided to an unaccustomed user accurately. As a result, the frequency of calling a full-time clerk decreases, and the cost required for the full-time clerk is reduced. Moreover, the spread of user operation terminal devices is promoted.

  Further, the user operation terminal device 100 detects the presence / absence of a line-of-sight state based on the staying state of the line of sight rather than the operation. Since the movement of the line of sight is performed prior to the operation, it is possible to determine whether or not to provide the operation support guide more quickly than the determination of the degree of ambiguity of the operation based on the operation. Furthermore, the user operation terminal device 100 according to the present embodiment can determine whether or not the user is lost in the operation based on the presence or absence of the line of sight before the line of sight reaches the operation target location T. Thereby, when the user is at a loss in the operation, the user can quickly receive the operation support guide before turning the line of sight toward the operation target location T.

  On the other hand, according to a general user operation terminal device, it is determined whether or not the user is lost in the operation based on the time until the end of the operation. For example, if the operation is not performed even after 10 seconds have passed since the operation instruction, it is determined that the operation is lost and the operation support guide is started. On the other hand, according to the user operation terminal device 100 of the present embodiment, an operation support guide can be provided before 10 seconds elapses, and a user who is lost in operation receives the operation support guide more quickly. It is done. This reduces the opportunity for an unfamiliar user to feel uncomfortable, and also reduces the frequency with which a full-time staff is called.

  In the user operation terminal device 100 according to the present embodiment, the line-of-sight location information is either or both of the line-of-sight dwell time and the number of line-of-sight dwells with respect to the operable portions A to G and the operation target location T. And the guide output means of the user operation terminal device 100 calculates the gaze dwell time or the number of gaze dwells at the operation target location T in the accumulation of the gaze dwell time or the gaze dwell times with respect to the operable locations A to G and the operation target location T. The presence / absence of a wandering line of sight is detected based on the target retention rate. As described above, it is difficult to set a uniform elapsed time for each user when detecting the presence / absence of a wandering state based on a certain elapsed time such as 10 seconds. On the other hand, the user operation terminal device 100 according to the present embodiment eliminates the need to set a uniform elapsed time based on the line-of-sight dwell time or the ratio of the number of line-of-sight dwells. Can be detected with higher accuracy.

  Further, the user operation terminal device 100 in the present embodiment determines that the line of sight is wandering when the target retention ratio is smaller than the first reference value (for example, 50% in the present embodiment). Thereby, the user operation terminal device 100 according to the present embodiment detects that the line of sight is wandering when the user's line of sight captures the operation target location T and the number of line of sight stays is insufficient, and the operation support A guide can be provided.

  In addition, the user operation terminal device 100 according to the present embodiment has a target staying rate greater than a second reference value (for example, 80% in the present embodiment) and a single line of sight with respect to the operation target location T. When the dwell time or the accumulated line-of-sight dwell time exceeds the reference time (for example, 5 seconds in this embodiment), it is determined that the line-of-sight wandering state is present. Thereby, the user operation terminal device 100 according to the present embodiment makes it possible to detect a case where the line of sight is stagnant at the operation target location T after capturing the operation target location T. Thereby, the user operation terminal device 100 can quickly provide an operation support guide to a user who cannot stare at the operation target location T and complete the operation.

  Further, the user operation terminal device 100 according to the present embodiment starts generating the line-of-sight location information after the reference time has elapsed. As a result, the line-of-sight state at a fixed time when the user receives an operation instruction is excluded from the determination target of the presence or absence of the line-of-sight state. As a result, the presence / absence of a wobbling state of the line of sight can be detected with higher accuracy based on the line-of-sight location information with higher accuracy.

[Second Embodiment]
In the second embodiment, the user terminal device operates as the line-of-sight location information for the operable locations A to G, the operation target location T, and the operation impossible locations other than the operation target location T and the operable locations A to G. The order of sight staying at the line of sight is generated, and the presence / absence of a wandering state is detected based on the order of sight staying. Here, the operation impossibility place O shows the place (not shown) except the operation possible places AG and the operation target place T. The inoperable portion O is, for example, a product, a wallet, another person, etc., and indicates a portion excluding the operable portions A to G and the operation target portion T.

[If you are not confused with the operation]
When the user performs the operation with certainty, for example, the line of sight immediately reaches the operation target location T. In the example of the self-check register 100, when the operation target location T is the barcode reader 12, 13, the user first takes the product with the line of sight toward the product, and takes the line of sight to the barcode reader 12, 13 (operation After moving to the target location T), the product is moved in front of the barcode readers 12 and 13 to read the barcode. In this case, the line of sight moves in the order of “operation impossible location O → operation target location T”. Alternatively, the user first turns the line of sight toward the barcode readers 12 and 13 (operation target location T), takes the product with the line of sight toward the product, and moves the line of sight to the barcode reader (operation target location T). Thereafter, the product is moved in front of the barcode readers 12 and 13 to read the barcode. In this case, the line of sight moves in the order of “operation target location T → operation impossible location O → operation target location T”. As described above, when the user is not confused about the operation, the line of sight is in the order of “operation impossible place O → operation target place T” or “operation target place T → operation impossible place O → operation target place T”. Moving.

  For a user who has become accustomed to some extent, the line of sight may not immediately reach the operation target location T, but eventually, after going through several operable locations A to G, the line of sight finally reaches the operation target location T. To reach. As described above, when the user is accustomed to some extent, the line of sight finally reaches the operation target location T even if the line of sight does not immediately reach the operation target location T.

[If you are confused about the operation]
On the other hand, when the user is confused about the operation, the line of sight continues to go around the operable portions A to G. Alternatively, once the line of sight reaches the operation target location T, it is not recognized that it is the operation target location T, and therefore, the other operable locations A to G are visited again. In this way, when the user is at a loss in operation, the user cannot recognize the operation target location T, and a state of a sight line that looks around the operable locations A to G is seen.

  Therefore, the user operation terminal device 100 in the second embodiment determines that the line of sight is wandering when the line-of-sight staying order is “operation target location T → operation possible location A to G”. This is the case, for example, when the line of sight moves as “operable location A → operation target location T → operable location D”. In this case, although the line of sight has reached the operation target location T, it has subsequently moved to the operable locations A to G, so that it is detected that the line of sight is wandering. In addition, when “operation target location T → operation possible locations A to G” is repeated a predetermined number of times, it may be determined that the line of sight is wandering. This is the case, for example, when the line of sight moves as “operable location A → operation target location T → operation possible location D → operation impossible location O → operation target location T → operation possible location G”. In this case, since the process of moving the line of sight to the operation target point T and moving again to the operable points A to G is repeated, it is considered that the user is lost in the operation, and it is detected that the line of sight is wandering. (When the predetermined number = 2).

  On the other hand, the user operation terminal device 100 according to the second embodiment determines that there is no line-of-sight state when the line-of-sight staying order is “operation target location T → operation impossible location O”. In this case, it is considered that the user is not confused but is performing an operation other than the operation, such as confirmation of the barcode position of the product or confirmation of the wallet after recognizing the operation target location T. This is the case, for example, when the line of sight moves as “operable location A → operation target location T → operation impossible location O”. In this case, although the line of sight has moved to another region after reaching the operation target point T, it is detected that the line of sight has moved to the non-operationable point O, so that the line of sight is not wandering.

  However, when “operation target location T → operation impossible location O” is repeated a predetermined number of times, it may be determined that the line of sight is wandering. This is the case, for example, when the line of sight moves as “operable location A → operation target location T → operation impossible location O → operation target location T → operation impossible location O”. In this case, since the process of moving the line of sight to the operation target point T and moving again to the non-operational point O is repeated, it is considered that the user is lost in the operation, and it is detected that the line of sight is wandering (predetermined) Number of times = 2).

  Subsequently, the detection process of the presence or absence of a state of wandering in the second embodiment will be described with reference to a flowchart.

[Flowchart of detection processing]
FIG. 7 is a flowchart for explaining the detection processing for the presence / absence of the line of sight of the user operation terminal device 100 according to the second embodiment. Similarly to the first embodiment, when the self-check register 100, which is the user operation terminal device 100, detects a user, it outputs an operation instruction (S31). Subsequently, the user operation terminal device 100 measures the user's line of sight with the camera 23 after a predetermined time has elapsed, measures the line-of-sight staying order in the operable locations A to G, the operation target location T, and the operation disabled location O, Record (S32). Specifically, the user operation terminal device 100 detects the user's line of sight based on the user image data generated by the camera 23. Subsequently, the user operation terminal device 100 generates and stores the line-of-sight staying order at the operable locations A to G, the operation target location T, and the operation impossible location O as the visual location information. Also in the second embodiment, the user operation terminal device 100 may start recording the line-of-sight staying order after a predetermined time has elapsed after instructing the user to perform an operation. As a result, the initial period during which the user is confused with the operation is excluded, and the presence or absence of a state of wandering eyes is detected with higher accuracy.

  Subsequently, the user operation terminal device 100 determines whether or not the user has turned his / her line of sight to the operation target location T (S33). When the user does not turn his / her line of sight to the operation target location T (NO in S33), the process returns to step S32. On the other hand, when the line of sight is directed to the operation target place T (YES in S33), it is subsequently determined whether or not the line of sight is directed to the operable places A to G (S34). Thus, it is determined whether or not the line of sight is directed to the operable locations A to G next to the operation target location T.

  When the line of sight is directed to the operable locations A to G next to the operation target location T (YES in S34), it is further determined whether or not this process has been repeated a predetermined number of times (S36). That is, it is determined whether or not the line-of-sight staying order “operation target location T → operation possible location A to G” has been repeated a predetermined number of times. If it has been repeated a predetermined number of times (YES in S36), voice guidance is started (S37). As described above, when the line of sight repeatedly travels around the operation target position T and the operable positions A to G, it is determined that the line of sight is wandering and an operation support guide is provided. In the flowchart of FIG. 7, the voice guidance is started when the process of moving the line of sight to the operable locations A to G next to the operation target location T is repeated a predetermined number of times (YES in S36). (S37), it is not limited to this example. When the line of sight once moves to the maneuverable positions A to G after the operation target position T (YES in S34), it may be determined that the line of sight is wandering and an operation support guide may be provided (S37).

  In addition, when the line of sight is directed to the operation non-operable places A to G after the operation target place T, that is, the operation impossible place O (NO in S34), the line of sight staying order “operation target place T → operation impossible” It is determined whether or not “location O” has been repeated a predetermined number of times (S35). If it has been repeated a predetermined number of times (YES in S35), voice guidance is started (S37). As described above, when the line of sight repeatedly travels around the operation target place T and the operation impossible place O, it is determined that the line of sight is wandering and an operation support guide is provided. On the other hand, in the case of NO in step S35, that is, when the line of sight has once moved to the operation impossible place O next to the operation target place T (NO in S35), it is determined that the line of sight is not wandering, and the operation support guide is Not provided. In this case, it is determined that the line of sight is wandering in order to show a case where the line of sight is directed to the wallet or the product after the operation target point T is recognized.

  Subsequently, it is determined whether or not the user's operation is completed (S38). When the user operation is completed (YES in S38), it is subsequently determined whether or not all operations are completed (S39). When all the operations are completed (YES in S39), the user operation terminal device 100 ends the process. On the other hand, when all the operations are not completed (NO in S39) and when the user operation is not completed (NO in S38), the user operation terminal device 100 performs the next operation instruction (S40). Then, the process proceeds to step S32.

  As described above, the user operation terminal device 100 according to the second embodiment includes the operable locations A to G, the operation target location T, and the operation disabled locations other than the operation target location T and the operable locations A to G. The presence or absence of a line-of-sight wandering state is detected based on the line-of-sight staying order of the line-of-sight portions with respect to O. As described above, the user operation terminal device 100 according to the second embodiment makes it possible to detect the presence / absence of a sight line wandering state with high accuracy and to effectively output an operation support guide based on the sight line retention order. be able to. In addition, the operation support guide can be provided more quickly.

  Further, the user operation terminal device 100 according to the present embodiment determines that there is a line-of-sight wandering state when the line-of-sight staying order shifts to the operable positions A to G next to the operation target position T. Alternatively, the user operation terminal device 100 determines that the line of sight is in a wandering state when the line-of-sight staying order repeats the first reference number of times (predetermined number of times) that the operation target position T moves to the operable positions A to G. .

  As described above, the user operation terminal device 100 determines that the line of sight is wandering when the line of sight is directed once or a predetermined number of times to the operable positions A to G following the operation target position T, and the operation support guide Can be provided. Thereby, even when the user's line of sight reaches the operation target location T, the user operation terminal device 100 quickly detects a case where the user's line of sight has not been recognized as the operation target location T, and uses the operation support guide. Can be provided.

  Further, the user operation terminal device 100 according to the present embodiment determines that there is no line-of-sight state when the line-of-sight staying order moves from the operation target place T to the operation impossible place O. Thereby, after the user's line of sight reaches the operation target location T, the user operation terminal device 100 can quickly detect the case where the user is performing something other than the operation, and can avoid providing the operation support guide. For this reason, an operation support guide is provided when the user searches for the barcode printing position on the product, or when other staff or acquaintances are distracted, and the user who is not at a loss of operation feels uncomfortable It is avoided.

  However, the user operation terminal device 100 determines that the line of sight is wandering when the line-of-sight staying order repeats the second reference number of times (predetermined number of times) to move from the operation target position T to the operation impossible position O. Thus, the user operation terminal device 100 repeatedly detects the case where the user's line of sight repeatedly travels around the operation target location T and the operation impossible location O, and the user cannot recognize the operation target location T, and the operation support guide is displayed. Can be provided.

  Moreover, the user operation terminal device 100 determines that the line of sight is wandering when the line-of-sight staying order moves to the operation impossible place O next to the operation target place T and further moves to the operation possible places A to G. Thereby, even when the user's line of sight reaches the operation target location T, the user operation terminal device 100 travels to other locations and quickly detects the case where the operation target location T cannot be recognized. An operation support guide can be provided.

  In addition, although the 1st, 2nd embodiment example was demonstrated separately, the detection method of the presence or absence of the wandering state in each embodiment may be used in combination. Thereby, the user operation terminal device 100 can detect the presence / absence of a wandering state with higher accuracy and can effectively output the operation support guide.

  Note that the user operation terminal device processing in this embodiment may be stored as a program in a computer-readable recording medium, and may be performed by the computer reading and executing the program.

  The above embodiment is summarized as follows.

(Appendix 1)
A user operation terminal device that accepts an operation by a user,
Line-of-sight detection means for detecting the line of sight of the user based on the image of the user;
Based on the detected line of sight, the operable part where the operation is possible, and the line-of-sight part information generating means for generating line-of-sight part information for the operation target part indicating the next operable part among the operable parts,
Based on the line-of-sight location information, the presence / absence of a line-of-sight state is detected based on the staying state of the line-of-sight with respect to the operation target location until the operation at the operation target location is completed. A user operation terminal device comprising: guide output means for outputting a support guide.

(Appendix 2)
In Appendix 1,
The line-of-sight location information is either or both of the line-of-sight dwell time with respect to the operable location and the operation target location, or the number of line-of-sight dwells,
The guide output unit is configured to sway the line of sight based on a target staying ratio of the line-of-sight dwell time or the number of line-of-sight dwells at the operation target position in the accumulation of the line-of-sight dwell time or the number of line-of-sight dwells with respect to the operable part and the operation target part A user operation terminal device that detects the presence or absence of a state.

(Appendix 3)
In Appendix 2,
The guide operation means is a user operation terminal device that determines that the line of sight is wandering when the target retention ratio is smaller than a first reference value.

(Appendix 4)
In Appendix 2 or 3,
The guide output means has the line of sight when the target staying ratio is larger than a second reference value and when the eye-gaze dwell time for one time or the accumulated line-of-sight dwell time for the operation target location exceeds a reference time. A user operation terminal device that determines that there is a wandering state.

(Appendix 5)
In Appendix 1,
The line-of-sight location information includes the operable location, the operation target location, and the line-of-sight residence order of the visual location relative to the operation impossible location other than the operation target location and the operable location,
The said guide output means is a user operation terminal device which detects the presence or absence of the said visual line wandering state based on the said visual line staying order.

(Appendix 6)
In Appendix 5,
The guide output unit is a user operation terminal device that determines that the line of sight is in a wandering state when the line-of-sight staying order moves to the operable part after the operation target part.

(Appendix 7)
In Appendix 6,
The user output terminal device that determines that the line of sight is wandering when the guide output unit repeats the first reference number of times that the line-of-sight staying order shifts to the operable part next to the operation target part.

(Appendix 8)
In Appendix 5 or 6,
The said guide output means is a user operation terminal device which determines that there is no state where the line of sight wanders when the line-of-sight staying order shifts to the non-operational part next to the operation target part.

(Appendix 9)
In Appendix 8,
The user output terminal device that determines that the line of sight is in a wandering state when the guide output unit repeats the second reference number of times that the line-of-sight staying order shifts to the non-operational area after the operation target area.

(Appendix 10)
In Appendix 8,
The guide output means is a user operation terminal device that determines that the line of sight is in a wandering state when the line-of-sight staying order moves to the non-operable part next to the operation target part and further to the operable part.

(Appendix 11)
In any one of appendices 1 to 10,
The line-of-sight location information generation means is a user operation terminal device that starts generating the line-of-sight location information after a lapse of a reference time.

(Appendix 12)
A computer-readable user operation terminal program for causing a computer to execute a user operation terminal process for receiving an operation by a user,
The user operation terminal process includes:
A line-of-sight detection step of detecting the line of sight of the user based on the image of the user;
Based on the detected line of sight, an operable position where the operation is possible, and a line-of-sight location information generating step for generating line-of-sight location information for an operation target location indicating the next operable location among the operable locations,
Based on the line-of-sight location information, the presence / absence of a line-of-sight state indicating the staying state of the line-of-sight with respect to the operation target location until the completion of the operation at the operation target location is detected. A user operation terminal program comprising: a guide instruction step for instructing output of a support guide.

100: user operation terminal device, 21: CPU, 22: display, 23: camera, 24: speaker, 25: memory, 26: bar code reading device, 27: touch panel, 28: bill / money receiving device, 29: hard disk

Claims (11)

A user operation terminal device that accepts an operation by a user,
Line-of-sight detection means for detecting the line of sight of the user based on the image of the user;
Based on the detected line of sight, the operation is possible operational positions, the operation target location indicating the next operational position of the operational portion, and gaze point with respect nonoperational position, other than the operable portion Line-of- sight location information generating means for generating line-of-sight location information having an eye- gaze retention order ;
Based on the line-of-sight location information, the presence / absence of a line-of-sight state is detected based on the staying state of the line-of-sight with respect to the operation target location until the operation at the operation target location is completed. A user operation terminal device comprising: guide output means for outputting a support guide. In claim 1,
The line-of-sight location information is either or both of the line-of-sight dwell time with respect to the operable location and the operation target location, or the number of line-of-sight dwells,
The guide output unit is configured to sway the line of sight based on a target staying ratio of the line-of-sight dwell time or the number of line-of-sight dwells at the operation target position in the accumulation of the line-of-sight dwell time or the number of line-of-sight dwells with respect to the operable part and the operation target part A user operation terminal device that detects the presence or absence of a state. In claim 2,
The guide operation means is a user operation terminal device that determines that the line of sight is wandering when the target retention ratio is smaller than a first reference value. In claim 2 or 3,
The guide output means has the line of sight when the target staying ratio is larger than a second reference value and when the eye-gaze dwell time for one time or the accumulated line-of-sight dwell time for the operation target location exceeds a reference time. A user operation terminal device that determines that there is a wandering state. In claim 1 ,
The guide output unit is a user operation terminal device that determines that the line of sight is in a wandering state when the line-of-sight staying order moves to the operable part after the operation target part. In claim 1 ,
The user output terminal device that determines that the line of sight is wandering when the guide output unit repeats the first reference number of times that the line-of-sight staying order shifts to the operable part next to the operation target part. In claim 1 ,
The said guide output means is a user operation terminal device which determines that there is no state where the line of sight wanders when the line-of-sight staying order shifts to the non-operational part next to the operation target part. In claim 1 ,
The user output terminal device that determines that the line of sight is in a wandering state when the guide output unit repeats the second reference number of times that the line-of-sight staying order shifts to the non-operational area after the operation target area. In claim 1 ,
The guide output means is a user operation terminal device that determines that the line of sight is in a wandering state when the line-of-sight staying order moves to the non-operable part next to the operation target part and further to the operable part. In any one of Claims 1 thru | or 9 ,
The line-of-sight location information generation means is a user operation terminal device that starts generating the line-of-sight location information after a lapse of a reference time. A computer-readable user operation terminal program for causing a computer to execute a user operation terminal process for receiving an operation by a user,
The user operation terminal process includes:
A line-of-sight detection step of detecting the line of sight of the user based on the image of the user;
Based on the detected line of sight, the operation is possible operational positions, the operation target location indicating the next operational position of the operational portion, and gaze point with respect nonoperational position, other than the operable portion A line-of- sight location information generating step of generating line-of-sight location information having the line- of-sight retention order ;
Based on the line-of-sight location information, the presence / absence of a line-of-sight state indicating the staying state of the line-of-sight with respect to the operation target location until the completion of the operation at the operation target location is detected. A user operation terminal program comprising: a guide instruction step for instructing output of a support guide.

Images