JP2018190012A - Customer service necessity determination apparatus, customer service necessity determination method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a customer service necessity determination apparatus and the like capable of detecting embarrassment of an operator of an information terminal and supporting smooth customer service.SOLUTION: A customer service necessity determination apparatus 1 acquires operation information for an operation unit 25 of an information terminal 10, and also acquires a photographed image of an operator 7 of the information terminal 10, and determines whether customer service is necessary or not, on the basis of the operation information and the image. As a result of the judgment, when the customer service is required, the fact is notified to a staff or the like. Therefore, the staff and the like can quickly and precisely notice the embarrassment (whether the operation is difficult or not) of the operator 7 of the information terminal 10, and it is possible to smoothly perform an appropriate customer service.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a customer service necessity determination device, a customer service necessity determination method, and a program, and more particularly, to a technology for supporting customer service response to an information terminal operator.

  In financial institutions and the like, information terminals such as ATMs (automated transaction devices) and tablets are used. The customer operates these information terminals by himself to carry out a desired transaction. However, when a customer unfamiliar with the operation feels confused about the operation, the staff faces the customer. The staff who handle such customer service visually confirms the appearance of a plurality of customers on the floor and always pays attention to whether or not there are customers who are confused about the operation.

  On the other hand, for example, Patent Document 1 proposes an automatic transaction apparatus that displays the guidance screen display when an ATM operation is monitored by the surrounding video to determine the input status at each transaction input operation stage and it is determined that the customer's operation is unfamiliar. Has been. For the determination of the operation, for example, moving image data is taken every second and compared with pre-programmed operation data to detect whether or not the customer is temporarily unable to operate due to an external factor. Then it is described.

JP-A-9-198545

  However, in the method of recognizing the operation disclosed in the automatic transaction apparatus disclosed in Patent Document 1 described above, since the comparison with a program prepared in advance for each pixel of the entire image is performed, the processing is heavy and the required resources are large. turn into.

  The present invention has been made in view of such a problem, and detects the confusion of the operator of the information terminal and can support smooth customer service, a customer service necessity determination device, a customer service necessity determination method, And to provide a program.

  In order to solve the above-described problem, the first invention is an operation information acquisition unit that acquires operation information for an information terminal, an image acquisition unit that acquires an image obtained by photographing an operator of the information terminal, and a physical body from the image. A skeleton determination unit that estimates a skeleton and determines whether or not the operation is difficult based on a similarity with a prestored skeleton posture model, and needs customer service based on the operation information and the image A determination means for determining whether or not the customer service is required.

  According to the first aspect of the present invention, whether or not the customer service necessity determination device is difficult to operate based on the operation information for the information terminal and the posture of the operator estimated from the image obtained by photographing the operator of the information terminal. It is determined whether or not customer service is necessary. Thereby, it is possible to detect the confusion of the operator of the information terminal and support smooth customer service. Further, by using the skeleton determination to detect the confusion of the operator of the information terminal, the processing load can be reduced as compared with the conventional motion recognition.

  In the first invention, it is desirable to provide notifying means for notifying that when customer service is required as a result of determination by the determining means. By receiving the notification, the staff can quickly know that there is an operator who is confused about the operation of the information terminal, so that the customer service can be handled more smoothly. In addition, the determination unit determines whether the operation is difficult based on the operation information based on the first process for determining whether the operation is difficult based on the operation information, and based on the image. A second process and a third process for estimating the operator's posture based on the image and determining whether the operation is difficult are performed, and the results of the first to third processes are integrated. It is desirable to determine whether customer service is necessary. Since it is determined whether or not the customer service is required based on the operation information, the movement of the operator's face / body, and the posture, an accurate determination result can be obtained.

  Furthermore, it is preferable that the determination means determines the priority for customer service based on the results of the first to third processes. Further, it is desirable that the notification means performs notification according to priority. As a result, the staff can know the priority of customer service, and can take appropriate actions, such as responding to a more urgent operator first.

  In the first process described above, it is desirable to determine whether or not the operation is difficult based on at least one of an input error to the operation unit of the information terminal and the number of screen return operations. The second process tracks the movement of the operator for each facial organ or body part included in the image, and determines that the operation is difficult when the magnitude of the movement exceeds a predetermined threshold value. It is desirable to do. Thereby, it becomes possible to improve the accuracy of the determination of difficulty of operation.

  According to a second aspect of the present invention, a computer obtains operation information for an information terminal, obtains an image obtained by photographing an operator of the information terminal, estimates a body skeleton from the image, and stores it in advance. Determining whether or not the operation is difficult based on the similarity to the posture model of the skeleton, and determining whether or not the customer service is necessary based on the operation information and the image. This is a method for determining the necessity of customer service.

  According to the second invention, it is determined whether or not the operation is difficult based on the operation information for the information terminal and the posture of the operator estimated from the image obtained by photographing the operator of the information terminal. Whether or not is necessary. Thereby, it is possible to detect the confusion of the operator of the information terminal and support smooth customer service. Further, by using the skeleton determination to detect the confusion of the operator of the information terminal, the processing load can be reduced as compared with the conventional motion recognition.

  According to a third aspect of the invention, the computer estimates operation skeleton of the body from the image, operation information acquisition means for acquiring operation information for the information terminal, image acquisition means for acquiring an image of the operator of the information terminal. Skeleton determination means for determining whether or not the operation is difficult based on the similarity with the skeleton posture model stored in advance, and whether or not customer service is required based on the operation information and the image It is a program for functioning as determination means for determining.

  According to the third invention, it is possible to cause a computer to function as the customer service necessity determining apparatus according to the first invention.

  According to the present invention, it is possible to provide a service necessity determination device, a service necessity determination method, and a program capable of detecting a confusion of an operator of an information terminal and supporting smooth customer service.

External view of the information terminal 10 provided with the service necessity determination apparatus 1 Functional configuration diagram of the service necessity determination apparatus 1 Internal configuration diagram of a computer that functions as the customer service necessity determination device 1 The figure which shows the flow of the whole process which the customer service necessity determination apparatus 1 performs The figure explaining the extraction of the facial organ 32 in the face / body tracking process The figure explaining posture estimation processing Conceptual diagram of shape matching processing between posture models 6a, 6b,... And skeleton 34 estimated from images The figure which shows another example of the whole structure of the hospitality necessity determination apparatus 1

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram showing an appearance of an ATM 10a as an example of an information terminal 10 provided with a customer service necessity determining device 1 according to the present invention, and FIG. 2 is a block diagram showing a functional configuration of the customer service necessity determining device 1. These are figures which show the example of the internal structure of the computer which functions as the customer service necessity determination apparatus 1. FIG.

  As shown in FIG. 1, the ATM 10a includes, as an operation unit 25 and a camera 20, as various processing units as an automatic transaction device, a passbook processing unit 21 that performs processing such as passbook insertion / discharge and bookkeeping, and card insertion A card processing unit 22 that performs processing such as discharge and reading of card information, a banknote processing unit 23 that loads and unloads banknotes, a coin processing unit 24 that loads and unloads coins, and the like. Moreover, the customer service necessity determination apparatus 1 according to the present invention is provided inside the casing of the ATM 10a.

  The camera 20 is a video camera that photographs the operator 7 and is attached at a position facing the operator 7 so that the face and body of the operator 7 can be photographed. The video (image) taken by the camera 20 is sent to the customer service necessity determination device 1. The start and stop of shooting by the camera 20 is performed according to an instruction from the control unit 11 of the customer service necessity determination device 1.

  The operation unit 25 includes a touch panel, various operation buttons, and the like, and inputs a signal corresponding to an operation by the operator 7 to the customer service necessity determination device 1.

  As shown in FIG. 2, the customer service necessity determination apparatus 1 includes an operation information acquisition unit 2, an image acquisition unit 3, a determination unit 4, a notification unit 5, and a storage unit 12.

  The operation information acquisition unit 2 acquires operation information for the operation unit 25 of the information terminal 10 and inputs the operation information to the determination unit 4. The operation information is information related to a pressing operation of an instruction operation button on the touch panel of the operation unit 25, operation timing, and the like.

  The image acquisition unit 3 acquires an image (video) taken by the camera 20 and inputs it to the determination unit 4.

  The determination unit 4 determines whether customer service is required based on the operation information acquired by the operation information acquisition unit 2 and the image (video) acquired by the image acquisition unit 3. The determination unit 4 includes an operation difficulty determination unit 40 and a customer service necessity determination unit 44.

  Based on the operation information and the image, the operation difficulty determination unit 40 determines whether or not the operator 7 feels that the operation is difficult (whether or not he / she is confused). Specifically, the operation difficulty determination unit 40 includes an operation event determination unit (first processing unit) 41, a face / body tracking unit (second processing unit) 42, and a posture estimation unit (third processing unit) 43. And determining whether or not the operation is difficult based on the result of the processing by each of the processing units 41, 42, and 43. Specific processing contents will be described later.

  The service necessity determination unit 44 obtains the results of the operation event determination unit (first processing unit) 41, the face / body tracking unit (second processing unit) 42, and the posture estimation unit (third processing unit) 43. Integrate to determine if customer service is required. Details of the determination process by the determination unit 4 (the operation difficulty determination unit 40 and the customer service necessity determination unit 44) will be described later.

  As a result of the determination by the determination unit 4, the notification unit 5 notifies that when customer service is required. For example, the notification is made by sounding an alarm sound of a portable terminal (not shown) possessed by the staff, transmitting information on the location of the information terminal 10 that needs customer service, and displaying the information on the display unit of the portable terminal. To do. Note that the method of notification is not limited to these, and any method may be used to notify the staff.

  The storage unit 12 stores posture model data 6 created in advance. The posture model data 6 is a model that defines a representative posture that is taken when the operator feels difficulty in operation (when he / she is confused with the operation). The one represented by data shall be used. The posture is a typical posture that is seen when the user feels confused about the operation, such as a posture in which arms are folded or a posture in which the neck is tilted. Data representing the postures of the plurality of patterns as skeleton data is referred to as posture models 6a, 6b,. The posture models 6a, 6b,... Are collectively referred to as posture model data 6.

  The network 8 includes a LAN (Local Area Network), a WAN (Wide Area Network) connected to a wider area, or a public communication line such as the Internet, a base station, and the like. The network 8 may be wired or wireless. The customer service necessity determination apparatus 1 can access a server (not shown) via the network 8 and can transmit and receive various programs and data.

  FIG. 3 is a diagram illustrating a configuration example of a computer that functions as the service necessity determination apparatus 1. As shown in FIG. 3, the service necessity determination apparatus 1 includes a control unit 11, a storage unit 12, a media input / output unit 13, a peripheral device I / F unit 14, an input unit 15, a display unit 16, a communication control unit 17, and the like. Are connected via a bus 18. The peripheral device I / F unit 14 is connected to the camera 20, the operation unit 25 of the ATM 10a, and the like. When the computer functions as the customer service necessity determination apparatus 1, the computer control unit 11 is realized by executing a program describing the processing shown in FIG.

  The control unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The CPU calls a program stored in the storage unit 12, ROM, recording medium or the like to a work memory area on the RAM and executes it, and drives and controls each unit connected via the bus 18. The ROM permanently holds a computer boot program, a program such as BIOS, data, and the like. The RAM temporarily holds the loaded program and data, and includes a work area used by the control unit 11 to perform various processes. The control part 11 functions as each means of the customer service necessity determination apparatus 1 by reading and executing the above program.

  The storage unit 12 is a storage device such as a hard disk drive, for example. The storage unit 12 stores a program executed by the control unit 11, data necessary for program execution, an operating system, and the like. These program codes are read by the control unit 11 as necessary, transferred to the RAM, and read and executed by the CPU.

  The media input / output unit 13 is, for example, a drive device for various recording media (media) such as a CD, a DVD, and an MO, and inputs / outputs (writes / reads) data to / from the media.

  The peripheral device I / F (interface) unit 14 is a port for connecting a peripheral device, and transmits / receives data to / from the peripheral device via the peripheral device I / F unit 14. The peripheral device I / F unit 14 is configured by a USB or the like, and usually includes a plurality of peripheral devices I / F. The connection form with the peripheral device may be wired or wireless.

The input unit 15 is an input device such as a touch device, a pointing device such as a keyboard and a mouse, and a numeric keypad, and outputs input data to the control unit 11.
The display unit 16 includes a display device such as a liquid crystal panel or a CRT monitor, and a logic circuit (video adapter or the like) for executing display processing in cooperation with the display device, and is input under the control of the control unit 11. Display information is displayed on a display device. The input unit 15 and the display unit 16 may be a touch panel display (operation unit 25) in which an input device such as a touch panel is integrally provided on the display screen.

The communication control unit 17 includes a communication control device, a communication port, and the like, and controls communication with the network 8 and the like.
The bus 18 is a path that mediates transmission / reception of control signals, data signals, and the like between the devices.

  Next, with reference to FIG. 4, the flow of the entire process executed by the customer service necessity determining apparatus 1 will be described. The control unit 11 of the customer service necessity determining apparatus 1 reads a program and data related to the processing shown in FIG. 4 from the storage unit 12 and executes processing based on the program and data.

  When the operation unit 25 of the ATM 10a is operated, the control unit 11 (operation information acquisition unit 2) of the customer service necessity determination device 1 acquires the executed operation information (step S101). The operation information is information in which the content of the operation (operation event) is associated with the operation time. For example, the operation information acquisition unit 2 collects information such as which button of the operation unit 25 is operated, or an operation instruction, interruption, and cancellation, and notifies the determination unit 4 of the information.

  Further, when detecting the operator 7, the control unit 11 (the image acquisition unit 3) transmits an imaging start instruction to the camera 20. The control unit 11 acquires an image (video) taken by the camera 20 (step S102). The camera 20 captures a range including the face and body of the operator 7. The images (videos) are sequentially sent to the determination unit 4 and used for determining difficulty in operation. The control unit 11 stops the photographing of the camera 20 when the operator leaves from the front of the ATM 10a.

  The control unit 11 (determination unit 4) determines whether or not the operator 7 needs customer service based on the operation information acquired in step S101 and the image (video) acquired in step S102. Specifically, the control unit 11 performs an operation event determination process based on the operation information (step S103), a face organ determination and face / body tracking process based on the image information (step S105 to step S106), and a posture estimation based on the image information. Then, the bone model fitting process (steps S108 to S109) is executed, and it is determined whether or not the operator 7 is confused by the operation (steps S104, S107, and S110).

  In the operation event determination process in step S103, the control unit 11 counts at least one or both of an input error to the operation unit 25 and the number of screen return operations for each predetermined time length. An input error is, for example, cancellation of an operation, operation of a correction button, or an illegal operation. The number of screen return times is the number of operations for returning to the previous screen or menu screen. In the operation difficulty determination process in step S104, it is determined that “there is confusion” when the number of input mistakes and the number of screen return operations counted in step S103 exceeds a threshold value. In addition, it is not limited to the number of input mistakes and the number of screen return operations, and for example, it may be determined whether the operation has been interrupted for a predetermined time or whether the operation is difficult (whether there is confusion) from the operation speed or the like.

  In the facial organ determination and face / body tracking processing in steps S105 to S106, the control unit 11 tracks (tracks) the movement of the operator 7 for each facial organ 32 or body part included in the image acquired in step S102. To do. In the operation difficulty determination process in step S107, it is determined that the operation is difficult when the magnitude of the movement of the facial organ 32 or the body part exceeds a predetermined threshold.

FIG. 5 is a diagram for explaining the process of extracting the facial organ 32 from the image 31. The control unit 11 recognizes a face from each frame 31 of an image (video) acquired from the camera 20 and further extracts a facial organ 32. The facial organs 32 are a right eyebrow 32a, a left eyebrow 32b, a right eye 32c, a left eye 32d, a nose 32e, a mouth 32f, a jaw 32g, and the like. As shown in FIG. 5A, the facial organs 32a to 32g are extracted. The control unit 11 sets a plurality of tracking points for the extracted facial organs 32a to 32g, and tracks (tracks) the movements of the facial organs 32a to 32g from the amount of movement of the tracking points between frames. If the amount of movement of the tracking point is equal to or greater than a predetermined threshold, it is determined that the user is confused. The motion amount threshold value is set in advance to an appropriate value for each of the facial organs 32a to 32g. For the process of estimating the facial organ, for example, a known algorithm as shown in Non-Patent Document 1 may be used (Non-Patent Document 1: Vahid Kazemi, one other person, “One Millisecond Face Alignment with an Ensemble of Regression”).
Trees ", [online], Royal Institute of Technology, Computer Vision and Active Perception Lab, [March 30, 2017 search], Internet, <URL: http://www.cv-foundation.org/openaccess/content_cvpr_2014 /papers/Kazemi_One_Millisecond_Face_2014_CVPR_paper.pdf>)

  Further, in the face / body tracking process in step S106, the movement of each part of the body is also tracked to determine whether or not it is difficult to operate. The recognition of the movement of each part of the body uses, for example, the result of skeleton estimation in the posture estimation process in step S108. For example, the movement of a specific body part such as an arm is tracked from the estimated skeleton 34.

FIG. 6 is a diagram illustrating skeleton estimation in the posture estimation process. As illustrated in FIG. 6, the control unit 11 estimates a human skeleton 34 from an image 33 of each frame of the video. In the following description, the skeleton information estimated from the image 33 is referred to as an estimated skeleton 34. In the example of FIG. 6 (b), the estimated skeleton 34 includes feature points 35a, 35b, 35c, 35d, 35e, 35f, which are determined from anatomical information such as arm joints, left and right ends of the face, and shoulders. 35g, 35h and lines 34a, 34b, 34c, 34d, 34e, 34f, 34g, 34h, 34i, 34j determined from the arrangement of these feature points 35a-35h. The control unit 11 tracks (tracks) the movement of the skeleton 34 estimated from the image 33 of each frame of the video. As the skeleton estimation and tracking algorithm, for example, a known technique as shown in Non-Patent Document 2 may be used (Non-Patent Document 2: Zhe Cao and three others, “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”). , [Online], 24 Nov 2016, The Robotics
Institute, Carnegie Mellon University, [March 30, 2017 search], Internet <URL: https://arxiv.org/pdf/1611.08050v1.pdf, https://github.com/ZheC/Realtime_Multi-Person_Pose_Estimation> )

  The control unit 11 determines whether or not the motion pattern of the estimated skeleton 34 matches or is similar to a predefined motion pattern. If the motion pattern matches or is similar, the operator 7 is confused (difficult to operate). Is determined).

  Note that if the face cannot be caught in front due to the installation position of the camera 20 in the processing of step S105 to step S106, the facial organ is not determined and tracked for the image, and is excluded from the operation difficult determination target. It shall be. Thereby, determination mistakes can be reduced.

  Further, in the processing from step S108 to step S110, the control unit 11 executes posture estimation and bone model fitting processing based on the image acquired from the camera 20, and the posture of the operator 7 (estimated skeleton 34) and posture model data. 6 is determined to be coincident (or similar). Thereby, it is determined whether or not the operator 7 is confused by the operation. As shown in FIG. 7, the posture model data 6 (6a, 6b,...) Is preliminarily created in a plurality of patterns for a posture representing confusion and stored in the storage unit 12. In the bone model fitting process, the control unit 11 performs rotation, enlargement, reduction, etc., and aligns the skeleton estimated from the image 33 (estimated skeleton 34) with each of the posture models 6a, 6b,. Then, for each of the feature points 35a to 35h, the distance from the corresponding point on the posture model 6a, 6b,. If the sum of the distances obtained for each point is smaller than a preset threshold, it is determined that they are similar, and it is determined that they are confused (difficult to operate).

  Returning to the description of FIG. The control unit 11 (the customer service necessity determination unit 44) integrates the results obtained in Step S104, Step S107, and Step S110, and determines whether or not the customer service is necessary (Step S111). Specifically, the determination result based on the operation information in step S104 is “operation difficult (confused)”, and one of the determination results based on the camera image in step S107 or step S110 is “no operation difficult (confused)”. If it is, it is determined that the urgency of customer service is low. In addition, when the determination result based on the operation information in step S104 is “no operation difficulty (confused)” and one of the determination results based on the camera image in step S107 or step S110 is “difficult operation (confused)” Determines that “service urgency: medium”. In addition, when the determination result based on the operation information in step S104 is “difficult to operate (confused)” and one of the determination results based on the camera image in step S107 or step S110 is “difficult to operate (confused)” Determines that “the urgency of customer service is high”. In this way, if the level of urgency is also determined along with the necessity of customer service, the staff can easily grasp the priority for customer service.

  The control part 11 notifies the determination result in step S111 (step S112). The notification is made, for example, by sounding an alarm sound of a portable terminal (not shown) possessed by the staff via the communication control unit 17 of the customer service necessity determining device 1 or the location of the information terminal 10 that needs customer service. This is done by sending information and displaying it on the display. Alternatively, the management center (server) connected via the network 8 is notified of information such as the installation location of the information terminal 10 that needs customer service, and the management center (server) contacts the staff. Also good. In addition, the control unit 11 desirably performs notification together with the priority according to the level of urgency determined in step S111. By notifying priority according to the level of urgency, the staff can perform appropriate customer service according to the necessity and priority of customer service.

  As described above, the customer service necessity determining apparatus 1 according to the present embodiment acquires operation information for the information terminal 10, acquires an image obtained by photographing the operator 7 of the information terminal 10, and acquires the operation information and the image. Based on the above, it is determined whether or not customer service is required, and if the result of determination is that customer service is required, the fact is notified. Thereby, the staff or the like can quickly and accurately notice the confusion (whether the operation is difficult) of the operator 7 of the information terminal 10 and can smoothly perform appropriate customer service.

  In the above-described embodiment, the customer service necessity determination device 1 according to the present invention has been described with respect to the device configuration provided in the information terminal 10 (ATM 10a) that is the operation target, but the present invention is limited to this. It is not a thing. For example, as shown in FIG. 8, the customer service necessity determination device 1 may be provided separately from the information terminal 10 and the camera 20 and connected to communicate via the network 8. In the example of FIG. 8, the information terminal 10 that is the operation target is a tablet 10b. In addition, the camera 20 is installed at a position where it is possible to take a picture of the operator 7 operating the tablet 10b. Alternatively, the camera 20 may be a camera (in-camera) provided on the touch panel display side of the tablet 10b.

  In the case of the device configuration shown in FIG. 8, the operation information acquisition unit 2 and the image acquisition unit 3 of the customer service necessity determination device 1 acquire the operation information of the tablet 10 b and the video of the camera 20 via the network 8. About the determination part 4, the notification part 5, and the memory | storage part 12 of the service necessity determination apparatus 1, it is the same as that of the case of FIG. Thus, since the service necessity determination apparatus 1 according to the present invention can be provided integrally with or separately from the information terminal 10, it can be applied to various usage forms.

  The preferred embodiments of the customer service necessity determining apparatus and the like according to the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications can be conceived within the scope of the technical idea disclosed in the present application, and these are naturally within the technical scope of the present invention. Understood.

1 ……………… Service Necessity Determining Device 2 ……………… Operation Information Acquisition Unit 3 ……………… Image Acquisition Unit 4 ……………… Determining Unit 5 ……………… Notification Part 6 ……………… Attitude model data 7 ……………… Operator 8 ……………… Network 10 …………… Information terminal 10a ………… ATM
10b ………… Tablet 11 ………… Control unit 12 ……… Storage unit 13 ………… Media input / output unit 14 ………… Peripheral device I / F unit 15 ……………… Input unit 16 ………… Display unit 17 ………… Communication control unit 18 …………… Bus 20 …………… Camera 25 …………… Operation unit 31, 33 …… Image 32 …… ……… Face organ 34 …………… Estimated skeleton 35 …………… Feature point 40 …………… Operation difficulty determination unit 41 …………… Operation event determination unit (first processing unit)
42 .......... Face / Body Tracking Unit (Second Processing Unit)
43 ........... Posture estimation unit (third processing unit)
44 ……………… Customer Service Necessity Determination Section

Claims (9)

Operation information acquisition means for acquiring operation information for the information terminal;
Image acquisition means for acquiring an image of the operator of the information terminal;
A skeleton determination unit that estimates a skeleton of the body from the image and determines whether or not the operation is difficult based on a similarity with a posture model of a skeleton stored in advance;
Determination means for determining whether or not customer service is required based on the operation information and the image;
A service necessity determination device comprising: As a result of determination by the determination means, notification means for notifying that when customer service is required,
The waiting-and-serving necessity determination apparatus of Claim 1 characterized by the above-mentioned.   A first process for determining whether or not the operation is difficult based on the operation information; and a first process for determining whether or not the operation is difficult from the movement of the face and the body based on the image. 2 and a third process for estimating the posture of the operator based on the image and determining whether the operation is difficult or not, and integrating the results of the first to third processes. The service necessity determination apparatus according to claim 1 or 2, wherein it is determined whether or not a service is required.   The service determination unit according to claim 3, wherein the determination unit determines a priority for handling a customer based on the results of the first to third processes.   5. The service necessity determination apparatus according to claim 4, wherein the notification means performs notification according to the priority.   The first process determines whether or not the operation is difficult based on at least one of an input error with respect to the operation unit of the information terminal and the number of screen return operations. The customer service necessity determination device described in 1.   In the second process, the movement of each operator's facial organ or body part included in the image is tracked, and the operation is difficult when the magnitude of the movement exceeds a predetermined threshold value. The customer service necessity determination device according to claim 3, wherein the determination is made. Computer
Obtaining operation information for the information terminal;
Obtaining an image of an operator of the information terminal;
Estimating a skeleton of the body from the image and determining whether or not the operation is difficult based on a similarity with a skeleton posture model stored in advance;
Determining whether customer service is required based on the operation information and the image;
A method for determining the necessity of customer service. Computer
Operation information acquisition means for acquiring operation information for the information terminal;
Image acquisition means for acquiring an image of the operator of the information terminal;
A skeleton determination unit that estimates a skeleton of the body from the image and determines whether or not the operation is difficult based on a similarity with a posture model of a skeleton stored in advance;
Determination means for determining whether or not customer service is required based on the operation information and the image;
Program to function as.

Images