JP6725859B1 - Information processing device and log management program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the amount of data to be investigated when a defect occurs. An information processing device (10) registers a log message indicating the contents of executed management target processing in an execution log (11a) every time the management target processing is executed, in association with the execution time of the management target processing. .. Further, the information processing device 10 acquires the observation data 3a, 3b, 3c indicating the state of the user 1 from the observation device 2 for observing the state of the user 1. Next, the information processing apparatus 10 determines the anger expression time at which the user 1 expresses anger based on the emotional expression of the user 1 represented by the observation data 3a, 3b, 3c. .. Next, the information processing device 10 determines the investigation target period of the defect of the management target processing based on the anger expression time. Then, the information processing apparatus 10 extracts the investigation target log message 4 associated with the execution time within the investigation target period from the log messages included in the execution log 11a. [Selection diagram] Figure 1

Description

The present invention relates to an information processing device and a log management program.

When there is something wrong while using an information terminal such as a personal computer or smartphone, the user who uses the information terminal can, for example, visit the support center of the company that sold the information terminal or the software used on the information terminal. Contact us. As a result, the user can obtain information such as a method for dealing with the malfunction from the support staff at the support center.

Many companies have installed servers for accepting inquiries from users. For example, a user can connect his or her own information terminal to a server on the company side and send inquiry contents to the server. The user can also send the hardware or software execution log of the information terminal in which the malfunction has occurred from the information terminal to the server on the company side. Upon receiving the report of the defect, the support staff on the company side determines the cause of the defect by comparing the detailed hearing result from the user with the continuous execution log accompanying the execution of the hardware or software. investigate. When the cause of the failure can be identified, the support staff will inform the user of how to handle the failure.

As a technology related to a system for supporting users, for example, a call center system has been proposed that can analyze the emotional state of a subject and predict the behavior of the subject thereafter. There are various techniques for analyzing human emotional states. For example, a non-verbal information analysis device has been proposed that collects, from non-verbal information expressed by a person, information that serves as a clue to infer emotions that are not expressed by the person. A utterance state determination device that improves the determination accuracy of the emotional state of a speaker based on how to enter a gap has also been proposed.

JP, 2009-294647, A JP, 2013-099373, A JP, 2017-049364, A

In the investigation of the cause of the failure, the more information obtained from the user, the higher the possibility of identifying the cause of the failure. Therefore, the information terminal may record a detailed execution log continuously and frequently. However, as a result of continuous and detailed recording of detailed execution logs, the amount of data in the execution logs that will be investigated when a defect occurs will be enormous. It takes longer for people to find out.

In one aspect, this case aims to reduce the amount of data to be investigated when a defect occurs.

In one proposal, an information processing apparatus including a storage unit and a processing unit shown below is provided.
The storage unit stores an execution log that is an execution history of the management target process. Each time the management target process is executed, the processing unit registers a log message indicating the contents of the executed management target process in the execution log in association with the execution time of the management target process. The processing unit also acquires observation data indicating the user's state from the observation device that observes the user's state. Next, the processing unit determines the anger expression time at which the user expresses the emotion of anger based on the expression state of the emotion of the user represented by the observation data. Next, the processing unit determines a period to be investigated for a defect in the management target process based on the anger expression time. Then, the processing unit extracts, from among the log messages included in the execution log, the investigation target log message associated with the execution time within the investigation target period.

According to one aspect, it is possible to reduce the amount of data to be investigated when a failure occurs.

It is a figure which shows an example of the information processing apparatus which concerns on 1st Embodiment. It is a figure which shows an example of a system structure. It is a figure which shows an example of the hardware of an information terminal. It is a block diagram which shows the function of an information terminal. It is a figure which shows an example of an emotion log. It is a figure which shows an example of an execution log. It is a figure showing an example of reaction of an information terminal according to an input by a user. It is a figure which shows an example of the improvement request screen. It is a figure which shows an example of the log transmission process for investigation. It is a figure which shows an example of the information transmitted to a server. It is a flow chart which shows an example of the procedure of emotion recognition processing. It is a flowchart which shows an example of the procedure of a microphone input emotion determination process. It is a flowchart which shows an example of the procedure of a camera input emotion determination process. It is a flow chart which shows an example of the procedure of application execution processing. It is a flowchart (1/2) which shows an example of the procedure of the log transmission process for investigation. It is a flowchart (2/2) which shows an example of the procedure of the log transmission process for investigation. It is a figure which shows the input example to an information terminal by a user. It is a figure which shows an example of the result of the emotion recognition process recorded on the emotion log. It is a figure which shows the example of extraction of the log message from an execution log. It is a figure which shows an example of the system using the emotion recognition service.

Hereinafter, the present embodiment will be described with reference to the drawings. Note that each embodiment can be implemented by combining a plurality of embodiments as long as there is no contradiction.
[First Embodiment]
First, the first embodiment will be described.

FIG. 1 is a diagram illustrating an example of an information processing device according to the first embodiment. FIG. 1 shows an information processing apparatus 10 that extracts an investigation target log message 4 useful for investigating a defect of software or hardware from an execution log 11a of a management target process. The information processing apparatus 10 can execute the extraction processing of the investigation target log message 4 by executing a log management program in which a processing procedure for extracting the investigation target log message 4 is described, for example.

An observation device 2 for observing the state of the user is connected to the information processing device 10. Examples of the observation device 2 include a camera 2a and a microphone 2b. The camera 2a captures an image showing the state of the user 1. The microphone 2b collects the voice uttered by the user 1.

The information processing device 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 is, for example, a memory included in the information processing device 10 or a storage device. The processing unit 12 is, for example, a processor included in the information processing device 10 or an arithmetic circuit.

The storage unit 11 stores an execution log 11a, which is an execution history of managed processes. The management target process is, for example, a process executed by the processing unit 12 using software or hardware that is a target of defect investigation.

The processing unit 12 executes management target processing in response to an operation from the user 1 (step S1). Then, each time the processing unit 12 executes a management target process, a log message indicating the contents of the executed management target process is registered in the execution log 11a in association with the execution time of the management target process (step S2).

Further, the processing unit 12 acquires the observation data 3a, 3b, 3c indicating the state of the user 1 from the observation device 2 (step S3). The processing unit 12 determines the anger expression time at which the user 1 expresses an angry emotion based on the expression state of the user's emotion represented by the acquired observation data 3a, 3b, 3c (step). S4). For example, the observation data 3a, 3b, 3c acquired from the camera 2a are image data of the user 1. The processing unit 12 analyzes the facial expression of the face of the user 1 shown in the image data, and determines whether or not there is an emotion of anger of the user 1 at the time of acquiring the image data. Then, the processing unit 12 determines the acquisition time of the image data in which the emotion of the anger of the user 1 is expressed as the anger expression time.

Next, the processing unit 12 determines the investigation target period for the defect in the management target processing based on the anger expression time (step S5). Note that the anger feeling is a kind of relatively rapid and temporary emotion called emotion, which is biologically a physiological reaction of biological material. Therefore, the processing unit 12 includes a period from the time before the anger expression time to the anger expression time by the duration of the anger based on the physiological reaction of the human to the investigation target period.

Then, the processing unit 12 extracts, from the log messages included in the execution log 11a, the log message associated with the execution time within the investigation target period as the investigation target log message 4 (step S6). For example, the processing unit 12 extracts the investigation target log message 4 and transmits the extracted investigation target log message 4 to the external device when an instruction to send information regarding a defect in the management target process to the external device is input. The destination external device is, for example, a computer such as a server used by a person in charge who supports the process to be managed.

According to such an information processing device 10, a failure occurs in the execution of the management target process, the user 1 remembers an emotion of anger, and the survey target period is determined based on the anger expression time at which the emotion is expressed. To be done. Then, only the log messages acquired within the survey target period are output as the survey target log message 4. As a result, it is possible to reduce the amount of data to be investigated when investigating a defect in the management target processing.

In the example of FIG. 1, the observation data 3a and 3b acquired at the times t1 and t2 do not show the anger feeling to the user 1. On the other hand, the observation data 3c acquired at the time t3 shows the emotion of anger to the user 1. Therefore, the processing unit 12 determines that the time t3 is the anger expression time. In this case, the processing unit 12 determines, for example, a period from time t4 to time t3, which is a predetermined time earlier than time t3, as the investigation target period. Time t4 is a time obtained by subtracting the human anger duration from time t3.

Log messages are registered in the execution log 11a at time T1 and time T2. Time T1 is a time before the survey period, but time T2 is a time within the survey period. Therefore, the processing unit 12 outputs the log message corresponding to the time T2 as the check target log message 4.

The person in charge of investigating the defect of the management target process investigates the cause of the defect based on the investigation target log message 4. At that time, the amount of data to be investigated may be smaller than that of the entire execution log 11a. Moreover, it is highly possible that the log message 4 to be investigated contains information indicating the cause of the defect. That is, when the user 1 feels angry with respect to the defect of the management target process, it is considered that the defect occurs in the management target process before the user 1's anger expression time. Further, the emotion of the anger of the user 1 lasts for the duration of the anger based on the physiological reaction of the human due to the occurrence of the defect in the processing to be managed. Therefore, by setting the period of the length of the anger duration up to the anger expression time of the user 1 as the survey target period, the time when the failure occurred in the managed process is included in the survey target period. Becomes Therefore, it can be expected that the log message acquired within the survey period includes information indicating the cause of the failure.

In addition, the processing unit 12 transmits the investigation target log message 4 to the external device when the instruction to send the information regarding the defect in the management target processing to the external device is input, so that the unnecessary investigation target log message is transmitted. 4 does not need to be extracted and transmitted. For example, if the survey target log message 4 is transmitted to the external device each time the anger emotion of the user 1 is detected, the survey target is included until the user 1 becomes displeased due to a cause other than the failure of the management target processing. Log message 4 is sent. On the other hand, for example, when the user 1 inputs an inquiry to the user support, the processing unit 12 sends the investigation target log message 4 to the user support server, so that the investigation target log irrelevant to the failure of the management target processing. It is possible to prevent the message 4 from being transmitted.

Each time the processing unit 12 acquires the observation data 3a, 3b, 3c, the processing unit 12 associates with the acquisition time of the observation data 3a, 3b, 3c and expresses the emotion of the anger of the user 1 represented in the observation data. The numerical value indicating the degree may be stored in the storage unit 11 in time series. In this case, the processing unit 12 specifies, from the storage unit 11 at a predetermined timing, one or more anger expression periods in which numerical values equal to or greater than the threshold value continue in time series. Then, the processing unit 12 determines the survey target period for each of the identified anger expression periods. Thereby, for example, when a defect of the management target process repeatedly occurs intermittently, the investigation target period can be determined for each of the generated defects. As a result, it is possible to make the log message 4 indicating the malfunction of the management target process that has occurred in the past the investigation target log message 4 without leaking.

[Second Embodiment]
Next, a second embodiment will be described. The second embodiment is intended to reduce the amount of log data to be transferred to a support server when a problem occurs during execution of application software in an information terminal such as a personal computer or a smartphone. Hereinafter, the application software will be simply referred to as “app”. The information terminal according to the second embodiment is an example of the information processing device 10 according to the first embodiment.

First, the importance of reducing the amount of log data transferred to the server for supporting the application will be described. In recent years, an increasing number of personal computer applications or smartphone applications are downloaded via the Internet. Requests for improvement and inquiries regarding such problems while using the application are not made through two-way communication between users and support such as face-to-face contact or telephone contact, but by unilateral short-text communication via the Internet such as email or inquiry form. Often done.

The investigation of the defect of the application by the support person is performed by comparing the result of hearing the defect occurrence situation from the user with the execution log output by the execution of the application. However, when accepting inquiries from users via email or inquiry form, which is often used for application support, it is difficult for the users to hear the details of the defect. Therefore, the execution log becomes more important for investigating the malfunction of the application.

Therefore, the application is designed to output log messages even for processing of extremely detailed events. As a result, a large number of execution logs are output in a short time while the application is being executed. When the execution log is sent from the information terminal to the server, for example, if the user can separate the message related to the failure and the unrelated log message, only the log message related to the failure should be sent from the information terminal to the server. You can However, a user who has not exchanged detailed information with a support person by telephone or the like cannot separate a message related to a defect from a message unrelated to the defect. Therefore, the user will send all the execution logs accumulated in the information terminal to the support server.

If a large amount of execution logs are sent to the server, problems will occur for both the support side and the user side. The problem with the support side is that the support side has to investigate a large amount of all execution logs, so the investigation takes a lot of labor and time. The problem on the user side is that a large amount of execution logs must be sent to the server, and even if the cause of the failure is an event that can be easily identified, it is necessary to wait for a long time to receive support. In addition, when the user uses the network in a charge system in which the fee is charged on a pay-as-you-go basis, there is a problem that the user will incur a non-negligible cost for transmitting the execution log.

Therefore, in the second embodiment, the information terminal identifies the period including the defect occurrence time and sends only the log message within the corresponding period to the server, so that the data to be investigated when investigating the defect of the application Reduce the amount. As a result, it is possible to reduce the research effort and the time required for the research on the support side. In addition, it is possible to reduce the waiting time for support regarding malfunctions on the user side and reduce the burden of network usage fees.

FIG. 2 is a diagram showing an example of a system configuration. A server 31 for supporting a user 41 of the information terminal 100 is provided in a support center 30 of a company that sells the information terminal 100 or a company that sells an application executed on the information terminal 100. The information terminal 100 is connected to the server 31 via the network 20. The user 41 of the information terminal 100 makes an inquiry to the support center 30 for a solution to the problem when a problem occurs while the application is being executed on the information terminal 100. For example, the user 41 inputs the inquiry content to the information terminal 100 and transmits the inquiry content to the server 31. At this time, the information terminal 100 also transmits the execution log of the application in which the failure has occurred to the server 31. The support staff member 32 of the support center 30 identifies the cause of the failure based on the inquiry content and the execution log sent from the information terminal 100 to the server 31.

When the information terminal 100 transmits the execution log to the server 31, the information terminal 100 estimates the time when the failure occurs based on the emotional change of the user 41 when the information terminal 100 is used. For example, when a failure occurs in the application, the greater the effect of the failure, the more uncomfortable the user 41 becomes, and the reflexive emotion of anger. Therefore, when the user 41 of the information terminal 100 is in an angry state, there is a possibility that a malfunction of the application causing the anger occurs.

Therefore, the information terminal 100 continuously observes the emotional state of the user 41 with the camera 108 and the microphone 109, and obtains the time when the user 41 remembers the emotion of anger and expresses the emotion. For example, the information terminal 100 can analyze the image of the user 41 captured by the camera 108 and specify the time when the emotion of anger is expressed by the user 41. The information terminal 100 can also analyze the sound collected by the microphone 109 and specify the time when the emotion of anger is expressed by the user 41.

The information terminal 100 estimates the defect occurrence time based on the time when the emotion of anger is expressed by the user 41. For example, when a person feels anger, the duration of one anger is limited. Therefore, the information terminal 100 estimates that the malfunction of the application has occurred within the duration of the anger duration including the time when the user 41 expresses the emotion of anger.

The duration of anger based on the human physiological reaction is, for example, 90 seconds from the occurrence of the cause of anger. That is, it is known that when a person feels anger, the chemical substance released from the brain keeps the body full of anger, but the chemical substance disappears from the body 90 seconds after the anger is triggered. Therefore, the information terminal 100 uses this biological fact to estimate the defect occurrence time, which is difficult to obtain from the comment of the defect report from the user, in a time width according to the duration of anger. By being able to estimate the failure occurrence time, the information terminal 100 can exclude log messages irrelevant to the failure from a large number of application execution logs and send only log messages related to the failure to the server 31.

FIG. 3 is a diagram illustrating an example of hardware of the information terminal. The information terminal 100 is entirely controlled by a processor 101. A memory 102 and a plurality of peripheral devices are connected to the processor 101 via a bus 112. The processor 101 may be a multiprocessor. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor). At least a part of the function realized by the processor 101 executing the program may be realized by an electronic circuit such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device).

The memory 102 is used as a main storage device of the information terminal 100. The memory 102 temporarily stores at least part of an OS (Operating System) program or an application program to be executed by the processor 101. Further, the memory 102 stores various data used for processing by the processor 101. As the memory 102, for example, a volatile semiconductor storage device such as a RAM (Random Access Memory) is used.

Peripheral devices connected to the bus 112 include a storage device 103, a graphic processing device 104, an input interface 105, an optical drive device 106, a device connection interface 107, a camera 108, a microphone 109, a speaker 110, and a network interface 111. ..

The storage device 103 electrically and magnetically writes and reads data to and from a built-in recording medium. The storage device 103 is used as an auxiliary storage device of a computer. The storage device 103 stores an OS program, application programs, and various data. As the storage device 103, for example, a HDD (Hard Disk Drive) or SSD (Solid State Drive) can be used.

A monitor 21 is connected to the graphic processing device 104. The graphic processing device 104 displays an image on the screen of the monitor 21 according to an instruction from the processor 101. Examples of the monitor 21 include a display device using an organic EL (Electro Luminescence) and a liquid crystal display device.

A keyboard 22 and a mouse 23 are connected to the input interface 105. The input interface 105 transmits signals sent from the keyboard 22 and the mouse 23 to the processor 101. Note that the mouse 23 is an example of a pointing device, and other pointing devices can be used. Other pointing devices include touch panels, tablets, touch pads, trackballs, and the like.

The optical drive device 106 uses laser light or the like to read the data recorded on the optical disc 24. The optical disc 24 is a portable recording medium on which data is recorded so that it can be read by reflection of light. The optical disc 24 includes a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), and a CD-R (Recordable)/RW (ReWritable).

The device connection interface 107 is a communication interface for connecting peripheral devices to the information terminal 100. For example, the device connection interface 107 can be connected to the memory device 25 and the memory reader/writer 26. The memory device 25 is a recording medium having a function of communicating with the device connection interface 107. The memory reader/writer 26 is a device that writes data in the memory card 27 or reads data from the memory card 27. The memory card 27 is a card-type recording medium.

The camera 108 captures an image of the front of the information terminal 100 and sends the image data to the processor 101. The microphone 109 converts a sound around the information terminal 100 into an electric signal and transmits the electric signal to the processor 101. The speaker 110 outputs sound based on the voice data sent from the processor 101.

The network interface 111 is connected to the network 20. The network interface 111 transmits/receives data to/from another computer or communication device via the network 20.

The information terminal 100 can realize the processing functions of the second embodiment with the above hardware configuration. The information processing apparatus 10 shown in the first embodiment can also be realized by the same hardware as the information terminal 100 shown in FIG.

The information terminal 100 realizes the processing function of the second embodiment by executing a program recorded in a computer-readable recording medium, for example. The program describing the processing content to be executed by the information terminal 100 can be recorded in various recording media. For example, a program to be executed by the information terminal 100 can be stored in the storage device 103. The processor 101 loads at least a part of the programs in the storage device 103 into the memory 102 and executes the programs. Further, the program to be executed by the information terminal 100 may be recorded in a portable recording medium such as the optical disc 24, the memory device 25, the memory card 27 or the like. The program stored in the portable recording medium becomes executable after being installed in the storage device 103 under the control of the processor 101, for example. Further, the processor 101 can directly read the program from the portable recording medium and execute it.

Next, the function of the information terminal 100 for transmitting the execution log of the application when a failure occurs in the application will be described.
FIG. 4 is a block diagram showing functions of the information terminal. The information terminal 100 includes a storage unit 120, an emotion recognition unit 130, an application execution unit 140, and a survey log transmission unit 150.

The storage unit 120 stores an emotion log 121 and an execution log 122. The emotion log 121 is information indicating a periodical measurement result of the presence or absence of the anger emotion of the user 41. The execution log 122 is information indicating a history of processing executed by the application. For example, a part of the storage area of the memory 102 or the storage device 103 is used as the storage unit 120.

The emotion recognition unit 130 determines whether there is a feeling of anger of the user based on the image data acquired from the camera 108 or the voice data acquired from the microphone 109. For example, the emotion recognition unit 130 recognizes the movement of the facial expression muscle based on the image data showing the face of the user 41, and the movement of the facial expression muscle matches the movement of the facial expression muscle when the face is a known angry face. The degree of anger is calculated based on the degree. It is also known that when a person has an emotion of anger, a specific muscle of the face becomes tense, and due to the effect of the tension of the muscle, a sound of a particular wavelength is mixed when the person utters a voice. .. Therefore, the emotion recognition unit 130 detects a sound of a specific wavelength emitted when the user 41 has an angry emotion, based on the voice data including the voice of the user 41, and detects the sound. The degree of anger is calculated according to the degree. When the emotion recognition unit 130 periodically determines whether there is anger, the emotion recognition unit 130 registers the determination result in the emotion log 121 in the storage unit 120.

The application execution unit 140 executes the application. For example, the application execution unit 140 recognizes an instruction from the user 41 based on an input from the camera 108 or the microphone 109, and executes processing according to the instruction. If the input is from the microphone 109, the application execution unit 140 recognizes the word spoken by the user 41 from the input voice data, and executes the process indicated by the word. For example, when the user 41 says “Tell me about tomorrow's weather”, the application executing unit 140 acquires weather information via the network 20 and displays the weather forecast for tomorrow on a screen or by voice.

The application execution unit 140 also has an online improvement request function for the support center 30 that supports the application. For example, the application execution unit 140 displays the improvement request screen for inputting the improvement request content in response to the voice input from the user 41 or the input by the keyboard operation. Then, the application execution unit 140 transmits an improvement request message including the improvement request content input on the improvement request screen to the server 31 of the support center 30. When transmitting the improvement request message, the application execution unit 140 outputs a survey log transmission instruction to the survey log transmission unit 150.

The survey log transmission unit 150 generates a survey log including some log messages registered in the execution log 122 based on the emotion log 121 in response to the survey log transmission instruction from the application execution unit 140. To do. For example, the investigation log transmitting unit 150 specifies a time period in which the user 41 has an emotion of anger of a predetermined amount or more based on the emotion log 121. Next, the investigation log transmitting unit 150 extracts, from the execution log 122, the log message registered in the specified time period and the predetermined period immediately before the specified time period (for example, 90 seconds). Then, the investigation log transmission unit 150 generates an investigation log including the extracted log message. The survey log transmission unit 150 transmits the generated survey log to the server 31 of the support center 30.

The line connecting the respective elements shown in FIG. 4 shows a part of the communication path, and a communication path other than the illustrated communication path can be set. Further, the function of each element shown in FIG. 4 can be realized, for example, by causing a computer to execute a program module corresponding to the element.

Next, the emotion log 121 and the execution log 122 stored in the storage unit 120 will be described in detail.
FIG. 5 is a diagram showing an example of the emotion log. In the emotion log 121, a microphone input anger probability and a camera input anger probability are set in association with the time when the emotion of the user 41 is determined.

The microphone input anger probability is a numerical value indicating the degree of anger of the user 41, which is calculated based on the voice data collected by the microphone 109. For example, the microphone input anger probability is a numerical value of 0 to 1. The higher the microphone input anger probability value, the higher the anger degree of the user 41.

The camera input anger probability is a numerical value indicating the degree of anger of the user 41, which is calculated based on the image data captured by the camera 108. For example, the camera input anger probability is a numerical value of 0 to 1. The larger the camera input anger probability value, the higher the anger degree of the user 41.

FIG. 6 is a diagram showing an example of the execution log. In the execution log 122, the output log message is set in association with the time when the application execution unit 140 outputs the log message. For example, when the application executing unit 140 recognizes the words of the user 41 based on the voice data, a log message including a character string indicating the recognized words is set in the execution log 122.

When an improvement request for an application is made by the information terminal 100 having the above-described configuration, the investigation log in which log messages not related to the content of the improvement request are excluded from the execution log 122 of the application is displayed on the server of the support center 30. 31 can be sent. The outline of the transmission process of the investigation log will be described below with reference to FIGS. 7 to 9.

FIG. 7: is a figure which shows an example of the reaction of the information terminal according to a user's input. In the example of FIG. 7, it is assumed that the user 41 operates the information terminal 100 via the assistant application running on the information terminal 100. The application execution unit 140 that executes the assistant application is an application that interprets a voice input by the user 41 and operates the function of the information terminal 100 according to the voice.

The user 41 is voice-inputting “Tell me about tomorrow's weather” at time T11. The application executing unit 140 correctly interprets the voice input, and outputs the voice "Tomorrow's weather is fine" at time T12.

After that, the user 41 voice-inputs "activate function A" at time T13. The application execution unit 140 mistakenly recognizes “function A” as “function E” and activates the function E at time T14. The user 41 is angry and has an angry expression because the application executing unit 140 is erroneously performing voice recognition.

The user 41 whose anger has subsided has input “inquiry” by voice at time T15 in order to request improvement of the function of the assistant application. The application execution unit 140 correctly interprets the voice input, and displays the improvement request screen 50 used for an inquiry to the support center 30 or an improvement request at time T16.

FIG. 8 is a diagram showing an example of the improvement request screen. The improvement request screen 50 is provided with a text box 51 for inputting the content of the improvement request and a button 52 for transmitting the improvement request. The user 41 uses the input device such as the keyboard 22 to input a character string indicating the content of the improvement request into the text box 51. When the input of the content of the improvement request is completed, the user 41 selects the button 52 with the pointer of the mouse 23 and presses the button of the mouse 23. Then, the application execution unit 140 transmits an improvement request message including the character string input in the text box 51 to the server 31.

Further, the application execution unit 140 instructs the survey log transmission unit 150 to transmit the survey log along with the transmission of the improvement request message. Then, the investigation log transmission unit 150 extracts a log message that may be related to the improvement request from the execution log 122, and transmits the investigation log including the extracted log message to the server 31.

FIG. 9 is a diagram illustrating an example of the investigation log transmission process. The survey log transmission unit 150 determines the survey target period based on the emotion log 121. For example, the investigation log transmitting unit 150 identifies a time when the microphone input anger probability or the camera input anger probability in the emotion log 121 exceeds a predetermined threshold. The threshold of the anger probability is “0.5”, for example.

FIG. 9 is a bar graph showing the microphone input anger probability and the camera input anger probability at times t11 to t16 when the emotional states immediately after the times T11 to T16 shown in FIG. 7 are determined. The microphone input anger probability has a value of "0" when there is no voice input from the user 41. As shown in the example of FIG. 7, the user 41 is angry with the operation of the information terminal 100 at time T14. Therefore, in the emotion log 121, the value of the camera input anger probability determined at time t14 immediately after time T14 exceeds the threshold value “0.5”. This time t14 is the anger expression time of the user 41.

For example, the investigation log transmitting unit 150 sets a time period of 90 seconds before the time as the investigation target period based on a time when at least one of the microphone input anger probability and the camera input anger probability exceeds a threshold value. In the example of FIG. 9, 90 seconds until time t14 is the investigation target period. Therefore, the investigation log transmission unit 150 extracts, from the execution log 122, the log message recorded within the period of 90 seconds until the time t14, and generates the investigation log 60 including the extracted log message.

Then, the survey log transmission unit 150 transmits the generated survey log 60 to the server 31. As a result, the improvement request message and the investigation log 60 are transmitted to the server 31.

FIG. 10 is a diagram showing an example of information transmitted to the server. As shown in FIG. 10, the improvement request message 61 and the investigation log 60 are transmitted from the information terminal 100 to the server 31. The support staff member 32 of the support center 30 investigates the cause of the dissatisfaction of the user 41 based on the improvement request message 61 and the investigation log 60.

For example, the improvement request message 61 includes a character string "There are many erroneous recognitions of voice input." Simply saying "there is a large number of erroneous recognitions of voice input" does not indicate what kind of phrase of the voice input is erroneously recognized. Therefore, the support staff 32 refers to the investigation log 60. Then, it can be understood that what the user 41 has pointed out as erroneous recognition is in the voice recognition process of "activate function E" by the application execution unit 140 of the information terminal 100.

In this way, the period that is highly likely to be related to the content of the improvement request is set as the survey target period, and among the log messages in the application execution log 122, only the log messages output within the survey target period are included in the survey log. By including it in 60, it becomes easy to conduct a survey in response to the improvement request. If all the log messages in the execution log 122 shown in FIG. 9 are to be investigated, there are many log messages indicating voice recognition, and it is unclear which voice recognition was erroneous recognition. On the other hand, in the investigation log 60 shown in FIG. 10, the log message indicating the voice recognition is limited, and the voice recognition process of the erroneous recognition can be easily identified.

For example, the improvement request message 61 shown in FIG. 10 does not describe in which keyword the failure occurred, but among the log messages included in the investigation log 60, only the log message at time T13 is related to voice recognition. Absent. Therefore, it is possible to specify the occurrence time of the defect and the recognition result that is erroneously recognized. As a result, the support staff 32 can grasp a defect that a word similar to “function E” is mistakenly recognized as “function E”.

Next, the processing executed by the emotion recognition unit 130, the application execution unit 140, and the research log transmission unit 150 in the information terminal 100 to transmit the research log 60 will be described in detail with reference to a flowchart.

First, a procedure of emotion recognition processing executed by the emotion recognition unit 130 will be described.
FIG. 11 is a flowchart showing an example of the procedure of emotion recognition processing. Hereinafter, the process illustrated in FIG. 11 will be described in order of step number.

[Step S101] The emotion recognition unit 130 starts recording ambient sound by the microphone 109 when the application execution unit 140 starts executing the application. For example, the emotion recognition unit 130 converts the sound wave input to the microphone 109 into voice data and stores the voice data in the memory 102 or the storage device 103.

[Step S102] The emotion recognition unit 130 waits for a certain period of time. For example, the emotion recognition unit 130 waits for 10 seconds.
[Step S103] The emotion recognition unit 130 performs a microphone input emotion determination process. Details of the microphone input emotion determination process will be described later (see FIG. 12).

[Step S104] The emotion recognition unit 130 performs a camera input emotion determination process. Details of the camera input emotion determination process will be described later (see FIG. 13).
[Step S105] The emotion recognition unit 130 determines whether or not the application execution unit 140 has finished executing the application. The emotion recognition unit 130 ends the emotion recognition process if the execution of the application is completed. Further, the emotion recognition unit 130 advances the process to step S102 if the execution of the application is not completed.

In this way, the microphone input emotion determination process and the camera input emotion determination process are repeatedly executed at constant time intervals while the application continues to be executed.
Next, the procedure of the microphone input emotion determination process will be described in detail.

FIG. 12 is a flowchart showing an example of the procedure of the microphone input emotion determination process. Hereinafter, the process illustrated in FIG. 12 will be described in order of step number.
[Step S111] The emotion recognition unit 130 acquires voice data for a certain time from the memory 102 or the storage device 103. For example, the emotion recognition unit 130 acquires voice data for the same time (10 seconds) as the waiting time in step S102 of FIG.

[Step S112] The emotion recognition unit 130 performs a Fast Fourier Transform (FFT) on the waveform of the sound indicated by the acquired voice data. The FFT can obtain the frequency and strength of the audio signal included in the acquired audio data.

[Step S113] The emotion recognition unit 130 determines that a signal whose strength exceeds a preset threshold value in the frequency range (200 to 3,000 Hz) of the vowel is generated based on the voice data for each frequency generated by the FFT. Determine if there is. For example, if the acquired voice data includes the voice of the user 41, there is a corresponding signal in the acquired voice data. If there is a corresponding signal, emotion recognition section 130 advances the process to step S115. If there is no corresponding signal, the emotion recognition unit 130 advances the process to step S114.

[Step S114] The emotion recognition unit 130 adds a record in which the current time is set to the emotion log 121, and records “0.00” in the microphone input anger probability of the added record. After that, the emotion recognition unit 130 ends the microphone input emotion determination process.

[Step S115] The emotion recognition unit 130 performs emotion determination processing based on voice data. For example, the emotion recognition unit 130 calculates a microphone input anger probability based on the intensity of a signal of a predetermined frequency that is emitted when the user 41 feels anger.

[Step S116] The emotion recognition unit 130 adds a record in which the current time is set to the emotion log 121, and records the value of the microphone input anger probability calculated in step S115 as the microphone input anger probability of the added record.

In this way, the emotion recognition unit 130 determines the presence or absence of the voice of the user 41 based on whether or not the voice data includes the waveform of the frequency of the vowel, and the emotion is recognized only when the voice of the user 41 is recorded. Judgment processing is performed. As a result, the emotion determination process based on the voice data can be suppressed during the period in which the voice of the user 41 is not recorded, and the process efficiency can be improved.

Next, the procedure of the camera input emotion determination process will be described in detail.
FIG. 13 is a flowchart showing an example of the procedure of the camera input emotion determination process. In the following, the process illustrated in FIG. 13 will be described in order of step number.

[Step S121] The emotion recognition unit 130 acquires current image data from the camera 108.
[Step S122] The emotion recognition unit 130 analyzes the image data and performs a face detection process in the image. For example, the emotion recognition unit 130 defines the relative positions and sizes of facial parts (eyes, nose, etc.) as the features of a human face, and searches for images that match the features from the images represented by the image data. To do.

[Step S123] The emotion recognition unit 130 determines whether a human face is present in the image data. The emotion recognition unit 130 advances the process to step S125 if the image data includes at least one face image. If the image data does not include a face image, the emotion recognition unit 130 advances the process to step S124.

[Step S124] The emotion recognition unit 130 records “0.00” in the camera input anger probability of the record added in the emotion log 121 in the immediately preceding microphone input emotion determination process. After that, the emotion recognition unit 130 ends the camera input emotion determination process.

[Step S125] The emotion recognition unit 130 performs emotion determination processing based on the face image. For example, the emotion recognition unit 130 calculates the camera input anger probability based on the degree of matching between the features of the face image and the features of the angry face.

[Step S126] The emotion recognition unit 130 records the value of the camera input anger probability calculated in step S125 as the camera input anger probability of the record added to the emotion log 121 in the immediately preceding microphone input emotion determination process.

As described above, the emotion recognition unit 130 performs the emotion determination process based on the face image only when the face of the user is included in the image data. As a result, the emotion determination process based on the face image can be suppressed during the period when the user 41 is not around the information terminal 100, and the process efficiency can be improved.

Next, the procedure of the application execution process will be described in detail.
FIG. 14 is a flowchart showing an example of the procedure of application execution processing. Hereinafter, the process illustrated in FIG. 14 will be described in order of step number.

[Step S201] The application execution unit 140 receives an input from the user 41 by an input from the keyboard or a voice input. Upon receiving the input, the application execution unit 140 records a log message including the input content in the execution log 122.

[Step S202] The application executing unit 140 determines whether the input content is an inquiry to the support center 30. For example, the application execution unit 140 determines that it is an inquiry when the voice input “inquiry” is acquired. If it is an inquiry, the application execution unit 140 advances the process to step S204. If the inquiry is not an inquiry, the application execution unit 140 advances the process to step S203.

[Step S203] The application execution unit 140 executes processing according to the input. At this time, the application execution unit 140 records a log message indicating the executed processing in the execution log 122. The application execution unit 140 then advances the process to step S209.

[Step S204] The application execution unit 140 displays the improvement request screen 50 (see FIG. 8) on the monitor 21. At this time, the application execution unit 140 records a log message indicating that the improvement request screen 50 is displayed in the execution log 122.

[Step S205] The application execution unit 140 receives an input of improvement request content in the text box 51 of the improvement request screen 50. The application execution unit 140 records a log message indicating the input content in the execution log 122 when the input content is confirmed (for example, when the “send” button 52 is pressed).

[Step S206] The application execution unit 140 determines whether or not the "send" button 52 has been pressed by operating the mouse 23. When the “send” button 52 is pressed, the application execution unit 140 advances the process to step S207. If the “send” button 52 is not pressed, the application execution unit 140 advances the process to step S205 and accepts an improvement request input. When the “send” button 52 is pressed, the application executing unit 140 records a log message indicating the pressing of the button 52 in the execution log 122.

[Step S207] The application execution unit 140 transmits an improvement request message to the server 31 of the support center 30. The transmitted improvement request message includes a character string indicating the content of the improvement request input in the text box 51. At this time, the application execution unit 140 records a log message indicating the transmission of the improvement request message in the execution log 122.

[Step S208] The application execution unit 140 instructs the survey log transmission unit 150 to transmit the survey log 60 to the server 31. At this time, the application execution unit 140 records a log message indicating the output of the transmission instruction of the investigation log 60 in the execution log 122.

[Step S209] The application execution unit 140 determines whether or not an instruction to end application execution has been input. When the application execution end instruction is input, the application execution unit 140 ends the application execution process. If the instruction to end the application execution is not input, the application execution unit 140 advances the process to step S201.

In this way, the application execution unit 140 records a log message indicating the executed processing in the execution log 122 each time it executes some processing according to the application. As a result, the execution log 122 records detailed information regarding the execution of the application.

The application execution unit 140 also instructs the survey log transmission unit 150 to transmit the survey log 60 when transmitting the improvement request message to the server 31. As a result, the investigation log transmission process by the investigation log transmission unit 150 is started.

Next, the investigation log transmission process will be described in detail.
FIG. 15 is a flowchart (1/2) showing an example of the procedure of the investigation log transmission process. Hereinafter, the process illustrated in FIG. 15 will be described in order of step number.

[Step S301] When the survey log transmission unit 150 receives the survey log 60 transmission instruction from the application execution unit 140, the survey log transmission unit 150 executes the following processes from step S302.
[Step S302] The survey log transmission unit 150 selects records one by one from the top row of the emotion log 121. Note that if the last record in the emotion log 121 has already been selected, the survey log transmission unit 150 ends the process of step S302 without selecting a record. The top row of the emotion log 121 is the row of the record with the oldest time registered in the emotion log 121.

[Step S303] The investigation log transmitting unit 150 determines whether or not a record has been selected in step S302. If the unselected record can be selected from the emotion log 121, the survey log transmission unit 150 advances the process to step S304. In addition, when the survey log transmission unit 150 has already selected the last record in the emotion log 121 and cannot select a new record, the process proceeds to step S311 (see FIG. 16).

[Step S304] The investigation log transmitting unit 150 determines whether the microphone input anger probability or the camera input anger probability of the selected record is equal to or greater than a threshold value. If at least one of the anger probabilities is equal to or higher than the threshold value, the survey log transmitting unit 150 advances the process to step S305. Further, the investigation log transmitting unit 150 advances the process to step S307 if any of the anger probabilities is less than the threshold value.

[Step S305] The survey log transmitting unit 150 determines whether or not the microphone input anger probability or the camera input anger probability of the record registered one row before the selected record in the emotion log 121 is equal to or greater than a threshold value. .. If at least one of the anger probabilities is equal to or higher than the threshold value, the investigation log transmitting unit 150 advances the process to step S302. Further, the investigation log transmitting unit 150 advances the process to step S306 if any anger probability is less than the threshold value.

[Step S306] The investigation log transmitting unit 150 registers the investigation target period whose start time is the time set in the selected record in the investigation target period list 70. At this point, the end time of the applicable survey period has not been set. After that, the investigation log transmitting unit 150 advances the process to step S302.

[Step S307] The investigation log transmitting unit 150 determines whether the microphone input anger probability or the camera input anger probability of the record registered one row before the selected record in the emotion log 121 is equal to or greater than a threshold value. .. If at least one of the anger probabilities is greater than or equal to the threshold value, the survey log transmitting unit 150 advances the process to step S308. Further, the investigation log transmitting unit 150 advances the process to step S302 if any anger probability is less than the threshold value.

[Step S308] The survey log transmitting unit 150 sets the time set in the selected record as the end time of the survey period whose end time is not set. After that, the investigation log transmitting unit 150 advances the process to step S302.

In the example illustrated in FIG. 15, when a plurality of records whose anger probability is equal to or higher than a threshold value are consecutive in time series in the emotion log 121, the survey log transmission unit 150 determines a survey target based on those records. The period is summarized in one. That is, while the records in which the anger probability is equal to or more than the threshold value are continuous, “YES” is obtained in steps S304 and S305, and the end time of the survey target period is not set. Then, when there is a record whose anger probability is lower than the threshold value after the record whose anger probability is equal to or higher than the threshold value, the time of the record is set as the end time of the investigation target period. In this way, the survey target period is set in the survey target period list 70 for each period (the anger expression period of the user 41) including the time of consecutive records whose anger probability is equal to or higher than the threshold value.

That is, when there are a plurality of anger expression periods, a plurality of survey target periods are set in the survey target period list 70. As a result, when the application execution failure occurs a plurality of times with an interval, a plurality of investigation target periods including the plurality of failure occurrence times are set in the investigation target period list 70. By extracting the log message of the investigation target based on the investigation target period list 70 as described above, it becomes possible to extract the log message related to each of the plurality of past failures without leaking.

FIG. 16 is a flowchart (2/2) showing an example of the procedure of the investigation log transmission process. In the following, the process illustrated in FIG. 16 will be described in order of step number.
[Step S311] The investigation log transmission unit 150 selects records one by one from the beginning of the execution log 122. In addition, when the last record in the execution log 122 has been selected, the investigation log transmitting unit 150 ends the process of step S311 without selecting a record. The first line of the execution log 122 is the line of the record with the oldest time registered in the execution log 122.

[Step S312] The investigation log transmitting unit 150 determines whether or not a record has been selected in step S311. If the unselected record can be selected from the execution log 122, the investigation log transmission unit 150 advances the process to step S313. In addition, when the investigation log transmission unit 150 has already selected the last record in the execution log 122 and cannot select a new record, the process proceeds to step S316.

[Step S313] The investigation log transmitting unit 150 refers to the investigation target period list 70 and compares the time at which the log message shown in the selected record was recorded with the investigation target period.

[Step S314] The investigation log transmitting unit 150 determines whether or not the time of the selected record is within any investigation target period. If the relevant time is within any of the survey target periods, the survey log transmitting unit 150 advances the process to step S315. If the relevant time is not included in any of the survey target periods, the survey log transmitting unit 150 advances the process to step S311.

[Step S315] The investigation log transmitting unit 150 writes the selected record in the investigation log 60. After that, the investigation log transmitting unit 150 advances the process to step S311.
[Step S316] The investigation log transmission unit 150 uploads the investigation log 60 to the server 31 when all records in the execution log 122 have been compared with the investigation target period.

In this way, among the log messages in the execution log 122, the investigation log 60 including only the log messages recorded within the investigation target period is transmitted to the server 31. The survey period is a predetermined period until the time when the user 41 expresses the feeling of anger, and the survey log 60 includes a log message regarding the processing of the application that causes the user 41 to get angry. Can be expected.

Hereinafter, with reference to FIGS. 17 to 19, a specific example of the log message extraction process of writing the log message recorded in the execution log 122 to the investigation log 60 will be described.
FIG. 17 is a diagram showing an example of input to the information terminal by the user. In the example of FIG. 17, the user 41 performs the activation operation of the voice assistant application on the information terminal 100 at the time “00:00:00”. In the information terminal 100, the application is activated in response to the activation operation, and the application execution unit 140 starts executing the application.

At the time “00:05:00”, the user 41 speaks to the information terminal 100 the keyword “keyword 1” for using the “function 1” of the voice assistant application. However, since the information terminal 100 does not execute the "function 1", the user 41 frowns at the time "00:05:11". Further, the user 41 speaks to the information terminal 100 at the time “00:05:15”, the keyword “keyword 1” for using the “function 1” of the voice assistant application. The information terminal 100 executes the "function 1" at the time "00:05:17" without any problem.

After that, the user 41 speaks out the keyword “keyword 2” for using the “function 2” of the voice assistant application at the time “01:00:00”. However, since the information terminal 100 does not execute the “function 2”, the user 41 frowns at the time “01:00:10”. Further, the user 41 speaks to the information terminal 100 at the time “01:00:20”, the keyword “keyword 2” for using the “function 2” of the voice assistant application. The information terminal 100 executes the "function 2" at the time "01:00:22" without any problem.

The user 41 speaks out "inquiry" at the time "01:30: 00" after the emotion of anger has subsided and the user was calm. In response to the voice input of “inquiry”, the information terminal 100 displays the improvement request screen 50 (see FIG. 8) for the voice assistant application at time “01:30:02”. The user 41 writes only "voice is sometimes not recognized" in the text box 51 for inputting the improvement request content on the improvement request screen, and presses the "send" button 52 at the time "01:35:00". ing.

In such a series of operation steps by the user 41, the information terminal 100 periodically performs emotion recognition processing and records the processing result in the emotion log 121.
FIG. 18 is a diagram showing an example of the result of the emotion recognition process recorded in the emotion log. As shown in FIG. 18, in the emotion log 121, the camera input anger probability (face_emotion_angle) “0.71” is recorded at the time “00:05:11.123”. Since “0.71” is greater than or equal to the threshold value “0.5”, the survey target period list 70 shows that the start time of the survey target period is 90 seconds before the time “00:05:11.123”. The time “00:03:41.123” is set. The anger probability shown in the next line at time "00:05:11.123" is less than the threshold value for both the microphone input anger probability (voice_emotion_angry) and the camera input probability. Therefore, in the survey target period list 70, the time “00:05:21.150” of the record in the next row is set as the end time of the survey target period whose start time is “00:03:41.123”. It is set.

In the emotion log 121, the camera input anger probability “0.54” is recorded at the time “01:00:12.231”. Since “0.54” is greater than or equal to the threshold value “0.5”, the survey target period list 70 indicates that the survey target period start time is 90 seconds before the time “01:00:12.231”. The time “00:58:42.231” is set. In addition, the anger probability shown in the next line at time “01:00:12.231” is less than the threshold value for both the microphone input anger probability and the camera input probability. Therefore, in the survey target period list 70, the time “01:00:22.231” of the record in the next row is set as the end time of the survey target period whose start time is “00:58:42.231”. It is set.

In this way, log messages to be written in the investigation log 60 are extracted from the execution log 122 based on the investigation period set in the investigation period list 70.
FIG. 19 is a diagram illustrating an example of extracting a log message from the execution log. As shown in FIG. 19, the execution log 122 records in detail the processing executed by the application execution unit 140, including the application activation and the operation to the improvement request screen shown in FIG. Of such execution logs 122, log messages recorded at times within the survey target period shown in the survey target period list 70 are extracted as targets for writing to the survey log 60, and are transmitted to the server 31.

As a result, the support staff member 32 of the support center 30 only needs to investigate the log message written in the investigation log 60, and the amount of log message to be investigated is small. Since the amount of log messages to be investigated is small, it is easier to investigate the cause of the problem than to investigate the entire execution log 122.

For example, in the log message within the survey target period “00:03:41.123 to 00:05:21.150”, the time is from “00:05:00” to “00:05:15” shown in FIG. The execution status of the processing corresponding to the performed voice input of “keyword 1” is shown. In addition, the log messages within the survey target period "00:58:42.231 to 01:00:22.231" have the time "01:00:00" to "01:00:20" shown in FIG. The execution status of the processing corresponding to the performed voice input of “keyword 2” is shown. By analyzing these log messages, the support staff can investigate in detail the situation in which the application executing unit 140 fails in voice recognition.

Further, the improvement request indicates that "voice is sometimes not recognized", but this alone does not tell when voice recognition may not have occurred. On the other hand, by referring to the time of the log message included in the investigation log 60, the time of occurrence of the defect can be grasped.

The information terminal 100 transmits the investigation log 60 to the server 31 when transmitting the improvement request input by the user 41 to the server 31. Thereby, unnecessary transmission of the investigation log 60 can be suppressed. For example, when the investigation log 60 is automatically transmitted to the server 31 when the expression of the anger feeling of the user 41 is detected, the investigation is performed even when the user 41 feels angry for a reason other than the malfunction of the application. The log 60 for use is transmitted. By transmitting the survey log 60 at the time of transmitting the improvement request from the user 41, it is possible to prevent the survey log 60 from being transmitted when the user 41 is angry for a reason other than the malfunction of the application. ..

Further, the support staff 32 does not know what kind of trouble has occurred even if he/she only refers to the investigation log 60 without requesting improvement from the user 41, and it is difficult to identify the cause. The information terminal 100 transmits the improvement request of the user 41 and the investigation log 60 to the server 31 at the same time, so that the support staff member 32 causes the support person 32 to describe the content of the failure indicated in the improvement request and the log message in the investigation log 60. It is possible to accurately identify the cause of the failure by comparing with.

[Other Embodiments]
In the second embodiment, the emotion recognition process is executed in the information terminal 100, but the emotion recognition service provided via the network can also be used.

FIG. 20 is a diagram showing an example of a system using the emotion recognition service. A server 80 that provides an emotion recognition service is connected to the network 20. For example, when the server 80 receives the image data from the information terminal 100, the server 80 calculates the anger feeling of the person whose image data is shown, and transmits the camera input anger probability indicating the anger feeling to the information terminal 100. To do. Further, when the server 80 receives the voice data from the information terminal 100, the server 80 calculates the degree of emotion of anger of the person based on the voice of the person included in the voice data, and the microphone input anger indicating the degree of emotion of anger is input. The probability is transmitted to the information terminal 100.

The information terminal 100 can cause the server 80 to execute the microphone input emotion determination processing and the camera input emotion determination processing. For example, the information terminal 100 causes the server 80 to execute the process of step S115 in the flowchart of the microphone input emotion determination process shown in FIG. In this case, the emotion recognition unit 130 of the information terminal 100 transmits voice data (10 seconds worth) in which the voice of the user 41 is recorded, for example, every 10 seconds to the server 80, and the server 80 notifies the user 80 of anger. Acquires the microphone input anger probability indicating the degree. Further, for example, the information terminal 100 causes the server 80 to execute the process of step S125 in the flowchart of the camera input emotion determination process shown in FIG. In this case, the emotion recognition unit 130 of the information terminal 100 transmits image data showing the face of the user 41 to the server 80, for example, every 10 seconds, and the server 80 inputs a microphone input indicating the degree of anger of the user. Get the anger probability.

By using the emotion recognition service provided by the server 80 in this way, the load on the information terminal 100 can be reduced. For example, when the information terminal 100 is a mobile terminal that operates on a battery, the load of the information terminal 100 can be reduced to suppress the power consumption and prolong the operating time of the battery.

When the emotion recognition service provided by the server 80 is used, the information terminal 100 determines the emotion based on the face image when, for example, the image shown in the latest image data is almost the same as the image of the previous image data. The execution of processing may be suppressed. In this case, the information terminal 100 sets the camera input anger probability calculated from the previous image data as the camera input anger probability of the latest image data, for example. As a result, when there is no change in the image data, it is not necessary to send the image data to the server 80, and the communication load can be reduced.

In the second embodiment, the survey log 60 is transmitted along with the transmission of the improvement request message, but the information terminal 100 may transmit the survey log 60 to the server 31 at another timing. .. For example, the information terminal 100 generates the investigation log 60 including the untransmitted log message in the server 31 each time the emotion determination process acquires the anger probability equal to or higher than the threshold, and the investigation log 60 is transmitted to the server 31. You may send it.

The information terminal 100 is not limited to the application, and software or hardware other than the application can also be the target of the defect investigation as long as it is a device or a function capable of outputting a log message according to the process.

Although the embodiment has been illustrated above, the configuration of each unit described in the embodiment can be replaced with another having the same function. In addition, other arbitrary components and steps may be added. Further, any two or more configurations (features) of the above-described embodiments may be combined.

1 User 2 Observation Device 2a Camera 2b Microphone 3a, 3b, 3c Observation Data 4 Survey Target Log Message 10 Information Processing Device 11 Storage Unit 11a Execution Log 12 Processing Unit

Claims (4)

A storage unit that stores an execution log that is an execution history of managed processes,
Every time the managed process is executed, a log message indicating the content of the executed managed process is registered in the execution log in association with the execution time of the managed process, and the state of the user is observed. The observation data indicating the state of the user is acquired from the observation device, and based on the expression state of the emotion of the user represented in the observation data, the anger table expressing the emotion of anger by the user. Judgment time, the period of time before the anger expression time by the duration of anger based on the physiological response of the human, including the period from the anger expression time, the investigation target period of the failure of the managed process. A processing unit that determines and extracts, from among the log messages included in the execution log, an investigation target log message associated with an execution time within the investigation target period;
Information processing device having a. Each time the processing unit acquires the observation data, in association with the acquisition time of the observation data, a numerical value indicating the degree of expression of the emotion of the anger of the user represented in the observation data in time series. It stores in the said memory|storage part, 1 or more anger expression periods with which the said numerical value more than a threshold value continues in a time series are specified from the said memory|storage part, and the said investigation target period is specified for each said anger expression period. decide,
Claim 1 Symbol placement of the information processing apparatus. The processing unit, when an instruction to send information regarding a defect in the management target processing to an external device is input, transmits the extracted investigation target log message to the external device,
The information processing apparatus according to claim 1 or 2 wherein. On the computer,
Every time a managed process is executed, a log message indicating the content of the executed managed process is registered in the execution log, which is the execution history of the managed process, in association with the execution time of the managed process. Then
Obtaining observation data indicating the state of the user from an observation device that observes the state of the user,
Based on the expression state of the emotion of the user represented in the observation data, to determine the anger expression time when the user expressed the feeling of anger,
Only the duration of anger based on the physiological reaction of the human, from the time before the anger expression time, including the period from the anger expression time , determines the investigation target period of the failure of the managed process,
Of the log messages included in the execution log, extract the investigation target log message associated with the execution time within the investigation target period,
A log management program that executes processing.

Images