JP2024083068A - Information processing method and information processing device - Google Patents

Abstract

[Problem] To improve the ease of use of a device by providing appropriate support to a user who has developed a specific emotion toward another user. [Solution] An information processing method for controlling an agent device 200 capable of executing multiple processes for users U1 and U2. When a predetermined process is selected from multiple processes in a state where multiple users U1 and U2 exist, this information processing method includes a detection process (step S506) for detecting a user U2 from among the multiple users U1 and U2 who has developed a specific emotion including at least one of a debt emotion and a gain emotion resulting from the selection, and a control process (steps S509, S511) for causing the agent device 200 to perform a performance for alleviating the specific emotion for user U2 in a manner that is difficult to perceive by users U1 other than user U2 when user U2 is detected. [Selected Figure] FIG.

Description

The present invention relates to an information processing method and an information processing device that controls a device capable of performing multiple processes for a user.

Conventionally, there are devices that can execute multiple processes for users. For example, a technology has been proposed that, if it is determined that the number of users in an elevator after the doors have been opened and closed is greater than the number of users before the doors were opened, output information from an output device in the elevator expressing gratitude to the users for their cooperation in boarding the elevator (see, for example, Patent Document 1).

JP 2022-14601 A

In the above-mentioned conventional technology, the information indicating gratitude is output from an output device in the elevator, so that the information is conveyed to both the new person in the elevator and the person originally in the elevator. In this case, it is conceivable that the person originally in the elevator would not want their feelings to be conveyed audibly to the new person. On the other hand, it is conceivable that when the new person in the elevator hears the information, they may feel awkward, unable to bear being in the situation, or feel sorry. In other words, the output of the information indicating gratitude may cause the new person in the elevator to have certain feelings, such as a sense of guilt, toward the person originally in the elevator.

The present invention aims to improve the ease of use (usability) of devices by providing appropriate follow-up to users who have specific feelings toward other users.

One aspect of the present invention is an information processing method for controlling a device capable of executing multiple processes for a user. This information processing method includes a detection process for detecting a user (first user) from among the multiple users who has experienced a specific emotion including at least one of a debt emotion and a gain emotion resulting from a selection of a specific process when multiple users are present, and a control process for causing the device to execute a performance that alleviates the specific emotion for the first user in a manner that is difficult to perceive by users other than the first user (second user) when the first user is detected.

According to the present invention, it is possible to improve the usability of a device by providing appropriate support to a user who has developed a particular emotion toward another user.

FIG. 1 is a simplified diagram showing an example of the configuration of the interior of a vehicle. FIG. 2 is a simplified diagram showing an example in which two users in a vehicle give different, conflicting instructions to an agent device. FIG. 3 is a front view showing, in a simplified form, an example of the external configuration of an agent device. FIG. 4 is a diagram showing an example of a system configuration of an information processing system installed in a vehicle. FIG. 5 shows an example of a transition when each state of an agent device is changed in response to the user's emotion after a process is executed in response to an instruction from the user. FIG. 6 shows an example of a transition when each state of the agent device is changed in response to the user's emotion after the process is executed in response to an instruction from the user. FIG. 7 is a flowchart showing an example of a control process in the information processing device. FIG. 8 is a flowchart showing an example of a control process in the information processing device. FIG. 9 is a flowchart showing an example of a control process in the information processing device.

The following describes an embodiment of the present invention with reference to the attached drawings.

[Agent device installation example]
Fig. 1 is a simplified diagram showing an example of the configuration of the interior of a vehicle C1. Fig. 1 shows an example of the exterior configuration when the area in front of the driver's seat and passenger seat (not shown) is viewed from the rear of the vehicle C1. To facilitate explanation, Fig. 1 omits the illustration of anything other than a dashboard 2, a steering wheel 3, a windshield 4, a rearview mirror 5, a camera 101, and an agent device 200.

The agent device 200 is a small robot installed on the dashboard 2 of the vehicle C1. In this embodiment, an example in which a robot that mimics a human being is used as the agent device 200 is shown. Note that FIG. 1 shows an example in which the agent device 200 is installed on the dashboard 2, but this is not limited thereto. For example, the agent device 200 may be installed on the upper part of the windshield 4. Also, FIG. 1 shows an example in which a robot that mimics a human face is used as the agent device 200, but this is not limited thereto. For example, the agent device 200 may be a robot that mimics a whole human body, a robot that mimics an animal such as a rabbit or a pig, a robot that mimics a virtual creature (for example, the face of an anime character), or a robot that mimics another object (for example, a television-type device, a radio-type device). In this way, it is possible to use an agent that has been mimicked as the agent device 200.

The agent device 200 executes various operations based on instructions from the information processing device 110 (see FIG. 4). For example, the agent device 200 outputs various information related to driving assistance when the user U1 drives the vehicle and surrounding facilities, etc., based on the control of the information processing device 110. As the driving assistance, a notification of a moving object in front or behind is assumed. For example, as a notification of a moving object in front, a voice output of "Be careful because there is a railroad crossing ahead" or a voice output of "There is a person ahead" can be used. In this way, the agent device 200 executes driving assistance. Also, for example, the agent device 200 executes various processes based on the control of the information processing device 110. For example, the agent device 200 executes processes according to instructions from the user U1 based on the instructions. Note that FIG. 1 shows an example in which, based on a voice S1 instruction from the user U1 to "Turn up the temperature of the air conditioner," a process of adjusting the temperature of the air conditioner is executed according to the instruction. In this case, the agent device 200 outputs voice information S2 indicating that the temperature adjustment of the air conditioner has been executed according to the instruction.

The camera 101 is provided on the ceiling inside the vehicle C1 and captures an image of a subject inside the vehicle C1 to generate an image (image data). The camera 101 is, for example, composed of one or more camera devices or image sensors capable of capturing an image of a subject. For example, the camera 101 can be provided on the upper part of the windshield 4, i.e., above the rearview mirror 5. Although FIG. 1 shows an example in which at least one camera 101 is provided, two or more imaging devices may be provided and all or some of the images from these imaging devices may be used. The installation locations of each imaging device are not limited to the example shown in FIG. 1 and can be changed as appropriate. One or more devices capable of capturing subjects present in all directions around the vehicle C1 and subjects inside the vehicle C1, for example, a 360-degree camera, may be used.

[Example of when different instructions are given to agent devices]
Fig. 2 is a simplified diagram showing an example of a case where two users U1 and U2 riding in a vehicle C1 give different, conflicting instructions to the agent device 200. Note that the example shown in Fig. 2 is the same as the example shown in Fig. 1 except that a user U2 is added to the configuration example shown in Fig. 1.

Figure 2 shows an example in which user U2, shivering from the cold, makes a speech S6 saying "Turn up the air conditioner temperature," and user U1, sweating from the heat, makes a speech S5 saying "Turn down the air conditioner temperature." In this way, when users U1 and U2 give conflicting instructions to the agent device 200, it is impossible for the agent device 200 to execute those instructions approximately simultaneously. For this reason, the agent device 200 selects and executes one of those instructions. For example, the agent device 200 can sense the situation inside the vehicle C1, users U1, U2, etc., and select and execute the appropriate instruction from among the instructions from users U1 and U2.

[Example of instruction selection when conflicting instructions are given by multiple users]
Here, a case where each instruction is issued by each of a plurality of users will be described. If each instruction is issued by each of a plurality of users at different timings, each instruction can be executed at the timing at which the instruction was issued. Here, a case where each instruction is issued by each of a plurality of users at approximately the same timing will be described. Note that approximately the same timing means, for example, an interval of about several seconds.

For example, if multiple users each issue an instruction with the same content at approximately the same time, the agent device 200 executes that instruction. For example, if users U1 and U2 issue the same instruction at the same time, such as "Turn up the air conditioner temperature" or "Turn on the radio," the instructions are the same, so the agent device 200 can execute the instruction.

For example, when multiple users issue different instructions at approximately the same time, the agent device 200 executes the instructions simultaneously or sequentially, unless the instructions are contradictory. For example, assume that user U1 issues an instruction to "turn up the air conditioner temperature" and user U2 issues an instruction to "turn on the radio" at approximately the same time. In this case, multiple users issue different instructions at approximately the same time, but since the instructions are not contradictory, the agent device 200 executes the instructions simultaneously or sequentially. Specifically, the agent device 200 executes control to raise the temperature setting of the air conditioner based on the instruction from user U1 to "turn up the air conditioner temperature," and executes control to turn on the radio based on the instruction from user U2 to "turn on the radio." These controls may be executed simultaneously or sequentially.

Also, for example, consider a case where multiple users each issue different instructions at approximately the same time, and these instructions are contradictory. For example, if user U1 issues an instruction to "turn down the air conditioner temperature" and user U2 issues an instruction to "turn up the air conditioner temperature" at approximately the same time, the instructions will be contradictory. Also, for example, if user U1 issues an instruction to "play radio station A" and user U2 issues an instruction to "play radio station B" at approximately the same time, the instructions will be contradictory.

In these cases, the agent device 200 cannot execute the conflicting instructions, and will select and execute one of the instructions. For example, as shown in FIG. 2, assume that user U1 issues an instruction to "lower the air conditioner temperature" and user U2 issues an instruction to "raise the air conditioner temperature" at approximately the same time. In this case, since users U1 and U2 issue conflicting instructions, it is impossible to execute these instructions at approximately the same time. In such a case, the agent device 200 (or the information processing device 110) selects an appropriate instruction from among the instructions issued by users U1 and U2 based on biometric information on users U1 and U2, the speech of users U1 and U2, the actions of users U1 and U2, the environment around users U1 and U2, etc.

For example, assume that user U1 continuously utters the voice "It's hot, it's hot" and that user U1 is sweating profusely and has unbuttoned his jacket, and that user U2 is determined to be in a normal state. In this case, the agent device 200 determines that user U1 is over the limit of heat, but user U2 can tolerate the cold, and selects to adopt user U1's instruction "Turn down the temperature of the air conditioner." On the other hand, assume that user U2 continuously utters the voice "It's cold, it's cold," and that user U2 is shivering and putting on a new jacket, and that user U1 is determined to be in a normal state. In this case, the agent device 200 determines that user U2 is over the limit of cold, but user U1 can tolerate the heat, and selects to adopt user U2's instruction "Turn up the temperature of the air conditioner." In this way, with regard to instructions related to the health of users U1 and U2, the agent device 200 selects and executes the instruction that is most suitable for the biological condition, etc. of each user from among the instructions. That is, the agent device 200 refers to the voices, movements, sweat, body temperature, etc. of users U1 and U2 to comprehensively determine which of them is in poor physical condition, and selects and executes the most appropriate instructions. In other words, the agent device 200 senses users U1 and U2 to determine which of them is in a critical situation, and selects and executes the instructions of the user in the critical situation. Note that the agent device 200 may also select and execute the most appropriate processing of instructions based on the urgency of each instruction, the execution time based on each instruction, the length of time it takes to complete the processing based on each instruction, etc.

Assume that user U1 issues the instruction "Play radio station A" and user U2 issues the instruction "Play radio station B" at approximately the same time. In this case, since users U1 and U2 issue conflicting instructions, it is impossible to execute these instructions at approximately the same time. In such a case, the agent device 200 similarly selects an appropriate instruction from among the instructions issued by users U1 and U2 based on the biometric information on users U1 and U2, the speech of users U1 and U2, the actions of users U1 and U2, the environment around users U1 and U2, etc. However, in the case of selecting a radio station, it is assumed that it may be difficult to select an appropriate broadcasting station based on each of these pieces of information. Therefore, for such instructions, a selection process may be performed based on a selection criterion that is set in advance.

The selection criteria here are criteria for selecting an instruction to be executed with priority when multiple conflicting and different instructions are issued. For example, the instruction to be selected with priority among multiple conflicting and different instructions may be determined based on the order in which the instructions were issued. In this case, it is possible to select the instruction issued first.

In addition, for example, when multiple conflicting and different instructions are issued, the user from whom the instruction is selected may be changed sequentially. For example, when multiple conflicting and different instructions are issued from users A to C, the instruction issued by user A is selected. In this case, when multiple conflicting and different instructions are issued from users A to C at the next timing, the instruction issued by user B is selected. Also, when multiple conflicting and different instructions are issued from users A to C at the next timing, the instruction issued by user C is selected.

Note that instructions to be prioritized may be set for each user. For example, if there are users A to C, the priority of user A may be set to 1st, that of user B to 2nd, and that of user C to 3rd. In this case, if multiple conflicting and different instructions are issued by users A to C, the instruction issued by user A is selected. Also, if multiple conflicting and different instructions are issued by users B and C, the instruction issued by user B is selected. Also, if multiple conflicting and different instructions are issued by users A and C, the instruction issued by user A is selected.

The instruction to be prioritized may also be determined based on the processing time required for the instruction. For example, assume that multiple conflicting instructions are issued by users A to C, and the processing time required for the instructions from user A is 11 minutes, the processing time required for the instructions from user B is 3 minutes, and the processing time required for the instructions from user C is 7 minutes. In this case, the instruction from user B is selected and executed first. Next, the instruction from user C is selected and executed, and finally the instruction from user B is selected and executed.

Note that conflicting instructions can be stored in the instruction information DB 132 (see FIG. 4). When conflicting instructions are issued, the instruction to be preferentially selected may be determined based on the instruction information stored in the instruction information DB 132 (see FIG. 4).

In this way, when multiple users issue different and conflicting instructions, at least one instruction selected from the multiple instructions will be executed. In this case, the user whose instruction is adopted will be in a relatively advantageous position, and the user whose instruction is not adopted will often be in a relatively disadvantaged position. In such a case, the user who is in a relatively advantageous position will often feel a sense of indebtedness, such as guilt or remorse, toward the user who is in a relatively disadvantaged position. In this way, when there are multiple users, the feelings of indebtedness (or negative feelings), such as guilt or remorse, that arise in some users toward other users will be referred to as indebtedness feelings in this embodiment. In other words, indebtedness feelings refer to feelings of indebtedness when a gain is made. In addition, indebtedness feelings are an example of a specific emotion.

On the other hand, users who are relatively at a disadvantage often feel dissatisfied with users who are relatively at a disadvantage. In this embodiment, when there are multiple users, dissatisfaction felt by some users toward other users is referred to as dissatisfaction. In other words, dissatisfaction means a feeling of dissatisfaction at having suffered a disadvantage. Dissatisfaction is an example of a specific emotion.

In this way, when multiple different and conflicting instructions are issued from multiple users, the user whose instructions are adopted may feel indebted to the other users. In this case, it is possible to provide some kind of follow-up to the user whose instructions are adopted. For example, assume that multiple different and conflicting instructions are issued from users U1 and U2, and the instruction from user U2 is adopted. In this case, user U2 may feel indebted to user U1. Therefore, it is possible for the agent device 200 to provide some kind of follow-up to user U2 who has developed a feeling of indebtedness. For example, it is possible for the agent device 200 to emit voice information such as "It's okay" to user U2 who has developed a feeling of indebtedness. However, when the agent device 200 emits voice information such as "It's okay," this voice information is transmitted to both users U1 and U2. In this case, it is also possible that user U1 does not want user U2 to hear his or her feelings of support for user U2 without permission. On the other hand, when user U2 hears that voice information, it is expected that he or she will feel awkward and will find it unbearable to be in that situation. In other words, it is expected that the output of that voice information will further increase user U2's sense of indebtedness toward user U1, and will worsen the user's experience of using the agent device 200. It is also expected that user U2 will feel even more uncomfortable.

In this embodiment, therefore, the agent device 200 executes a performance that alleviates the feelings of indebtedness felt by user U2, in a manner that is difficult for user U1 to perceive. In this way, by providing appropriate support to user U2 who has developed feelings of indebtedness, it is possible to increase the ease of use of the agent device 200. It is also possible to eliminate the unpleasant feelings of user U2. The performance executed by the agent device 200 will be described in detail with reference to Figures 5, 6, etc.

This embodiment can also be applied to cases where multiple conflicting instructions are not issued. For example, it is assumed that a situation will occur in which execution based on an instruction from a certain user (user A) takes a long time, causing execution based on an instruction from another user (user B) to wait. For example, assume that user U1 issues an instruction "Tell me a restaurant nearby that serves delicious yakiniku" at approximately the same time, and user U2 issues an instruction "Tell me the history of the Kamakura Shogunate." In this case, although instructions with conflicting contents are not issued from users U1 and U2, it is impossible to execute each of these instructions at approximately the same time. Therefore, even in such a case, as described above, the order of execution of appropriate instructions is selected from among the instructions issued by users U1 and U2 based on the selection criteria. In this case, it is assumed that, for example, the instruction from user U2 is selected first, and the output time of information related to "Tell me the history of the Kamakura Shogunate" will be long. In this case, user U1 will be made to wait until his/her instruction is executed. As a result, user U2, whose instructions were adopted first, is in a position to relatively gain, while user U1, whose instructions were not adopted first, is in a position to relatively lose. In this case, user U2, who is in a relatively advantageous position, is likely to develop a feeling of indebtedness toward user U1, who is in a relatively disadvantaged position. Therefore, even in such a case, as described above, it is possible to cause the agent device 200 to execute a performance that alleviates the feeling of indebtedness of user U2, in a manner that is difficult for user U1 to perceive.

The user's feelings of gain or loss regarding gain or loss can be estimated based on whether or not the instruction was adopted by the agent device 200. The user's feelings of gain or loss regarding gain or loss can be estimated based on at least one of the following: the user's biometric information (e.g., information related to heart rate, body temperature, surface temperature of the face, blood flow, and excitement level), the user's speech (e.g., a predetermined keyword such as "It's cold"), and the user's movement (e.g., the magnitude, speed, and facial expression of body movement). Similarly, the user's feelings of indebtedness and dissatisfaction can be estimated based on at least one of the user's biometric information, the user's speech, and the user's movement. For example, the user's feelings of indebtedness can be detected based on the user's facial expression, the user's movement, the user's voice, and the like. The user's facial expression and the user's movement can be acquired based on an image acquired by the camera 101. The user's facial expression also includes the user's gaze. The user's voice can be acquired based on voice information acquired by the voice input unit 103 (see FIG. 4). The method for estimating the user's feelings of gain/loss, debt, and dissatisfaction will be described in detail with reference to Figure 4 etc.

In this embodiment, the following description will be given using an example of users U1 and U2 who are occupants of the vehicle C1 and sit in the driver's seat. However, this embodiment can also be applied to a case where three or more occupants are in the vehicle C1.

[Example of agent device appearance]
3 is a front view showing a simplified example of the external configuration of the agent device 200. In this embodiment, an example is shown in which a face image is displayed on the display unit 210 (see FIG. 4) when the agent device 200 is on and in a normal state. The agent device 200 can be turned on in response to an on operation of the vehicle C1. The agent device 200 can also be turned on in response to a user operation on the agent device 200. The on/off operation of the vehicle C1 refers to the on or off operation of a start key related to starting or stopping the vehicle C1.

3A shows a perspective view of the agent device 200. FIG. 3B shows a front view of the agent device 200. The agent device 200 shown in FIG. 3B corresponds to the agent device 200 shown in FIG. 1. FIG. 3C shows a top view of the agent device 200. FIG. 3D shows a side view of the agent device 200. Also, FIG. 3E and FIG. 3F show an example of an external configuration when a specific performance for alleviating the debt feeling is executed as an operation mode of the agent device 200. Also, FIG. 3G shows an example of an external configuration when a performance of a confused expression is executed as an operation mode of the agent device 200. Also, FIG. 3H shows an example of an external configuration when a performance after the debt feeling is alleviated is executed as an operation mode of the agent device 200. The specific performance for alleviating the debt feeling is executed at the timing when it is determined that the debt feeling has occurred. Also, the performance after the debt feeling is alleviated is executed at the timing when it is determined that the debt feeling has been resolved after the execution of the specific performance.

The agent device 200 is a device composed of a roughly spherical main body 201, and a display 210 is provided on the surface of the main body 201. Various images, for example, eyes E1, E2, and mouth M1 are displayed on the display 210. The display 210 may be provided on the entire surface of the main body 201, or on a portion of the surface of the main body 201. Furthermore, when the display 210 is provided on a portion of the surface of the main body 201, the physical parts that make up the face (for example, ears and nose) may be provided on the portions other than the portion where the display 210 is provided.

In this embodiment, an example is shown in which the agent device 200 is installed on the dashboard 2 of the vehicle C1 in a direction in which the eyes E1 and E2 face the inside of the vehicle. For this reason, in the following description, one side of the agent device 200 including the part where the eyes E1 and E2 are displayed will be referred to as the front or face. In addition, the front of the agent device 200 will be referred to as the front side of the agent device 200, and the side opposite the front of the agent device 200 will be referred to as the rear side of the agent device 200. In addition, in the agent device 200, the right side in FIG. 3 will be referred to as the left side of the agent device 200, and the left side in FIG. 3 will be referred to as the right side of the agent device 200.

The main body 201 rotates in the forward and backward directions of the agent device 200, i.e., in the direction of arrow A1 (see Figs. 3A and 3D), with the lower part of the agent device 200 as the rotation center. Similarly, the main body 201 rotates in the left and right directions of the agent device 200, i.e., in the direction of arrow A2 (see Fig. 3B), with the lower part of the agent device 200 as the rotation center. Similarly, the main body 201 rotates in the direction of rotation around the rigid direction as the axis, i.e., in the direction of arrow A3 (see Fig. 3C), with the lower part of the agent device 200 as the rotation center. In this way, a drive unit 230 (see Fig. 4) for driving the main body 201 is provided at the lower part of the agent device 200. It is then possible to rotate the main body 201 of the agent device 200 left and right, rotate the main body 201 forward and backward, swing the main body 201 left and right, and swing the main body 201 forward and backward.

The display unit 210 can be configured as a display medium capable of displaying various images. As this display medium, various display media such as a light-emitting sheet, a liquid crystal display, an organic EL (Electro Luminescence), etc. can be used. For example, as shown in Figs. 3(A) to (D), it is possible to display normal eye parts E1, E2, and mouth part M1.

For example, as shown in FIG. 3(E), a sweat image SW1 can be displayed along with frowning eyes E1, E2. By vibrating the main body 201, it is also possible to express a state in which the agent device 200 is shaking. For example, as shown in FIG. 3(F), the portion B1 corresponding to the upper part of the face can be colored a predetermined color (e.g., blue) indicating an anxious expression, and a vertical line image V1 indicating an anxious expression can be displayed in part of that portion B1. By shaking the main body 201 in the left-right direction (the directions of arrows A4, A5) as shown in FIG. 3(E) and (F), it is possible to further emphasize an anxious attitude.

Also, for example, as shown in FIG. 3(G), a troubled-looking mouth M1 and eyes E1, E2 can be displayed. Note that the facial expression shown in FIG. 3(G) is an example of a bewildered facial expression. Also, for example, as shown in FIG. 3(H), a smiling mouth M1 and eyes E1, E2 can be displayed, and a cheek image RS1 with red cheeks can be displayed. Note that the facial expression shown in FIG. 3(H) is an example of a relieved facial expression.

In this way, by setting the color of at least a portion of the display unit 210 to a specific color or by deforming the eyes E1, E2 and mouth M1, it is possible to display the agent device 200 in such a manner that it appears anxious, confused, relieved, or the like. Note that the display mode shown here is merely an example, and each expression may be realized by other display modes. For example, an anxious expression may be realized by quickly moving the eyes E1, E2 from side to side, by looking around the eyes E1, E2, by quickly moving the main body unit 201 left and right, or by looking around the main body unit 201 left and right.

In this manner, in this embodiment, it is possible to change the agent device 200 by an operation mode that changes at least a portion of each part of the agent device 200, and a display mode that changes the display state of at least a portion of the surface of the agent device 200. Furthermore, an occupant of the vehicle C1 who sees the change can recognize that the agent device 200 is making some kind of movement. For this reason, in this embodiment, these changes in the agent device 200 are described as the operation mode of the agent device 200.

Note that these operating modes of the agent device 200 are merely examples, and other operating modes are also possible. For example, the agent device 200 can be slid within a specified range on the dashboard 2. For example, the agent device 200 can be slid forward, backward, left, and right. In addition, for example, it is also possible to create an effect in which the agent device 200 temporarily floats.

Note that FIG. 3 shows an example in which each facial expression is expressed two-dimensionally by changing the display mode of the display unit 210, but each facial expression may be expressed three-dimensionally by changing each physically installed component such as the eyes, nose, and mouth. In this case, eye movement may be expressed, for example, by moving the eye component or by displaying a black or white image on the surface of the eye component.

In addition, although this embodiment shows an example of an agent device 200 that only has a face, other body parts (e.g., hands, feet) may also be provided in the agent device 200. In this case, each of these parts (e.g., hands, feet) may be changed to realize each attitude (e.g., a relaxed attitude, a relieved attitude). For example, each attitude may be expressed by changing the hands to touch the head or face, or by flapping the feet or hands.

These shapes, display modes, and audio output modes of the agent device 200 are merely examples, and other shapes, display modes, and audio output modes are also possible. Also, while FIG. 3 shows an example of an agent device 200 in which an approximately spherical main body 201 functions as a face, the present invention is not limited to this. For example, the agent device 200 may be a device in which a flat main body (e.g., a planar display panel) functions as a face. Also, the agent device 200 may be a device in which the face and body are configured as an integrated housing. Also, the agent device 200 may be a device in which the face and body are configured as separate bodies.

In addition, in this embodiment, an example is shown in which a face image is displayed as an image that represents lifelikeness (animal display), but other images (for example, the entire body of an agent, a mechanical object, or a virtual face) may also be displayed as an image that represents lifelikeness.

[Example of configuration of information processing system]
FIG. 4 is a diagram showing an example of a system configuration of an information processing system 100 installed in the vehicle C1.

The information processing system 100 includes a camera 101, a position information acquisition sensor 102, a voice input unit 103, sensors 104, an information processing device 110, and an agent device 200. The information processing device 110 and the agent device 200 are connected by a communication method using wired communication or wireless communication. The information processing device 110 is connected to a network 20 by a communication method using wireless communication. The network 20 is a network such as a public line network or the Internet. The agent device 200 may also be connected to the network 20 by a communication method using wireless communication. Although FIG. 3 shows an example in which the information processing device 110 and the agent device 200 are configured as separate entities, the information processing device 110 and the agent device 200 may also be configured as an integrated device.

The camera 101 captures an image of a subject and generates an image (image data) based on the control of the information processing device 110, and outputs image information related to the generated image to the information processing device 110. The camera 101 is provided at least inside the vehicle C1, and captures an image of a subject inside the vehicle C1 to generate an image (image data). Note that FIG. 1 shows the camera 101 provided inside the vehicle C1. As described above, the camera 101 is composed of, for example, one or more camera devices or image sensors capable of capturing an image of a subject. For example, one camera 101 may be provided in front of the vehicle C1 to capture an image of a subject from in front of the vehicle C1 to generate an image (image data), and another camera 101 may be provided behind the vehicle C1 to capture an image of a subject behind the vehicle C1 to generate an image (image data).

The position information acquisition sensor 102 acquires position information on the position where the vehicle C1 is located, and outputs the acquired position information to the information processing device 110. For example, this can be realized by a GNSS receiver that acquires position information using the Global Navigation Satellite System (GNSS). The position information includes data on the position such as latitude, longitude, and altitude at the time of receiving the GNSS signal. The position information may also be acquired by other methods of acquiring position information. For example, the position information may be derived using information from access points or base stations in the vicinity. The position information may also be acquired using a beacon. For example, the state of the vehicle C1, for example, whether it is running, stopped, or reversing, can be determined based on the information acquired by the position information acquisition sensor 102.

In addition, for example, the road on which the vehicle C1 is traveling can be determined based on the position information acquired by the position information acquisition sensor 102.

The audio input unit 103 is provided inside the vehicle C1 and acquires sounds inside the vehicle C1 based on the control of the information processing device 110, and outputs sound information related to the acquired sounds to the information processing device 110. As the audio input unit 103, for example, one or more microphones or sound acquisition sensors can be used.

The sensors 104 are various sensors installed in the vehicle C1, and output detection information acquired by each sensor to the information processing device 110. The sensors are, for example, LiDAR (Light Detection And Ranging), RADAR (Radio Detection And Ranging), Sonar, a human presence sensor, a vehicle speed sensor, an acceleration sensor, a seating sensor, a seat belt sensor, a non-contact temperature sensor, etc. Note that these are only examples, and other sensors may be used. Also, only some of these sensors may be used.

The human presence sensor is a sensor that detects the presence, number, location, and status of people inside the vehicle C1. For example, a human presence sensor that uses infrared rays, ultrasound, visible light, an image sensor, or the like can be used. For example, if a person is seated in the driver's seat, the passenger seat, or the back seat, the presence of that person can be detected by the human presence sensor. Furthermore, by using both a human presence sensor and a seating sensor, it is possible to increase the accuracy of detecting the seating status of each seat. These also make it possible for the agent device 200 to identify the location of the person of interest.

The seating sensor (or seat sensor) is a sensor that detects whether or not an occupant is seated in each seat of the vehicle C1. The seat belt sensor is a sensor that detects whether or not an occupant is wearing a seat belt.

The non-contact temperature sensor is a sensor that detects the body temperature of an occupant of the vehicle C1 in a non-contact manner. For example, the temperature of the surface of the face of the user U1 can be measured by the non-contact temperature sensor. Publicly known sensors can be used for each of these sensors.

The information processing device 110 includes a control unit 120, a storage unit 130, and a communication unit 140. The communication unit 140 exchanges various information with other devices using wired or wireless communication based on the control of the control unit 120. For example, the communication unit 140 exchanges various information with an electronic device MC1 owned by a user U1, an electronic device MC2 owned by a user U2, and the like. In this case, the communication unit 140 may perform wireless communication with the electronic devices MC1 and MC2 via the network 20, or may perform direct wireless communication (for example, short-range wireless communication) with the electronic devices MC1 and MC2 without using the network 20. The electronic devices MC1 and MC2 are, for example, information processing devices such as smartphones, tablet terminals, and portable personal computers. For example, the electronic devices MC1 and MC2 function as output devices that display various images on a display unit and output sound from a sound output unit.

The control unit 120 controls each unit based on various programs stored in the storage unit 130. The control unit 120 is realized by a processing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). For example, the calculation speed can be increased by performing image processing using the GPU. The calculation speed can be further increased by performing parallel calculations using the GPU. The vehicle ECU (Electronic Control Unit) of the vehicle C1 may be used as the control unit 120, or a processing device different from the vehicle ECU may be provided as the control unit 120. The control unit 120 has a conversion function that converts characters into speech. This conversion function is realized by, for example, TTS (Text to Speech).

The control unit 120 executes a control process to control the operating state of the agent device 200 based on the information output from the camera 101, the position information acquisition sensor 102, the voice input unit 103, the sensors 104, the communication unit 140, etc. Specifically, the control unit 120 includes an instruction receiving unit 121, a profit/loss emotion determination unit 122, a debt emotion determination unit 123, a dissatisfaction emotion determination unit 124, and an agent control unit 125.

The instruction receiving unit 121 receives instruction information regarding one or more instructions issued by the occupant of the vehicle C1, and outputs the received instruction information to the profit/loss emotion determination unit 122, the debt emotion determination unit 123, the dissatisfaction emotion determination unit 124, and the agent control unit 125. For example, the voice (voice giving instructions) uttered by the occupant of the vehicle C1 is acquired by the voice input unit 103, and information regarding this voice is output to the instruction receiving unit 121 as instruction information. Here, as described above, when multiple instructions are issued by multiple occupants of the vehicle C1, it is assumed that these instructions are different and contradictory. In such a case, the instruction receiving unit 121 executes a selection process to select an instruction to be executed by the agent device 200 from the multiple contradictory and different instructions, and outputs selection information regarding the selected instruction to the profit/loss emotion determination unit 122, the debt emotion determination unit 123, the dissatisfaction emotion determination unit 124, and the agent control unit 125. Note that this selection process is the same as the selection process described above.

The profit/loss emotion determination unit 122 determines the profit/loss emotion of the occupant of the vehicle C1 based on the selection result of the selection process by the instruction receiving unit 121 when multiple conflicting instructions are issued by the occupant of the vehicle C1. The profit/loss emotion determination unit 122 then outputs the determination result to the debt emotion determination unit 123, the dissatisfaction emotion determination unit 124, and the agent control unit 125. The profit/loss emotion determination unit 122 may also determine the profit/loss emotion of the occupant of the vehicle C1 based on the selection result of the selection process by the instruction receiving unit 121, as well as each piece of information output from the camera 101, the position information acquisition sensor 102, the voice input unit 103, the sensors 104, the communication unit 140, etc. The method of determining the profit/loss emotion will be described later.

The debt emotion determination unit 123 determines the debt emotion of the occupant of the vehicle C1 when multiple conflicting instructions are issued by the occupant of the vehicle C1, and outputs the determination result to the agent control unit 125. Specifically, the debt emotion determination unit 123 determines the debt emotion of the occupant of the vehicle C1 based on the judgment result of the gain/loss emotion output from the gain/loss emotion determination unit 122 and each piece of information output from the camera 101, the position information acquisition sensor 102, the voice input unit 103, the sensors 104, the communication unit 140, etc. The method of determining the debt emotion will be described later.

The dissatisfied emotion determination unit 124 determines the dissatisfied emotion of the occupant of the vehicle C1 when multiple conflicting instructions are issued by the occupant of the vehicle C1, and outputs the determination result to the agent control unit 125. Specifically, the dissatisfied emotion determination unit 124 determines the dissatisfied emotion of the occupant of the vehicle C1 based on the determination result of the gain/loss emotion output from the gain/loss emotion determination unit 122 and each piece of information output from the camera 101, the position information acquisition sensor 102, the voice input unit 103, the sensors 104, the communication unit 140, etc. The method of determining dissatisfaction will be described later.

In this way, the gain/loss emotion determination unit 122, the liability emotion determination unit 123, and the dissatisfaction emotion determination unit 124 determine, among the occupants of the vehicle C1, users who have gained, users who have lost, users who have incurred liabilities, users who have incurred dissatisfaction, etc. In other words, the gain/loss emotion determination unit 122, the liability emotion determination unit 123, and the dissatisfaction emotion determination unit 124 function as detection units that detect specific emotions, unpleasant states, etc., related to the occupants of the vehicle C1.

The agent control unit 125 controls the operating state of the agent device 200. Specifically, the agent control unit 125 controls the operating state of the agent device 200 based on the instruction information output from the instruction receiving unit 121, the judgment results by the profit/loss emotion judgment unit 122, the debt emotion judgment unit 123, and the dissatisfaction emotion judgment unit 124, and each piece of information output from the camera 101, the position information acquisition sensor 102, the voice input unit 103, the sensors 104, the communication unit 140, and the like. For example, the agent control unit 125 displays each part (e.g., the eyes E1, E2, and the mouth M1) constituting the face of the mimicked agent on the display unit 210. The agent control unit 125 also displays information (notification information) to be notified to the occupant of the vehicle C1 on the display unit 210. The agent control unit 125 also outputs the voice of the mimicked agent, the voice corresponding to the information to be notified to the occupant of the vehicle C1, and the like from the sound output unit 220.

For example, the agent control unit 125 determines the notification information to be displayed on the display unit 210 and the notification information to be output from the sound output unit 220, and executes control to output the notification information. Note that notification information related to objects present inside the vehicle C1 is stored in the notification information DB 133. For example, in the case of notification information related to fastening a seat belt, audio information related to fastening a seat belt is output as the notification information. Note that an image showing a seat belt may be displayed on the display unit 210 as the notification information.

For example, when the agent control unit 125 detects a user who has developed a feeling of indebtedness (or a feeling of gain), it causes the agent device 200 to execute a performance that alleviates the feeling of indebtedness of the user in a manner that is difficult for other users to perceive. For example, as shown in FIG. 5 (D) and FIG. 6 (D1) to (D3), it causes the agent device 200 to execute a predetermined performance. As this performance, it is possible to execute a predetermined sound output, a predetermined behavior, etc.

Furthermore, for example, when the agent control unit 125 detects a user who is feeling dissatisfied, it causes the agent device 200 to execute a performance that alleviates the user's feeling of dissatisfaction in a manner that is difficult for other users to perceive. For example, as shown in Figures 5 (E) and 5 (F), it causes the agent device 200 to execute a predetermined performance. This performance can include a predetermined voice output, a predetermined behavior, and the like. In this way, the agent control unit 125 functions as a device that follows one of the occupants of the vehicle C1 in a manner that is difficult for the other users to perceive.

The storage unit 130 is a storage medium that stores various types of information. For example, the storage unit 130 stores various types of information (e.g., a control program, an agent information DB 131, an instruction information DB 132, an alarm information DB 133, a map information DB 134, a surrounding situation DB 135, and a user state DB 136) that is required for the control unit 120 to perform various processes. The storage unit 130 also stores various types of information acquired via the communication unit 140. For example, the storage unit 130 can be a read only memory (ROM), a random access memory (RAM), a static random access memory (SRAM), a hard disk drive (HDD), a solid state drive (SSD), or a combination of these.

The agent information DB 131 stores information required to realize various operations of the agent device 200. For example, facial image information (e.g., eyes E1, E2, sweat image SW1, vertical line image V1, cheek image RS1, mouth M1) displayed on the display unit 210 and audio information output from the sound output unit 220 are stored in the agent information DB 131. In addition, for example, operation information for operating the agent device 200 when performing an effect to alleviate feelings of indebtedness or an effect to alleviate feelings of dissatisfaction is stored in the agent information DB 131.

The instruction information DB132 stores information required to identify multiple conflicting instructions issued by the occupants of the vehicle C1. For example, information regarding combinations of multiple conflicting instructions (e.g., a combination of increasing the air conditioner temperature and decreasing the air conditioner temperature, and a selection of radio stations issued at the same time) is stored in the instruction information DB132.

Information DB133 is stored information required to output notification information regarding objects present inside or outside vehicle C1. For example, in the case of notification information regarding a half-open door, image information showing the door and audio information for notifying the driver of the half-open door are stored as notification information in the notification information DB133. In addition, in the case of notification information regarding fastening a seat belt, image information showing the seat belt and audio information for notifying the driver to fasten the seat belt are stored as notification information in the notification information DB133.

The map information DB 134 stores map information such as road information on roads required for route guidance of the vehicle C1. The map information includes road gradients, road intersections, the number of lanes on the road, road width information, road undulation information, and bad road conditions. The map information also includes road signs indicating speed limits and one-way streets, crosswalks, dividing lines, and other road markings. The map information may also include information on facilities such as road structures (e.g., traffic lights and telephone poles), buildings, and tourist information about the surrounding area. The agent information DB 131, instruction information DB 132, notification information DB 133, and map information DB 134 may be stored in the memory unit 130 of the vehicle C1 and used, or may be obtained from an external device via the network 20 and used.

The surrounding situation DB 135 stores information about the surrounding situation of the vehicle C1. This information is acquired and stored based on the information output from the camera 101, the position information acquisition sensor 102, the voice input unit 103, the sensors 104, the communication unit 140, etc.

The information stored in the surrounding conditions DB135 can be used for learning to improve the detection accuracy when detecting information about the surrounding conditions of the vehicle C1.

The user status DB 136 stores information about the status of the user U1. This information is acquired and stored based on the information output from the camera 101, the voice input unit 103, the sensors 104, the communication unit 140, etc.

The user status DB136 can be used for learning to improve the detection accuracy when detecting information about the status of user U1.

The agent device 200 is a robot device that performs various operations based on instructions from the information processing device 110. The agent device 200 includes a display unit 210, a sound output unit 220, and a drive unit 230. The display unit 210, the sound output unit 220, and the drive unit 230 are controlled based on a control unit (not shown) included in the agent device 200.

The display unit 210 is a display unit that displays various images based on instructions from the information processing device 110. As the display unit 210, for example, a display panel such as an organic EL (Electro Luminescence) panel or an LCD (Liquid Crystal Display) panel can be used. The display unit 210 may be configured as a touch panel that allows a user to input operations by touching or approaching the display surface with their finger, or may be configured as a separate user interface.

The sound output unit 220 outputs various sounds based on instructions from the information processing device 110. For example, one or more speakers can be used as the sound output unit 220. Note that the display unit 210 and the sound output unit 220 are examples of user interfaces, and some of them may be omitted, or other user interfaces may be used.

The driving unit 230 drives the main body 201 of the agent device 200 and each part of the agent device 200 based on instructions from the information processing device 110. For example, the driving unit 230 is a driving device that realizes a mechanism for rotating and vibrating the main body 201, as shown in Figures 3 (A) to (H). For example, the driving unit 230 is composed of a motor, servo motor, etc. that can move the agent device 200 and rotate the main body 201.

[Example of determining user's gain/loss feelings]
Next, a method for determining the user's feelings of gain or loss will be described. As described above, the user's feelings of gain or loss can be determined based on the result of the selection of the instruction from the user by the instruction receiving unit 121, the situation after the execution of the processing by the agent device 200, and the like. The situation after the execution of the processing by the agent device 200 is, for example, environmental information related to the surrounding environment that was the target of the processing, detected information related to each user that was the target of the processing, and the like. The detected information related to each user is, for example, the user's biological information (for example, heart rate, blood vessels), sounds emitted by the user, the user's appearance (for example, facial expression, movement), and the like.

[Example of judgment based on the selection result of the instruction by the agent device]
For example, when multiple users issue multiple conflicting instructions, the instruction receiving unit 121 selects an instruction to be executed by the agent device 200 from among the instructions. This selection process is the same as the selection process described above. Next, the gain/loss emotion determination unit 122 determines the user who issued the instruction selected by the instruction receiving unit 121 from among the instructions based on the instruction information and selection information output from the instruction receiving unit 121 as a user who has experienced a gain emotion. On the other hand, the gain/loss emotion determination unit 122 determines the users other than the user who has been determined to have experienced a gain emotion as a user who has experienced a loss emotion. For example, assume that user U1 issues an instruction to "lower the temperature of the air conditioner" and user U2 issues an instruction to "increase the temperature of the air conditioner." In this case, the agent device 200 selects an instruction to be executed based on the surrounding situation, each of the detection information related to users U1 and U2, and the like. For example, assume that the instruction "increase the temperature of the air conditioner" from user U2 is selected. In this case, the gain/loss emotion determination unit 122 determines user U2 as a user who has experienced a gain emotion, and determines user U1 as a user who has experienced a loss emotion.

The users and their positions aboard the vehicle C1 can be obtained based on the images of the interior of the vehicle C1 acquired by the camera 101. The users and their positions aboard the vehicle C1 can also be obtained based on the detection information detected by the human sensor and the seat sensor. The positions of the users who have issued instructions to the agent device 200 among the users aboard the vehicle C1 can be detected based on the direction from which the voice corresponding to the instruction was emitted. A known voice recognition technology can be used to detect the direction from which the voice was emitted. The positions of the users who have issued instructions to the agent device 200 among the users aboard the vehicle C1 can be obtained based on the images of the interior of the vehicle C1 acquired by the camera 101. For example, the positions can be detected based on facial expressions (particularly mouth movements) that emit the voice corresponding to the instruction. A known image recognition technology can be used to detect the voice based on facial expressions.

[Example of determination based on the situation after execution of processing by the agent device]
As described above, the agent device 200 executes appropriate processing based on the surrounding situation, each detected information related to the user, and the like. For example, based on each information from the camera 101, the voice input unit 103, and the sensors 104, the agent device 200 can detect the heat of the users U1 and U2 who are riding in the vehicle C1. For example, assume that a large amount of sweat of the user U1 is detected based on the face image of the user U1 acquired by the camera 101, and an increase in the body temperature of the user U1 is detected based on the non-contact temperature sensor of the sensors 104. In this case, the agent device 200 executes control to lower the temperature setting of the air conditioner. In addition, in this case, the gain/loss emotion determination unit 122 determines the user U1 who caused the processing executed by the agent device 200 as a user who has experienced a gain emotion. On the other hand, the gain/loss emotion determination unit 122 determines the user U2 other than the user U1 who has been determined to have experienced a gain emotion as a user who has experienced a loss emotion. In this way, it is possible to determine the user who caused the processing executed by the agent device 200 as the user who experienced a gain emotion, and determine other users as users who experienced a loss emotion.

[Example of determining user's debt sentiment]
Next, a method for determining a user's debt feeling will be described. In this embodiment, the debt feeling of a user is determined for a user who has a gain feeling. Specifically, the debt feeling of a user can be determined based on the user's biometric information (e.g., heart rate, blood vessels), the sound emitted by the user, and the user's appearance (e.g., facial expression, movement). The debt feeling of users other than the user who has a gain feeling may also be determined, and the determination result may be used.

As described above, when there are multiple users, some users may feel indebted to other users. In other words, indebtedness is an emotion that occurs in some users when there are multiple users. For example, when a certain phenomenon occurs for multiple users, it is assumed that some users will be in a relatively advantageous position and other users will be in a relatively disadvantaged position. For example, when a phenomenon occurs in which a user U1 who is sensitive to heat and a user U2 who is sensitive to cold raise the temperature setting of an air conditioner, it is assumed that user U2 will be in a relatively advantageous position and user U1 will be in a relatively disadvantaged position. In such a case, user U2 who is in a relatively advantageous position often feels indebted, such as feeling sorry or sorry, toward user U1 who is in a relatively disadvantaged position. Therefore, in this embodiment, appropriate follow-up is provided to the user who has felt indebted to other users to alleviate the indebtedness.

For example, when the user U2 feels a sense of indebtedness, it is assumed that the face of the user U2 will have a sad or apologetic expression. Therefore, by analyzing an image including the face of the user U2, the change in the expression of the user U2 can be acquired, and the indebtedness of the user U2 can be calculated based on the degree of the change. For example, when the expression of the user U2 changes suddenly and the expression after the change becomes a predetermined expression (for example, a sad expression, apologetic expression), it is possible to determine that the degree of the indebtedness of the user U2 is high. Note that the degree of the indebtedness of the user U2 is a numerical representation of the indebtedness of the user U2, and it is possible to determine that the indebtedness of the user U2 is high as the numerical value increases. It is also possible to determine that the degree of the indebtedness of the user U2 is high when the face of the user U2 has a predetermined expression (for example, a sad expression, apologetic expression). Note that the image including the face of the user U2 can be acquired by the camera 101 (see FIG. 4). It is also possible to use a known facial expression recognition technology (for example, a facial expression determination program) to determine the facial expression. The degree of indebtedness may also be referred to as the degree of certainty of indebtedness.

Note that the indebtedness of user U2 may be determined using other information related to the face. For example, when user U2 feels indebted, it is expected that user U2 may break out in a cold sweat on his/her face. Therefore, based on an image (including user U2's face) acquired by camera 101, it is possible to detect whether user U2 has broken out in a cold sweat on his/her face, and to determine that user U2 has broken out in a cold sweat on his/her face.

Also, for example, when user U2 feels a sense of indebtedness, it is assumed that user U2 will often turn his/her gaze towards user U1. For example, it is assumed that user U2 will frequently turn his/her gaze towards user U1 with a sad expression. Therefore, based on the image (including user U2's face) acquired by camera 101, etc., it is possible to detect the number of times user U2 turns his/her gaze towards user U1, and to determine user U2's sense of indebtedness based on the number of times. For example, if the number of times user U2 turns his/her gaze towards user U1 is equal to or greater than a threshold, it is possible to determine that user U2's sense of indebtedness is high. In other words, it is possible to determine whether user U2's sense of indebtedness is high or not based on the frequency with which user U2 turns his/her gaze towards user U1 as the movement of user U2. Also, it is possible to determine whether user U2's sense of indebtedness is high or not based on a change in user U2's body temperature.

In addition, when a feeling of indebtedness arises in user U2, it is assumed that user U2 will utter a specific keyword (e.g., I'm sorry, I actually wanted to turn down the temperature, it's hot, isn't it?) or will speak in a lower tone of voice. The specific keyword is, for example, a phrase that expresses the feeling of user U2 being sorry or feeling sorry toward user U1. The tone of voice means the pitch, tempo, volume, etc. of the voice. For example, when a feeling of indebtedness arises in user U2, it is assumed that user U2 will utter "I'm sorry" in a lower voice. Therefore, by analyzing the voice uttered by user U2, the specific keyword uttered by user U2 (e.g., I'm sorry) and the tone of voice uttered by user U2 are obtained. Then, when user U2 utters a specific keyword (e.g., I'm sorry), or when the tone of voice uttered by user U2 is below a threshold, it is possible to determine that user U2 has a high feeling of indebtedness. The voice uttered by user U2 can be acquired by the voice input unit 103 (see FIG. 4). A known sound recognition technology (e.g., a sound determination program) can be used to determine the voice. A known estimation technology can also be used to estimate emotions by combining facial expressions and voice. For example, it is possible to estimate a debt emotion by combining facial expressions and voice, and to quantify the debt emotion (degree of debt emotion). Other numerical values related to the debt emotion of user U2 that can be calculated based on at least one of the biometric information of user U2, the content of the speech of user U2, and the actions of user U2 may also be used.

Note that two or more of the above-mentioned criteria for determining indebtedness may be used to comprehensively determine the indebtedness feeling. This can improve the accuracy of calculating the degree of indebtedness feeling.

[Example of Determining User's Dissatisfaction]
Next, a method for determining the dissatisfaction of a user will be described. In this embodiment, the determination of the dissatisfaction of a user is performed for a user who has experienced a loss. Specifically, the dissatisfaction of a user can be determined based on the user's biometric information (e.g., heart rate, blood vessels), the sound emitted by the user, and the user's appearance (e.g., facial expression, movement). The dissatisfaction of users other than the user who has experienced a loss may also be determined, and the determination result may be used.

As described above, when there are multiple users, some users may feel indebted to other users. On the other hand, users other than the user who feels indebted may feel dissatisfied. In other words, dissatisfaction is a feeling that occurs in some users when there are multiple users. For example, when a phenomenon occurs in which the temperature setting of an air conditioner is increased for user U1 who is sensitive to heat and user U2 who is sensitive to cold, it is assumed that user U2 will be in a relatively advantageous position and user U1 will be in a relatively disadvantaged position. In such a case, user U1 who is in a relatively disadvantaged position often feels dissatisfied with user U2 who is in a relatively advantageous position. Therefore, in this embodiment, appropriate follow-up is provided to users who feel dissatisfied with other users to alleviate their dissatisfaction.

The user's dissatisfaction can be determined based on various information related to the user's face (e.g., facial expression, mouth movement, gaze), the user's voice, etc., in the same way as the user's debt. The degree of the user's dissatisfaction can be calculated based on various information related to the user's face, the user's voice, etc., in the same way as the degree of the user's debt. The degree of the user's dissatisfaction is a numerical representation of the user's dissatisfaction, and it is possible to determine that the user's dissatisfaction is high as the numerical value increases. However, the content of the user's debt is different from the content of the facial expression, speech, etc. For example, when the user U1 becomes dissatisfied, it is assumed that the user U1's face will have an angry expression, or that the user U1 will speak in an angry voice, a strong tone, or a loud voice. For example, when the user U1 becomes dissatisfied, it is assumed that the user U1's movements will become stronger and larger. Therefore, in the process of determining the user's dissatisfaction, it is possible to make a determination based on these factors.

In addition, two or more of the conditions for determining dissatisfaction may be used to comprehensively determine dissatisfaction. This can improve the accuracy of calculating the degree of dissatisfaction.

[Example of changing each aspect of the agent device according to the user's emotions]
Next, a transition example in which the aspects of the agent device 200 are changed in response to the emotions of the users U1 and U2 will be described.

[Example of transition when changing each state of the agent device according to the user's emotion after executing a process according to the user's instruction]
5 and 6 are diagrams showing examples of transitions when each aspect of the agent device 200 is changed according to the emotions of users U1 and U2 after processing according to instructions from users U1 and U2 is executed. Specifically, in a case where conflicting instructions are given by users U1 and U2, a transition example is shown in which each aspect of the agent device 200 is changed according to the emotions of users U1 and U2 after processing according to instructions from user U2 is executed.

Figure 5 (A) shows an example of a case where two users U1 and U2 in a vehicle C1 give different, conflicting instructions to the agent device 200. Note that the example shown in Figure 5 (A) is the same as that in Figure 2. That is, an example is shown of a case where user U2, who is shivering from the cold, utters voice S12 saying "Turn up the air conditioner temperature," and user U1, who is sweating from the heat, utters voice S11 saying "Turn down the air conditioner temperature." In this way, when users U1 and U2 give different, conflicting instructions to the agent device 200, as described above, the agent device 200 selects and executes one of those instructions.

Figure 5 (B) shows an example of the behavior of the agent device 200 when a selected instruction is executed after each instruction shown in Figure 5 (A) has been given. Figure 5 (B) shows an example of a case where an instruction from user U2 is selected and a temperature adjustment process to increase the temperature setting of the air conditioner is executed. In this case, the agent device 200 emits voice information S13 saying "Increasing the air conditioner temperature." This allows both users U1 and U2 to understand that the instruction from user U2 has been executed by the agent device 200.

As shown in FIG. 5(B), user U2 for whom the instruction was selected will be able to alleviate the coldness by having the temperature setting of the air conditioner in the passenger compartment of vehicle C1 increased, which is expected to cause positive emotions (e.g., joy, trust). On the other hand, user U1 for whom the instruction was not selected will feel hotter as the temperature setting of the air conditioner in the passenger compartment of vehicle C1 is increased, which is expected to cause negative emotions (e.g., confusion, disappointment, sadness, emptiness, anger).

In this way, when conflicting and different instructions are issued by multiple users, at least one instruction will be adopted. For this reason, it is expected that the user whose instructions are adopted will experience positive emotions, and the user whose instructions are not adopted will experience negative emotions. However, as described above, it is expected that in addition to positive emotions, the user whose instructions are adopted will also experience feelings of indebtedness, gain, etc. toward the user whose instructions are not adopted. Therefore, in this embodiment, when conflicting and different instructions are issued by multiple users, at least one of those instructions is selected and executed, and then sensing is performed for each user. Then, each user's feelings of gain/loss, indebtedness, dissatisfaction, etc. are determined.

Figure 5 (C) shows a schematic diagram of the emotions of users U1 and U2 after the instruction of user U2 is adopted as shown in Figure 5 (B). Specifically, it is assumed that user U1, whose instruction was not adopted, will have the feeling AF1 of "it's too hot" as the temperature setting of the air conditioner is raised and the temperature inside the vehicle C1 rises. On the other hand, user U2, whose instruction was adopted, will feel comfortable as the temperature setting of the air conditioner is raised and the temperature inside the vehicle C1 rises, but will have the feeling AF2 of "I'm sorry, Mr. B, it's hot, isn't it?" in response to user U1 who looks hot. In this case, the agent device 200 executes each aspect based on the degree of indebtedness, the degree of dissatisfaction, etc. of users U1 and U2.

[Example of performing a performance to reduce a target user's sense of indebtedness in a manner that is difficult for other users to perceive]
FIG. 5D shows an example of the manner of the agent device 200 when the degree of the indebtedness of the user U2 is equal to or greater than the first threshold. In this case, since the degree of the indebtedness of the user U2 to the user U1 is high, the agent device 200 follows the user U2. However, if the agent device 200 follows the user U2 in a manner that is perceived by the user U1, the user U1 may not want the user U1 to freely convey his/her feelings to the user U2. On the other hand, if the agent device 200 follows the user U2 in a manner that is perceived by the user U1, the user U2 may feel uncomfortable. Therefore, in this embodiment, if the degree of the indebtedness of the user U2 to the user U1 is high, the agent device 200 follows the user U2 in a manner that is difficult for the user U1 to perceive. The manner that is difficult for the user U1 to perceive will be described later. The first threshold is a threshold for determining that the user's indebtedness is high, and can be set appropriately based on experimental data, etc.

As shown in FIG. 5(D), the agent device 200, with a facial expression designed to ease the indebtedness of user U2 toward user U1, faces the user U2 and emits voice information S14 saying, "It's cold. Can't be helped." The face of the agent device 200 can be turned toward the user U2 by the rotational movement of the drive unit 230. However, it is possible to turn the face of the agent device 200 toward the user U2 by changing the display content of the display unit 210 of the agent device 200. For example, it is possible to turn the face of the agent device 200 toward the user U2 by moving the display positions of the eyes and mouth on the spherical face of the agent device 200 toward the user U2.

Facial expressions that alleviate the indebtedness of user U2 include, for example, facial expressions that sympathize with user U2, facial expressions that cooperate with user U2, facial expressions that sympathize with user U2, and facial expressions that apologise to user U2. The agent device 200 may also be caused to perform a behaviour that alleviates the indebtedness of user U2 towards user U1. For example, as shown in FIG. 5(D), it is possible to realise a behaviour that sympathises with and sympathises with user U2 who is shivering from the cold by vibrating the agent device 200. It is also possible to realise a behaviour that sympathises with and sympathises with user U2 who is shivering from the cold by using hands and feet, etc. This allows appropriate follow-up to be performed on user U2 who has developed an indebtedness towards user U1, and it becomes possible to resolve the indebtedness of user U2. For example, it is possible to suppress the feeling of apologising that has developed in user U2.

In FIG. 5(D), an example is shown in which the agent device 200 performs a predetermined behavior with the face of the agent device 200 facing the direction of the user U2, in a behavior that is difficult for the user U1 to perceive. However, the agent device 200 may perform other behaviors that are difficult for the user U1 to perceive. For example, as a behavior that is difficult for the user U1 to perceive, it is possible to execute at least one of a first behavior in which the agent device 200 performs a performance in a manner that is difficult for the user U1 to perceive, and a second behavior in which the agent device 200 performs a performance at a timing that is difficult for the user U1 to perceive.

Note that a mode that is difficult for other users to perceive can also be referred to as, for example, a mode that is not noticed by other users, a mode that is difficult for other users to recognize, a mode that is difficult for other users to see, etc.

[Example of a method that is difficult for users to notice]
The first aspect (a method that is difficult for the user U1 to perceive) is, for example, an aspect in which the agent device 200 is made to behave in a predetermined manner while being pointed in the direction of the user U2 (aspect 1A), an aspect in which a sound that can only be heard by the user U2 is output from the agent device 200 (aspect 1B), an aspect in which predetermined information is output from the electronic device MC2 possessed by the user U2 based on the control of the information processing device 110 (or the agent device 200) (aspect 1C), an aspect in which predetermined information is displayed from another display area based on the control of the information processing device 110 (or the agent device 200) (aspect 1D), etc.

In the first A aspect, for example, as shown in FIG. 5(D), it is possible to cause the agent device 200 to perform a predetermined behavior (trembling movement) while facing the face of the agent device 200 toward the user U2.

In addition, in the first A aspect, for example, it is possible to cause the agent device 200 to perform a predetermined mouth movement while the face of the agent device 200 is facing the direction of the user U2. As this mouth movement, for example, it is possible to execute a mouth movement (so-called lip-syncing) that produces a predetermined sound (for example, "It can't be helped").

In addition, instead of making the agent device 200 make a predetermined mouth movement, predetermined information (e.g., text information "It's cold. Can't be helped") may be displayed on the display unit 210 of the agent device 200. An example of this is shown in FIG. 6 (D1). As shown in FIG. 6 (D1), with the display unit 210 of the agent device 200 facing the direction of the user U2, it is possible to display predetermined information TX1 (e.g., the text "Can't be helped") on the display unit 210 to set it as the first mode. Note that other information (e.g., an image that sympathizes with the cold) may also be displayed on the display unit 210 to set it as the first mode.

In the first B mode, as shown in FIG. 5(D), with the face of the agent device 200 facing the direction of the user U2, it is possible to output a sound that can only be heard by the user U2 from the sound output unit 220 of the agent device 200. If at least one of the speakers constituting the sound output unit 220 of the agent device 200 is located near the ears of the user U2, it is possible to make the sound only audible to the user U2 by emitting a small sound from the speaker located near the ears of the user U2. If at least one of the speakers constituting the sound output unit 220 of the agent device 200 is a directional speaker, it is possible to make the sound only audible to the user U2 by emitting a small sound from the directional speaker toward the user U2.

In this way, in the first A and first B aspects, the agent device 200 is caused to express a behavior that alleviates the feeling of indebtedness, to express a facial expression that alleviates the feeling of indebtedness, or to output a sound that alleviates the feeling of indebtedness from the agent device 200. Note that the behavior that alleviates the feeling of indebtedness can be, for example, at least one of the following: a movement of looking into the eyes of the user U2, a movement of nodding while looking at the user U2, a predetermined mouth movement while looking at the user U2, and displaying a predetermined image or predetermined text on the surface of the agent device 200.

In the first C aspect, it is possible to output specific information from the electronic device MC2 possessed by the user U2 based on the control of the information processing device 110 (or the agent device 200). An example of this is shown in FIG. 6 (D2). For example, the specific information (e.g., the text "It's cold. Can't be helped.") may be displayed on the display unit 30 of the electronic device MC2, or the specific information (e.g., the text "It's cold. Can't be helped.") may be output in a low volume from the audio output unit of the electronic device MC2. In addition, for example, other information (e.g., a sympathetic smiling face of the agent device 200) may be displayed on the display unit 30 of the electronic device MC2.

Here, an identification method for identifying the electronic device MC1 owned by the user U1 and the electronic device MC2 owned by the user U2 will be described. For example, it is possible that a user who gets into the vehicle C1 registers the electronic device he or she owns in advance in the vehicle C1. For example, user identification information (e.g., face identification information, voice identification information) that can identify the user and device identification information of the electronic device are associated and registered in the information processing device 110, and it is possible to determine the electronic device owned by the user of each seat based on each of these identification information. For example, it is possible to recognize the user sitting in each seat based on the user's face image and user identification information (e.g., face identification information) included in the image acquired by the camera 101. In this case, it is possible to use a known personal recognition technology. For example, it is possible to identify that the face of the user sitting in the passenger seat is the face of the user U2. In this case, it is possible to recognize the electronic device M2 owned by the user U2 sitting in the passenger seat. As a result, the agent control unit 125 can grasp that the user U2 who owns the electronic device MC2 that outputs predetermined information is sitting in the passenger seat. The agent control unit 125 can then cause the electronic device MC2 to output specific information for the user U2.

For example, the information processing device 110 can detect radio waves emitted from electronic devices carried by a user in the vehicle C1. Therefore, for example, only the device identification information of the electronic devices may be registered in advance in the information processing device 110, and the location of the registered electronic devices may be determined using the radio wave strength of each electronic device. For example, multiple antennas may be set in the passenger compartment of the vehicle C1, and the location of each electronic device may be estimated based on the radio wave strength of each electronic device received by each of these antennas. Note that known location estimation techniques may be used for this location estimation. In this case, it is possible to recognize that the electronic device M2 is located in the passenger seat. This allows the agent control unit 125 to recognize that the user U2 seated in the passenger seat is carrying an electronic device MC2 that outputs specified information.

For example, when a user gets into the vehicle C1, the electronic device carried by the user may be detected. For example, assume that a plurality of users get into the vehicle C1 at different times. In this case, it is possible to sequentially detect radio waves emitted from the electronic devices carried by the users who get into the vehicle C1. In addition, the seat in which the user who gets into the vehicle C1 sits can be detected based on information from the sensors 104 (e.g., seating sensor, seat belt sensor) and the camera 101. In such a case, the control unit 120 can associate the newly detected electronic device with the position of the seat where the user was detected to be seated at the detection timing (or the timing immediately after the detection) and record it in the user state DB 136 of the storage unit 130. For example, assume that the user U2 first gets into the vehicle C1 on the passenger seat side, and then the user U1 gets into the vehicle C1 on the driver's seat side. In this case, the control unit 120 associates the first detected electronic device MC2 with the position of the passenger seat where the user was detected to be seated at the detection timing and records it in the user state DB 136. Then, the control unit 120 associates the next detected electronic device MC1 with the driver's seat position where the user was detected to be seated at that detection timing, and records them in the user state DB 136. This allows the agent control unit 125 to know that the user U2 seated in the passenger seat is carrying an electronic device MC2 that outputs predetermined information.

It is also assumed that multiple users will get into the vehicle C1 at approximately the same time. In this case, it is also assumed that the seating positions and the electronic devices cannot be associated with each other by the above-mentioned method. Therefore, when multiple users get into the vehicle C1 at approximately the same time, the control unit 120 sequentially detects radio waves emitted from each electronic device owned by each user who gets into the vehicle C1. Then, the control unit 120 displays a selection button (e.g., a driver's seat button, a passenger seat button) for selecting a seat on each electronic device from which the radio waves are detected. For example, a user U2 sitting in the passenger seat selects the passenger seat button from among the selection buttons displayed on the display unit 30 of the electronic device MC2. When seat information related to this selected seat is transmitted from the electronic device MC2 to the information processing device 110, the control unit 120 records the passenger seat position corresponding to the seat information and the electronic device MC2 that transmitted the seat information in the user state DB 136 in association with each other. It is also possible to record other seats and electronic devices in the same manner. This allows the agent control unit 125 to grasp that the user U2 sitting in the passenger seat has an electronic device MC2 that outputs predetermined information. Note that these are just examples, and other methods may be used.

In addition, in the first D aspect, it is possible to display specific information on the HUD (Head Up Display) of the vehicle C1 based on the control of the information processing device 110 (or the agent device 200). An example of this is shown in FIG. 6 (D3). In this case, a HUD display device for realizing the HUD is provided on the upper part of the dashboard 2, near the boundary with the windshield 4. Note that in FIG. 6 (D3), the outline of the head of the user U2 is simplified and shown by a dotted line for ease of explanation.

The HUD display device is a display device, such as a projector or optical system, that projects light onto the display area 4a of the windshield 4 and uses the reflected light to realize a HUD display that shows a virtual image to users U1 and U2. That is, the light projected from the HUD display device onto the display area 4a of the windshield 4 is reflected by the windshield 4, and the reflected light heads toward the eyes of users U1 and U2. The reflected light that is projected onto the display area 4a and enters the eyes of users U1 and U2 is displayed together with actual objects seen through the windshield 4, superimposed on those objects. In this way, the HUD display device realizes a HUD display by displaying a virtual image using the windshield 4.

For example, the HUD display device displays predetermined information TX2 (e.g., the text "It's cold. Can't be helped.") in the display area 4a based on the control of the information processing device 110 (see FIG. 4). The windshield 4 also functions as a display medium for the HUD of the vehicle C1. For example, other information (e.g., a sympathetic smiling face of the agent device 200) may also be displayed in the display area 4a.

[Example of timing that is difficult for users to perceive]
The second aspect (timing that is difficult for user U1 to perceive) is, for example, a timing (first timing) that is set based on the degree to which user U1 is concentrating on something, and a timing (second timing) when user U1 moves away from the location where the agent device 200 is located.

The degree to which the user U1 is concentrating on something can be determined, for example, based on the driving scene in the vehicle C1. For example, it can be determined based on the shape of the road on which the vehicle C1 is traveling. For example, if the road on which the vehicle C1 is traveling is a mountain road and has a series of curves, it is determined that the degree to which the user U1 is concentrating on driving is high. Also, for example, if the road on which the vehicle C1 is traveling is a narrow road, or if the vehicle C1 is traveling at a high speed, it is determined that the degree to which the user U1 is concentrating on driving is high. The shape of the road on which the vehicle C1 is traveling can be obtained, for example, based on an image of the surroundings of the vehicle C1 (for example, an image in front of the vehicle C1) acquired by the camera 101. Also, it can be obtained based on the current location of the vehicle C1 acquired by the position information acquisition sensor 102 and the map information stored in the map information DB 134. Also, the vehicle speed of the vehicle C1 can be obtained based on a vehicle speed sensor.

In addition, for example, the driving scene can be determined based on the eye movement of the user U1 who is driving the vehicle C1. For example, if the eye movement of the user U1 is frequently moving left and right, it is determined that the user U1 is highly concentrated on driving. Note that the eye movement of the user U1 can be obtained based on, for example, a facial image of the user U1 acquired by the camera 101.

In this way, it is possible to determine the degree to which user U1 is concentrating on something based on at least one of the sensing of user U1 and the driving scene. Note that the degree to which user U1 is concentrating on something may also be determined based on other information. Then, when it is determined that user U1 is concentrating on something to a high degree, it is possible to determine that this is a timing that is difficult for user U1 to perceive.

The timing (second timing) at which the user U1 leaves the location where the agent device 200 is located can be determined, for example, based on whether the user U1 has exited the vehicle C1. For example, if the user U1 exits the vehicle C1 to take a toilet break, etc., it can be determined that the user U1 has left the location where the agent device 200 is located. In this case, it can be determined that this is a timing that is difficult for the user U1 to perceive. Note that whether the user U1 has exited the location where the agent device 200 is located can be determined based on the detection results by detecting whether the user U1 has exited the vehicle C1 using various sensors.

For example, if a relatively long time has passed since it was determined that user U2 had developed feelings of indebtedness, the effect of the performance for user U2 (performance to alleviate feelings of indebtedness) may be weakened. Therefore, based on the amount of time that has passed since it was determined that user U2 had developed feelings of indebtedness, it may be determined whether or not to execute a performance for user U2 (performance to alleviate feelings of indebtedness) at a timing that is difficult for user U1 to perceive.

For example, if the road on which the vehicle C1 is traveling is a series of simple roads such as an expressway, it is possible to set a time of about 30 minutes as the time for performing a performance (performance to alleviate feelings of indebtedness) for user U2 at a timing that is difficult for user U1 to perceive. In this case, if the set time (about 30 minutes) has elapsed since it was determined that user U2 has developed feelings of indebtedness, a performance (performance to alleviate feelings of indebtedness) for user U2 is performed in another mode (first mode). For example, a priority may be set for each of the first A mode, the first B mode, the first C mode, and the first D mode, and an embodiment may be selected from the first A mode, the first B mode, the first C mode, and the first D mode based on this priority. For example, it is possible to set a high priority for a mode that can be followed up quickly.

In addition, a time limit for performing a performance (performance to alleviate feelings of indebtedness) for user U2 may be set at a timing that is difficult for user U1 to perceive based on biometric information about users U1 and U2. For example, the time limit may be set based on the facial expressions of users U1 and U2. For example, if a gloomy facial expression of user U2 and an angry facial expression of user U1 are continuously detected after it is determined that a feeling of indebtedness has arisen in user U2, the time limit may be set to a relatively long time. On the other hand, for example, if a smile of user U2 is detected after it is determined that a feeling of indebtedness has arisen in user U2, the performance (performance to alleviate feelings of indebtedness) for user U2 may be stopped. In this way, the timing of following up with user U2 may be changed based on biometric information about users U1 and U2.

[Example of executing a performance that alleviates dissatisfaction of a target user in a manner that is difficult for other users to perceive]
FIG. 5E shows an example of the manner of the agent device 200 when the degree of the indebtedness of the user U2 is equal to or greater than the first threshold and the degree of the dissatisfaction of the user U1 is equal to or greater than the second threshold. In this case, since the dissatisfaction of the user U1 with respect to the user U2 is high, the agent device 200 follows up on the dissatisfaction of the user U1. However, if the dissatisfaction of the user U1 is followed up on in a manner that is perceived by the user U2, it is assumed that the user U2 does not want his/her feelings to be conveyed to the user U1 without permission. On the other hand, if the dissatisfaction of the user U1 is followed up on in a manner that is perceived by the user U2, it is assumed that the anger of the user U1 increases. Therefore, in this embodiment, if the degree of the dissatisfaction of the user U1 with respect to the user U2 is high, the agent device 200 follows up on the dissatisfaction of the user U1 in a manner that is difficult for the user U2 to perceive. The manner that is difficult for the user U2 to perceive is the same as the case of following up on the indebtedness described above. The second threshold is a threshold for determining that the user's dissatisfaction is high, and can be appropriately set based on experimental data, etc.

As shown in FIG. 5(E), the agent device 200, with a facial expression designed to ease the dissatisfaction of user U1 toward user U2, turns toward user U1 and emits voice information S15 saying, "I'm sorry we didn't adopt your opinion." The face of the agent device 200 can be turned toward user U1 by the rotational movement of the drive unit 230. Also, in the same way as in the case of following up on the debt feeling described above, the face of the agent device 200 can be turned toward user U2 by changing the display content of the display unit 210 of the agent device 200.

Facial expressions that ease the dissatisfaction of user U1 include, for example, facial expressions that sympathize with user U1, expressions that cooperate with user U1, and expressions that sympathize with user U1. The agent device 200 may also be caused to execute a behavior that eases the dissatisfaction of user U1 with respect to user U2. For example, as shown in FIG. 5(E), by displaying sweat on the surface of the agent device 200, it is possible to realize a behavior of sympathizing with, sympathizing with, and apologizing to user U1 with an impatient expression. Also, it is possible to realize a behavior of sympathizing with, sympathizing with, and apologizing to dissatisfied user U1 by using hands and feet, etc. The agent device 200 may also be caused to execute a behavior of silently bowing together with an apologetic facial expression. This allows appropriate follow-up to be performed on user U1 who has developed dissatisfaction with user U2, and makes it possible to resolve the dissatisfaction of user U1. For example, it is possible to apologize to user U1 for causing dissatisfaction in a manner that is difficult for user U2 to perceive.

The effects intended to alleviate user U1's feelings of dissatisfaction can be appropriately set by modifying part of the effects intended to alleviate user U2's feelings of indebtedness.

Figure 5 (F) shows an example of the behavior of the agent device 200 when the degree of indebtedness of user U2 is less than the first threshold, but the degree of dissatisfaction of user U1 is equal to or greater than the second threshold. In this case, as in the example shown in Figure 5 (E), user U1's dissatisfaction with user U2 is high, so follow-up is performed to address user U1's dissatisfaction. However, in this case, because the degree of indebtedness of user U2 is less than the first threshold, no performance is performed to alleviate user U2's indebtedness.

[Example of operation of information processing device]
7 is a flowchart showing an example of a control process in the information processing device 110. This control process is executed by the control unit 120 based on a program stored in the storage unit 130. This control process is executed at the timing when an occupant of the vehicle C1 issues an instruction to the agent device 200. This control process will be described with reference to FIGS. 1 to 6 as appropriate.

In step S501, the instruction receiving unit 121 receives an instruction issued to the agent device 200 by an occupant of the vehicle C1. The instruction receiving unit 121 can receive an instruction for the agent device 200 based on, for example, the voice of the occupant of the vehicle C1 acquired by the voice input unit 103, each action of the occupant of the vehicle C1 acquired by the camera 101 (e.g., facial expressions, hand movements), and the operation contents of the occupant of the vehicle C1 acquired by the sensors 104. Note that FIG. 7 shows an example in which two users U1 and U2 are riding in the vehicle C1.

In step S502, the instruction receiving unit 121 determines whether multiple different, conflicting instructions have been issued from multiple users. Specifically, if the instructions acquired in step S501 are from multiple users and each of these instructions is a conflicting, different instruction, the instruction receiving unit 121 determines that multiple different, conflicting instructions have been issued from multiple users. For example, if user U1 issues an instruction to "play radio station A" and user U2 issues an instruction to "play radio station B" at approximately the same time, it is determined that multiple different, conflicting instructions have been issued from multiple users. Also, for example, if user U1 issues an instruction to "lower the temperature of the air conditioner" and user U2 issues an instruction to "raise the temperature of the air conditioner" at approximately the same time, it is determined that multiple different, conflicting instructions have been issued from multiple users. On the other hand, for example, if user U1 issues an instruction to "turn down the air conditioner temperature" and user U2 issues an instruction to "play radio station B" at approximately the same time, it is determined that multiple conflicting and different instructions have not been issued by multiple users. If multiple conflicting and different instructions have been issued by multiple users, the process proceeds to step S504. On the other hand, if multiple conflicting and different instructions have not been issued by multiple users, the process proceeds to step S503.

In step S503, the agent control unit 125 executes processing based on instructions from the user. Specifically, the agent control unit 125 executes processing according to the instructions acquired in step S501. For example, if user U1 issues an instruction to "lower the air conditioner temperature" and user U2 issues an instruction to "play radio station B" at approximately the same time, the agent control unit 125 executes processing according to each of these instructions consecutively or simultaneously.

In step S504, the instruction receiving unit 121 selects an instruction to be processed from among the multiple instructions acquired in step S501, and the agent control unit 125 executes processing based on the selected instruction. This selection process is executed based on the biometric information of each user, the speech content of each user, the actions of each user, the environment around each user, and the like, similar to the selection process described above. In this case, the gain/loss emotion determination unit 122 determines that the user (first user) who issued the instruction selected as the processing target is the user who has generated gain information, and determines the other user (second user) as the user who has generated loss information.

In step S505, the debt emotion determination unit 123 performs sensing on the user (first user) who issued the instruction selected as the processing target in step S504. That is, sensing is performed on the first user who is determined by the profit/loss emotion determination unit 122 to have generated gain information. The debt emotion determination unit 123 then calculates the degree of the first user's debt emotion toward the second user. Specifically, the debt emotion determination unit 123 calculates the degree of debt emotion based on each piece of information acquired from each sensor. The method of calculating the degree of debt emotion is the same as the calculation method described above.

In step S506, the debt emotion determination unit 123 determines whether the degree of debt emotion calculated in step S505 is equal to or greater than a first threshold. If the degree of debt emotion is equal to or greater than the first threshold, the process proceeds to step S507. On the other hand, if the degree of debt emotion is less than the first threshold, the process proceeds to step S512. Note that the first threshold is the same as the first threshold described above.

Here, if the certainty of the sensing is high, it is assumed that the certainty of each process of the control unit 120 based on that sensing is also high. On the other hand, if the certainty of the sensing is low, it is assumed that the certainty of each process of the control unit 120 based on that sensing is also low. For example, if the certainty of the sensing is high, it is possible to appropriately determine the first user's feelings of indebtedness, etc. For this reason, it is possible to proceed to each process from step S505 onwards on the condition that the certainty of the sensing is equal to or greater than a threshold. In this way, by determining whether or not a control process that controls the state of the agent device 200 is necessary based on the certainty of the sensing, it is possible to branch off the risk that the agent device 200 will take an action that the user cannot understand.

In step S507, the dissatisfaction determination unit 124 performs sensing on a user (second user) other than the first user based on each piece of information acquired from each sensor. Specifically, the dissatisfaction determination unit 124 calculates the degree of dissatisfaction with respect to the non-adoption of the second user's instruction based on each piece of information acquired from each sensor. The method of calculating the degree of dissatisfaction is the same as the calculation method described above. Note that, when there are three or more users, it is also assumed that at least one of the multiple users will have a feeling of indebtedness, and at least one of the multiple users will have a feeling of dissatisfaction. In such a case, the first user and the second user are set based on the determination result by the profit/loss determination unit 122, and sensing is performed for each of the first user and the second user to calculate the degree of indebtedness and the degree of dissatisfaction.

In step S508, the dissatisfaction determination unit 124 determines whether the degree of dissatisfaction calculated in step S507 is equal to or greater than a second threshold. If the degree of dissatisfaction is equal to or greater than the second threshold, the process proceeds to step S509. On the other hand, if the degree of dissatisfaction is less than the second threshold, the process proceeds to step S511. The second threshold is the same as the second threshold described above.

In step S509, the agent control unit 125 controls the agent device 200 to execute a performance that alleviates the first user's feelings of indebtedness in a manner that is difficult for the second user to perceive. For example, it is possible to cause the agent device 200 to execute a performance that alleviates the first user's feelings of indebtedness in the manner shown in FIG. 5 (D) and FIG. 6 (D1) to (D3).

In step S510, the agent control unit 125 controls the agent device 200 to execute a performance in which the second user apologizes in a manner that is difficult for the first user to perceive. For example, it is possible to cause the agent device 200 to execute a performance that alleviates the second user's feelings of dissatisfaction in the manner shown in Figures 5 (E) and (F).

Note that the process of step S511 corresponds to step S509, and therefore a description thereof will be omitted here. Also, the processes of steps S512 to S514 correspond to steps S507, S508, and S510, and therefore a description thereof will be omitted here. In this way, when the degree of the first user's feeling of indebtedness toward the second user is less than the first threshold value, and the degree of the second user's feeling of dissatisfaction regarding the non-adoption of the instruction is equal to or greater than the second threshold value, the first user is not followed up (step S509), and only the second user is followed up (step S514).

In step S515, the agent control unit 125 controls the agent device 200 to execute a confused performance. For example, it is possible to cause the agent device 200 to execute the mode shown in FIG. 3(G). For example, the agent device 200 may show both users U1 and U2 a confused expression, and output audio information such as "Was that the right answer?" from the agent device 200.

Here, it is assumed that user U2, who is shivering from the cold, thinks of user U1, who looks hot, and gives an insincere instruction (turn down the air conditioner temperature). In this way, when the adopted instruction is not sincere, it is assumed that the user who gave the insincere instruction is unlikely to develop feelings of indebtedness toward other users. Therefore, when it is estimated that the adopted instruction is not sincere, it may be assumed that the user who gave the instruction deliberately gave an instruction that would cause a loss to himself, and the agent device 200 may not be made to execute a performance that alleviates the specific emotion. Alternatively, a different follow-up may be performed. For example, a performance may be executed in which a voice such as "It was actually cold, wasn't it?" or "You were thinking about the other person" is output, or a performance that praises user U2 may be executed.

Whether or not an instruction is the user's true intention can be estimated based on at least one of the following: biometric information about the user, the content of the speech, and the user's actions. For example, it is possible to use image recognition technology that detects lies based on the user's facial expressions, and voice recognition technology that detects lies based on the user's voice.

Figure 7 shows an example in which follow-up is performed for both users who have developed feelings of indebtedness and users who have developed feelings of dissatisfaction. However, some of these steps may be omitted. For example, follow-up may be performed only for users who have developed feelings of indebtedness, and follow-up for users who have developed feelings of dissatisfaction may be omitted. In this case, it is possible to omit the processes of steps S507, S508, and S510 to S515.

That is, while FIG. 7 shows an example in which sensing is performed for both the first user and the second user, it is also possible to omit sensing for the second user (steps S507, S512) and perform only sensing for the first user (step S505). In this case, it is possible to perform a performance (step S509) that alleviates the first user's emotions in a manner that is difficult for the second user to perceive, using only the first user's emotions, without performing sensing for the second user (or even if sensing for the second user cannot be performed).

Also, FIG. 7 shows an example in which the agent device 200 executes each process based on instructions from the user. However, it is also assumed that the agent device 200 will execute each process automatically based on the user's surrounding environment, etc. In this case, too, it is assumed that each user will feel a gain or loss, a debt, or a dissatisfaction depending on the processing result of the agent device 200. Therefore, FIG. 8 shows an example in which the agent device 200 automatically executes each process based on the user's surrounding environment, etc.

[Example of operation of information processing device]
Fig. 8 is a flow chart showing an example of a control process in the information processing device 110. This control process is executed by the control unit 120 based on a program stored in the storage unit 130. This control process is constantly executed every control period when the agent device 200 is in an on state. This control process will be explained with reference to Figs. 1 to 7 as appropriate. This control process is a partial modification of the control process shown in Fig. 7, and therefore processes common to Fig. 7 are denoted by the same reference numerals and some of the explanations thereof will be omitted.

In step S521, the agent control unit 125 senses each user riding in vehicle C1 and their surroundings.

In step S522, the agent control unit 125 executes a predetermined process based on the sensing in step S521. For example, when sensing is performed on the surroundings of the vehicle C1 and the users U1 and U2, and the temperature inside the vehicle C1 is below a predetermined temperature and user U2 is detected shivering from the cold, the agent control unit 125 executes control to raise the set temperature of the air conditioner.

In step S523, the debt emotion determination unit 123 senses each user who is riding in the vehicle C1.

In step S524, the debt emotion determination unit 123 calculates the degree of debt emotion and the degree of dissatisfaction emotion of each user riding in the vehicle C1. Based on these degrees, it is possible to determine the users who have developed debt emotions and the users who have developed dissatisfaction emotions. For example, if two users U1 and U2 are riding in the vehicle C1, when one of them develops a debt emotion, the other may develop a dissatisfaction emotion. Also, if three users are riding in the vehicle C1, when one or two of them develop a debt emotion, the remaining users may develop a dissatisfaction emotion. In this way, the degree of debt emotion and the degree of dissatisfaction emotion are calculated for each of the multiple users, and the user who has developed a debt emotion is set as the first user, and the user who has developed a dissatisfaction emotion is set as the second user. For example, when the temperature inside the vehicle C1 is below a predetermined temperature and user U2 who is shivering from the cold is detected, when the control to increase the set temperature of the air conditioner is executed, it is assumed that user U2 will develop a sense of gain and a sense of debt.

In this way, when the first user feels indebted, the agent device 200 performs automatic processing, making it possible to follow up with the first user in a manner that is difficult for the second user to perceive.

Figure 8 shows an example in which follow-up is performed for both users who have developed feelings of indebtedness and users who have developed feelings of dissatisfaction. However, some of these steps may be omitted. For example, follow-up may be performed only for users who have developed feelings of indebtedness, and follow-up for users who have developed feelings of dissatisfaction may be omitted. In this case, it is possible to omit the processes of steps S507, S508, S510 to S514, and some of the processes of step S524.

For example, in step S524, only the degree of indebtedness of each user riding in vehicle C1 may be calculated, and sensing of the second user (steps S507, S512) may be omitted. In this case, it is possible to perform a performance (step S509) that alleviates the first user's emotion in a manner that is difficult for the second user to perceive, using only the indebtedness of the first user, without performing sensing of the second user in steps S507 and S512 (or even if sensing of the second user cannot be performed).

In FIG. 8, an example is shown in which each user is followed up based on the feelings of indebtedness and dissatisfaction that each user has experienced. However, it is also possible to follow up with each user by determining whether they have experienced a relative gain or loss without using these feelings. Thus, FIG. 9 shows an example of following up with each user based on whether they have experienced a relative gain or loss.

[Example of operation of information processing device]
Fig. 9 is a flow chart showing an example of a control process in the information processing device 110. This control process is executed by the control unit 120 based on a program stored in the storage unit 130. This control process is constantly executed every control period when the agent device 200 is in an on state. This control process will be explained with reference to Figs. 1 to 8 as appropriate. This control process is a partial modification of the control process shown in Fig. 8, and therefore processes common to Fig. 8 are denoted by the same reference numerals and some of the explanations thereof will be omitted.

In step S531, the profit/loss emotion determination unit 131 calculates the degree of profit/loss emotion of each user riding in the vehicle C1. In this way, the degree of profit/loss emotion and the degree of loss emotion are calculated for each of the multiple users, and the user who has experienced a profit emotion is the first user, and the user who has experienced a loss emotion is the second user. For example, when the temperature inside the vehicle C1 is below a predetermined temperature and user U2 who is shivering from the cold is detected, it is possible to determine that user U2 has experienced a profit emotion and user U1 has experienced a loss emotion when control is executed to increase the set temperature of the air conditioner. The degree of profit/loss emotion can be calculated based on the user's biological information, etc., in the same way as the calculation method for the degree of debt emotion and the degree of dissatisfaction emotion described above. For example, the degree of profit/loss emotion can be calculated based on the user's facial expression, speech, movement, etc.

In step S532, the profit/loss emotion determination unit 131 determines whether the degree of the first user's profit/loss emotion toward the second user calculated in step S531 is equal to or greater than a third threshold. If the degree of the profit/loss emotion is equal to or greater than the third threshold, the process proceeds to step S509. On the other hand, if the degree of the profit/loss emotion is less than the third threshold, the control process ends. The third threshold is a threshold for determining whether the user's profit/loss emotion is high, and can be set appropriately based on experimental data, etc.

In this way, when the first user benefits from automatic processing performed by the agent device 200, it is possible to follow up on the first user in a manner that is difficult for the second user to perceive.

[Examples of agent devices that can be installed outside of vehicles]
The above describes an example of the agent device 200 installed in the vehicle C1 and the agent image displayed on the vehicle C1. However, the present embodiment can also be applied to an agent device that can be removed from the vehicle C1, an agent device that can be installed outside the vehicle, and the like. For example, it is assumed that a user who possesses a portable agent device installs the agent device on the dashboard 2 of the vehicle C1 when getting into the vehicle C1, and carries the agent device when getting off the vehicle C1. It is also assumed that the user uses the agent device at home. It is also assumed that various devices (e.g., game devices, glasses-type displays, AR (Augmented Reality) glasses) capable of displaying an agent image (corresponding to the agent device 200) are used. It is also assumed that the present embodiment can be applied to an elevator control device capable of controlling the agent device, and an elevator control device capable of displaying an agent image (corresponding to the agent device 200).

For example, in a home, it is conceivable to execute control over each device installed in the home. For example, it is envisaged to execute control over heating the bath, cooking with kitchen utensils, outputting music, opening the front door and windows, etc. In these cases, it is envisaged to provide notification information regarding the bath, each kitchen utensil, doors, windows, audio output, etc.

For example, when a certain user feels a certain emotion (debt emotion, gain emotion) in response to a process based on instructions from multiple users or an automatic process based on sensing, a performance is executed to alleviate the certain emotion for the user. When a certain user feels dissatisfied (or a loss emotion) in response to a process based on instructions from multiple users or an automatic process based on sensing, a performance is executed to alleviate the dissatisfied emotion for the user (for example, an apology performance). Note that the user's specific emotion, dissatisfied emotion, etc. can be calculated based on the user's biometric information (for example, heart rate, blood vessels), the sound emitted by the user, and the user's appearance (for example, facial expression, movement), as in the above-mentioned example. Also, the performance to alleviate the certain emotion is the same as the example shown in FIG. 5 (D), FIG. 6 (D1) to (D3). Also, the performance to alleviate the dissatisfied emotion is the same as the example shown in FIG. 5 (E) (F). Note that each of these performances may be executed by the agent image using, for example, a device capable of displaying the agent image (for example, a game device, a glasses-type display, AR glasses). In these cases, the device capable of displaying the agent image corresponds to the agent device 200.

For example, when an elevator control device executes an effect to ease feelings of indebtedness, if a voice is output in the elevator, it may be heard by everyone. Therefore, the voice related to the effect is transmitted only to specific targets. For example, to a person who has just entered the elevator, voice such as "It's crowded, but you're not bothering anyone" or "It can't be helped, since you're in a hurry" can be output using a directional speaker so that only that person can hear it, or an agent image with a facial expression expressing these can be displayed on an electronic device (e.g., a smartphone) carried by the person. Some of these effects may be executed, for example, after the person who has just entered the elevator gets off the elevator. The position of the person who has just entered the elevator can be determined based on an image of the inside of the elevator acquired by a camera installed in the elevator.

Here, we will explain a method for detecting electronic devices carried by a person who has newly entered an elevator. For example, assume that the elevator is equipped with a communication unit capable of wireless communication with electronic devices. In this case, the elevator control device can detect radio waves emitted from an electronic device carried by a person entering the elevator. Therefore, when a new user enters the elevator, the newly detected electronic device can be detected as an electronic device carried by the new user. In addition, known communication technology can be used as a transmission method for transmitting information to unregistered electronic devices.

In this way, this embodiment can be applied to both a person already in the elevator (user B) and a new person getting in the elevator (user A). For example, assume that user A gives an instruction to get in the elevator (for example, pressing a button to instruct up or down), while user B gives an instruction to move to a specific floor. In this case, the elevator control device executes control to stop at the floor where user A is located and open the door based on the instruction from user A. In this case, after users A and B give different instructions, the elevator control device will adopt the instruction from user A first. For this reason, as described above, it is also assumed that user A will have a feeling of indebtedness toward user B. In such a case, the elevator control device will follow up on user A in a manner that is difficult for user B to perceive.

[Examples of Effects of This Embodiment]
In this way, in this embodiment, the process executed by the agent device 200 is determined based on the instruction from the user, the surrounding environment information, and the like. In this case, the emotions (e.g., gain/loss emotions, debt emotions, dissatisfaction emotions) of each user are estimated, and the agent device 200 provides appropriate follow-up to the user who is estimated to have a particular emotion. For example, if there is a user who has experienced a gain emotion due to the execution of the process by the agent device 200, and the user experiences an emotion (e.g., debt emotions) that makes the user want to apologize to other users, a performance is executed to ease the emotion of the user. In this way, by executing a performance in which the agent device 200 appears to be acting for other users, not for its own benefit, it is possible to realize a performance that enhances the user's acceptance and empathy.

In addition, in this embodiment, in a situation where there are multiple users, after the agent device 200 processes, it follows at least one of the users in a manner that is difficult for the other users to notice. This makes it possible to prevent one of the users from feeling uncomfortable.

In addition, in this embodiment, in a situation where there are multiple users, when the agent device 200 adopts the instructions (or opinions) of a first user, the agent device 200 follows up on the first user in a manner that is difficult for the second user to perceive. This makes it possible to eliminate the feeling of indebtedness (e.g., regret) that the first user feels toward the second user.

In addition, in this embodiment, in a situation where there are multiple users, the agent device 200 can perform a predetermined process and then follow up with the multiple users. In particular, in this embodiment, it is possible to realize appropriate follow-up with a user who has developed a positive emotion through a selection made by the agent device 200.

In addition, in this embodiment, by performing a performance (e.g., behavior) that suppresses the emotion of any occupant of vehicle C1 who has experienced a negative emotion in a manner that is difficult for others to perceive, it is possible to suppress all negative emotions among the occupants of vehicle C1.

In addition, in this embodiment, it is possible to achieve a more comfortable drive assistance by using the agent device 200 to suppress negative emotions felt by the occupants of the vehicle C1.

In addition, in this embodiment, in addition to providing support to users who have gained (users who have developed feelings of gain) and users who have developed feelings of indebtedness, some kind of response is also provided to users who have incurred losses (users who have developed feelings of loss) and users who have developed feelings of dissatisfaction. This makes it possible to appropriately suppress various emotions that arise in the occupants of vehicle C1.

[Example of executing processing in other devices or systems]
In the above, examples have been shown in which the control processes and the like are executed in the agent device 200 and the information processing device 110 (or the information processing system 100), but all or part of each of these processes may be executed in other devices. In this case, the information processing system is made up of each device that executes part of each of these processes. For example, at least part of each process can be executed using various information processing devices and various electronic devices, such as in-vehicle devices, devices that can be used by users (e.g., smartphones, tablet terminals, personal computers, car navigation devices, IVIs), and servers that can be connected via a predetermined network such as the Internet.

In addition, a part (or all) of the information processing system capable of executing the functions of the information processing device 110 (or the information processing system 100) may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[Configuration Example and Effects of This Embodiment]
The information processing method according to the present embodiment is an information processing method for controlling an agent device 200 (an example of a device) capable of executing a plurality of processes for users U1 and U2. This information processing method includes a detection process (steps S506, S532) for detecting a user U2 (an example of a first user) who has a specific emotion including at least one of a debt emotion and a gain emotion caused by the selection from among the plurality of users U1 and U2 when a predetermined process is selected from among a plurality of processes in a state where a plurality of users U1 and U2 exist, and a control process (steps S509, S511) for causing the agent device 200 to perform a performance that alleviates the specific emotion for the user U2 in a manner that is difficult to be perceived by a user U1 (an example of a second user) other than the user U2 when the user U2 is detected. Also, the program according to the present embodiment is a program for causing a computer to execute each of these processes. In other words, the program according to the present embodiment is a program for causing a computer to realize each function that the information processing device 110 can execute.

According to this configuration, when user U2 develops a specific emotion toward user U1 due to the selection of a specific process, it is possible to execute a performance that alleviates the specific emotion in a manner that is difficult for user U1 to perceive. This makes it possible to provide appropriate support to user U2, and to increase the ease of use of the agent device 200. It is also possible to eliminate the unpleasant feelings of user U2.

In the information processing method according to this embodiment, the aspect that is difficult for the user U1 (an example of a second user) to perceive is at least one of a first aspect in which the agent device 200 (an example of a device) executes a performance in a manner that is difficult for the user U1 to perceive, and a second aspect in which the agent device 200 executes a performance at a timing that is difficult for the user U1 to perceive.

With this configuration, it is possible to select an appropriate mode from the first mode and the second mode as a mode that is difficult for user U1 to perceive, and it is possible to realize appropriate follow-up for user U2.

In the information processing method according to this embodiment, the agent device 200 (an example of a device) is an agent device capable of communicating with multiple users U1 and U2, and the first aspect is at least one of the following: having the agent device 200 behave in a predetermined manner while facing the agent device 200 toward the user U2 (an example of a first user); having the agent device 200 output a sound that can only be heard by the user U2; having the electronic device MC2 held by the user U2 output predetermined information related to the agent device 200; and having the predetermined information related to the agent device 200 displayed in a display area (e.g., display area 4a) of a device other than the agent device 200 (e.g., a HUD, a HUD display device).

With this configuration, it is possible to select an appropriate mode from among multiple modes as a mode that is difficult for user U1 to perceive, depending on the environment in which multiple users U1 and U2 exist, and it is possible to realize appropriate follow-up for user U2.

In the information processing method according to this embodiment, the agent device 200 (one example of a device) is an agent device capable of communicating with multiple users U1 and U2, and the presentation is at least one of a presentation that expresses a behavior that alleviates a specific emotion and a presentation that outputs a sound that alleviates a specific emotion.

With this configuration, it is possible to select an appropriate effect from among multiple effects to alleviate a particular emotion, and it is possible to provide appropriate support to user U2.

In the information processing method according to this embodiment, the agent device 200 is a device with a face, and the behavior of the agent device 200 is at least one of the following: the eyes of the agent device 200 gazing at the eyes of a user U2 (an example of a first user), the face of the agent device 200 nodding while looking at the user U2, a predetermined mouth movement performed by the agent device 200 while looking at the user U2, and displaying a predetermined image or predetermined text on the surface of the agent device 200.

With this configuration, it is possible to adopt an appropriate behavior from among multiple behaviors as the behavior of the agent device 200, and it is possible to realize appropriate follow-up for the user U2.

In the information processing method according to this embodiment, the timing of the second aspect is at least one of the timing set based on the concentration state of the user U1 (an example of a second user) and the timing when the user U1 moves away from the location where the agent device 200 (an example of a device) is located. Note that the timing set based on the concentration state of the user U1 means, for example, the timing set based on the degree to which the user U1 is concentrating on something. The degree to which the user U1 is concentrating on something can be determined, for example, based on the driving scene in the vehicle C1.

With this configuration, it is possible to select an appropriate timing from among multiple timings as the timing of the second mode, and it is possible to realize appropriate follow-up for user U2.

In the information processing method according to this embodiment, the agent device 200 (one example of a device) is an agent device capable of executing multiple processes based on instructions from multiple users U1, U2, and in the detection process (step S506), when multiple conflicting and different instructions are issued from each of the multiple users U1, U2, the user U2 who issued an instruction selected by the agent device 200 from among the multiple instructions is inferred as the user who has experienced a gained emotion as a result of that selection, and the user U2 who has experienced the gained emotion is detected as the first user.

With this configuration, when multiple users U1 and U2 issue multiple, conflicting instructions, it becomes possible to appropriately detect user U2 who is the target of the performance to alleviate a specific emotion.

In the information processing method according to this embodiment, the detection process (step S506) estimates whether the instruction selected by the agent device 200 is the true intention of the user U2 based on at least one of the biometric information, speech content, and movement of the user U2 who issued the instruction, and if the instruction is estimated to be not the true intention of the user U2, the agent device 200 is not caused to execute a performance to alleviate the specific emotion. For example, if the instruction is estimated to be not the true intention of the user U2, it is possible to detect the user U2 as a user who has suffered a loss due to the instruction that was not the true intention.

With this configuration, for user U2 who has issued an instruction that is not sincere, no effect is performed to alleviate a particular emotion, making it possible to reduce unnecessary effects. This makes it possible to reduce the amount of calculation processing that is required after detecting user U2 who has issued an instruction that is not sincere, and also to reduce power consumption.

In the information processing method according to the present embodiment, in the detection process (steps S506 and S532), a user U2 who has gained a feeling of gain from the selection is estimated from among the multiple users U1 and U2 based on at least one element of environmental information (e.g., the temperature in the cabin of the vehicle C1) surrounding the multiple users U1 and U2, bioinformation regarding the multiple users U1 and U2, the contents of speech regarding the multiple users U1 and U2, and the actions of the multiple users U1 and U2, and the user U2 who has gained a feeling of gain is detected as the first user. For example, it is possible to detect a user U2 who is estimated to have gained a feeling of gain from the selection from among the users U1 and U2 based on the temperature in the cabin of the vehicle C1, the facial expressions of the users U1 and U2, the transition of the gaze of the users U1 and U2, the contents of speech of the users U1 and U2, the tone of speech of the users U1 and U2, the movements of the users U1 and U2, etc. For example, assume that the temperature inside the vehicle C1 is below a predetermined value, the user U2 repeatedly utters the voice "It's cold, it's cold," the user U2 is shivering, and the user U1 is wearing a new jacket, and the user U1 is determined to be in a normal state. In this case, the agent device 200 determines that the user U2 is beyond the limit of cold, but the user U1 can tolerate the heat, and selects to adopt the command from the user U2 to "turn up the air conditioner temperature." In addition, in the detection process (steps S506 and S532), the user U2 is detected as a first user who has experienced a specific emotion based on each of these elements (the temperature inside the vehicle, the voice saying "It's cold, it's cold," the shivering, and the wearing of a new jacket). The agent device 200 (or the information processing device 110) can automatically execute at least one of a number of processes (e.g., setting the temperature of an air conditioner, providing notification information, controlling a radio) based on various elements such as environmental information around the users U1 and U2, biometric information about the users U1 and U2, speech content about the users U1 and U2, and actions about the users U1 and U2. In this way, when a predetermined process (e.g., setting the temperature of an air conditioner) is executed automatically, the detection process (steps S506, S532) detects the first user who is presumed to have experienced a gaining emotion based on these various elements.

With this configuration, it is possible to estimate users who have experienced a feeling of gain using appropriate parameters. It is also possible to appropriately detect users who have experienced a feeling of gain based on the elements used when selecting a process to be executed from among multiple processes.

The information processing device 110 is an information processing device that controls an agent device 200 (an example of a device) capable of executing a plurality of processes for users U1 and U2. When a predetermined process is selected from a plurality of processes in a state in which a plurality of users U1 and U2 exist, the information processing device 110 includes a gain/loss emotion determination unit 122 and a debt emotion determination unit 123 (an example of a detection unit) that detect a user U2 (an example of a first user) who has a specific emotion including at least one of a debt emotion and a gain emotion due to the selection from among the plurality of users U1 and U2, and an agent control unit 125 (an example of a control unit) that causes the agent device 200 to perform a performance that alleviates the specific emotion for the user U2 in a manner that is difficult to perceive by a user U1 (an example of a second user) other than the user U2 when the user U2 is detected. The information processing device 110 may be a device built into the agent device 200, or may be a device different from the agent device 200. In addition, instead of the information processing device 110, an information processing system configured of a plurality of devices capable of executing each process realized by the information processing device 110 may be used.

According to this configuration, when user U2 develops a specific emotion toward user U1 due to the selection of a specific process, it is possible to execute a performance that alleviates the specific emotion in a manner that is difficult for user U1 to perceive. This makes it possible to provide appropriate support to user U2, and to increase the ease of use of the agent device 200. It is also possible to eliminate the unpleasant feelings of user U2.

Note that each processing step shown in this embodiment is an example for realizing this embodiment, and the order of some of the processing steps may be changed within the scope that allows this embodiment to be realized, and some of the processing steps may be omitted or other processing steps may be added.

Note that each process shown in this embodiment is executed based on a program for causing a computer to execute each processing procedure. Therefore, this embodiment can also be understood as an embodiment of a program that realizes the function of executing each of these processes, and a recording medium that stores the program. For example, the program can be stored in the storage device of the information processing device by an update process for adding a new function to the information processing device. This makes it possible to cause the updated information processing device to execute each of the processes shown in this embodiment.

The above describes embodiments of the present invention, but the above embodiments merely show some examples of applications of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments.

100 Information processing system, 110 Information processing device, 120 Control unit, 121 Instruction receiving unit, 122 Profit/loss emotion determination unit, 123 Debt emotion determination unit, 124 Dissatisfaction emotion determination unit, 125 Agent control unit, 130 Memory unit, 131 Agent information DB, 132 Instruction information DB, 133 Notification information DB, 134 Map information DB, 135 Surrounding situation DB, 136 User state DB, 140 Communication unit, 200 Agent device, 210 Display unit, 220 Sound output unit, 230 Drive unit, MC1, MC2 Electronic device

Claims (10)

An information processing method for controlling a device capable of executing a plurality of processes for a user, comprising:
a detection process for detecting, when a predetermined process is selected from among a plurality of processes in a state where a plurality of users exists, a first user who is a user who has experienced a specific emotion including at least one of a debt emotion and a gain emotion due to the selection from among the plurality of users;
and when the first user is detected, a control process that causes the device to execute a performance that alleviates the specific emotion for the first user in a manner that is difficult to perceive by a second user other than the first user. 2. The information processing method according to claim 1,
The aspect that is difficult to be perceived by the second user is at least one of a first aspect in which the device executes the performance in a manner that is difficult to be perceived by the second user, and a second aspect in which the device executes the performance at a timing that is difficult to be perceived by the second user.
Information processing methods. 3. The information processing method according to claim 2,
the device is an agent device capable of communicating with the plurality of users,
The first aspect is at least one of an aspect of causing the agent device to perform a predetermined behavior while facing the agent device toward the first user, an aspect of causing the agent device to output a sound that can only be heard by the first user, an aspect of causing an electronic device possessed by the first user to output predetermined information related to the agent device, and an aspect of causing a device other than the agent device to display predetermined information related to the agent device in a display area.
Information processing methods. 2. The information processing method according to claim 1,
the device is an agent device capable of communicating with the plurality of users,
The effect is at least one of an effect of expressing a behavior that alleviates the specific emotion and an effect of outputting a sound that alleviates the specific emotion.
Information processing methods. 5. The information processing method according to claim 3, further comprising:
the agent device is a device having a face part,
The behavior is at least one of the following: a movement of the eyes of the agent device gazing at the eyes of the first user; a movement of the face of the agent device nodding while looking at the first user; a predetermined mouth movement performed by the agent device while looking at the first user; and displaying a predetermined image or predetermined text on a surface of the agent device.
Information processing methods. 3. The information processing method according to claim 2,
The timing is at least one of a timing set based on a concentration state of the second user and a timing when the second user moves away from a location where the device is located.
Information processing methods. 2. The information processing method according to claim 1,
the device is an agent device capable of executing the plurality of processes based on instructions from the plurality of users;
in the detection process, when a plurality of different and conflicting instructions are issued from each of the plurality of users, a user who issued an instruction selected by the agent device from among the plurality of instructions is inferred as a user who has experienced the gained emotion as a result of the selection, and the user who has experienced the gained emotion is detected as the first user;
Information processing methods. 8. An information processing method according to claim 7,
In the detection process, it is estimated whether the instruction selected by the agent device is the true intention of the user based on at least one of biometric information, speech content, and movement of the user who issued the instruction;
If it is estimated that the instruction is not the user's true intention, the device is not allowed to execute the performance.
Information processing methods. 2. The information processing method according to claim 1,
In the detection process, a user who has experienced the feeling of gain due to the selection is estimated from among the plurality of users based on at least one element of environmental information around the plurality of users, biological information about the plurality of users, speech contents about the plurality of users, and actions about the plurality of users, and the user who has experienced the feeling of gain is detected as the first user.
Information processing methods. An information processing device that controls a device capable of executing a plurality of processes for a user,
a detection unit that detects, when a predetermined process is selected from a plurality of processes in a state where a plurality of users exists, a first user who is a user from the plurality of users and who has experienced a specific emotion including at least one of a debt emotion and a gain emotion due to the selection;
An information processing device comprising: a control unit that, when the first user is detected, causes the device to execute a performance that alleviates the specific emotion for the first user in a manner that is difficult to perceive by a second user other than the first user.

Images