JP2010244190A - Communication terminal and information display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide appropriate information responsive to a psychological state of a user, by accurately predicting the psychological state of the user. <P>SOLUTION: A communication terminal 1 includes: a sound collection unit 11 for collecting surrounding sound to acquire it as sound information; a user voice extraction unit 15 for extracting user voice information representing voice of the user from the sound information; an another person's voice extraction unit 16 for extracting another person's voice information representing voice of another person other than the user from the sound information; an emotion analysis unit 17 for identifying an emotional level of the user, based on the user voice information and identifying an emotional level of the another person based on the another person's voice information; an output information decision unit 19 for deciding information to be provided to the user based on the emotional level of the user and the emotional level of the another person; and an information display unit 20 for displaying output information to present it to the user. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a communication terminal and an information display method.

  In a conventional communication terminal such as a mobile phone, a technique for analyzing a voice during a call and estimating a speaker's emotion is known (for example, Patent Document 1). In addition, by using a technique for estimating the emotion of a speaker as described in Patent Document 1, a pictogram or symbol corresponding to the emotion of the speaker (user) is appropriately selected in a voice input format mail creation function. A technique for adding to a text input by voice is known (for example, Patent Document 2).

JP 2006-106711 A JP 2006-277567 A

  Here, in order to improve the convenience and affinity of the communication terminal, the communication terminal presents an appropriate message on the display, changes the character conversion candidate and pictogram according to the situation where the user is placed. It is preferable if possible.

  However, even if the user's emotion is estimated using only the user's own voice information as in the user's emotion estimation method described in Patent Document 1 and Patent Document 2, the situation where the user is placed is completely Is likely not understandable. This is because the situation in which the user is placed changes not only by the user's own emotions but also by the influence of the surrounding environment and the relative relationship with the other person who is talking. . Therefore, as described in Patent Document 2, even if information to be provided to the user is selected based only on the estimated emotion of the user, it does not match the situation in which the user is actually placed and is appropriate for the user. May not be useful information.

  The present invention has been made to solve the above-described problem, and accurately estimates a situation where a user is placed and can provide appropriate information according to the situation of the user and an information display. It aims to provide a method.

  In order to solve the above-described problems, a communication terminal according to the present invention includes a sound collecting unit that collects ambient sounds and acquires the sound as sound information, and a user voice indicating the user's voice from the sound information acquired by the sound collecting unit. User voice extraction means for extracting information, other person voice extraction means for extracting other person voice information indicating the voice of another person other than the user from the sound information acquired by the sound collection means, and user voice extraction means Based on the user voice information, the user's emotion level is identified, and based on the other person's voice information extracted by the other person's voice extraction means, the emotion analysis means for identifying the other person's emotion level and the emotion analysis means Information determining means for determining information to be provided to the user based on the sentiment level of the user and the emotion level of the other person, and displaying the information determined by the information determining means to the user Characterized in that it comprises a Shimesuru display means.

  Similarly, in order to solve the above-described problem, an information display method according to the present invention includes a sound collection step of collecting ambient sound and acquiring it as sound information, and a user's voice from the sound information acquired in the sound collection step. A user voice extraction step for extracting user voice information indicating a voice, a voice extraction step for extracting voice information of another person other than the user from the sound information acquired in the sound collection step, and a user voice extraction An emotion analysis step for identifying a user's emotion level based on the user voice information extracted in the step, and an emotion analysis step for identifying the emotion level of the other person based on the other person's voice information extracted in the other person's voice extraction step; Information determination for determining information to be provided to the user based on the emotion level of the user identified in the analysis step and the emotion level of the other person A step, a display step of presenting to the user by displaying the information determined by the information determining step, characterized in that it comprises a.

  According to such a communication terminal and an information display method, the emotion level of the other person is identified based on the voice of the other person extracted from the sound information of the surroundings of the user, while identifying the emotion level of the user based on the voice of the user. Are also identified. Then, information to be provided to the user is determined based on the emotion level of these users and the emotion level of others. For this reason, the situation where the user is placed by comparing the emotion level of the user himself with the emotion level of others around the user, and also taking into account the influence of the relative relationship between the user and others around the user As a result, it is possible to provide appropriate information according to the user's situation.

  In the communication terminal according to the present invention, it is preferable that the sound collecting means always collects surrounding sounds and obtains them as sound information. As a result, it is possible to recognize the situation where the user is placed even during a call, and to provide the user with more appropriate information.

  According to the communication terminal and the information display method according to the present invention, it is possible to accurately estimate the user's psychological state and provide appropriate information according to the user's psychological state.

It is a functional block diagram of the communication terminal which concerns on one Embodiment of this invention. It is a hardware block diagram of a communication terminal. It is a figure which shows an example of a structure of an output information storage part. It is a figure which shows the grounds which select the situation of the place assumed by the emotion level of a user and others in each output information shown in FIG. 3, and the output information. It is a figure which shows an example of a structure of an output information storage part. In each output information shown in FIG. 5, it is a figure which shows the condition of the place assumed by the emotion level of a user and others, and the basis for selecting the output information. It is a flowchart which shows the sample audio | voice registration process performed in the communication terminal of this embodiment. It is a flowchart which shows the information display process performed in the communication terminal of this embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a functional block diagram of a communication terminal according to an embodiment of the present invention. As shown in FIG. 1, the communication terminal 1 includes a sound collection unit (sound collection unit) 11, a voice registration unit 12, a user voice storage unit 13, a sound information storage unit 14, and a user voice extraction unit (user voice extraction unit) 15. , Other person voice extraction unit (other person voice extraction unit) 16, emotion analysis unit (emotion analysis unit) 17, output information storage unit 18, output information determination unit (information determination unit) 19, and information display unit (display unit) 20 is provided.

  FIG. 2 is a hardware configuration diagram of the communication terminal 1. The communication terminal 1 physically includes a CPU (Central Processing Unit) 111, a RAM (Random Access Memory) 112 and a ROM (Read Only Memory) 113 that are main storage devices, and an operation unit 114 such as an input key that is an input device. , A display device 115, a wireless communication unit 116, a microphone 117, and the like. Each function of the communication terminal 1 shown in FIG. 1 is configured such that predetermined computer software is loaded on hardware such as the CPU 111 and the RAM 112 shown in FIG. This is realized by operating the communication unit 116 and the microphone 117 and reading and writing data in the RAM 112 and the ROM 113.

  Next, each function of the communication terminal 1 shown in FIG. 1 will be described.

  The sound collecting unit 11 always collects surrounding sounds and acquires them as sound information. The sound collection unit 11 transmits the acquired sound information to the sound information storage unit 14. The sound collection unit 11 transmits the acquired sound information to the voice registration unit 12 when the communication terminal 1 executes a user voice registration process described later. Note that the sound collecting unit 11 is generally a microphone 117 attached to the communication terminal 1.

  The voice registration unit 12 is a part for registering user voice. The voice registration unit 12 receives the sound information acquired by the sound collection unit 11 in the user voice registration process described later, and extracts the voice information from the sound information. Then, the extracted voice information is transmitted to and stored in the user voice storage unit 13 as sample voice information indicating the sample voice of the user of the communication terminal 1.

  The user voice storage unit 13 is a part that holds sample voice information indicating the sample voice of the user of the communication terminal 1. When the user voice storage unit 13 receives the sample voice information from the voice registration unit 12, the user voice storage unit 13 updates the held sample voice information. In addition, the user voice storage unit 13 provides the held sample voice information to the user voice extraction unit 15.

  The sound information storage unit 14 is a part that holds sound information acquired by the sound collection unit 11. In this embodiment, the sound information storage unit 14 is configured to always hold the latest five minutes of sound information among the sound information transmitted from the sound collection unit 11. The sound information storage unit 14 provides the held sound information to the user voice extraction unit 15.

  The user voice extraction unit 15 is a part that extracts user voice information indicating the voice of the user from the sound information held in the sound information storage unit 14. Specifically, the user voice extraction unit 15 periodically acquires the sound information from the sound information storage unit 14 and uses the sample voice information held in the user voice storage unit 13 to obtain the sound information from the user. Check if audio is included. When the user information is included in the sound information, the user sound information is extracted from the sound information, the extracted user sound information is transmitted to the emotion analysis unit 17, and the sound information excluding the user sound information is sent to the other person. It transmits to the voice extraction unit 16.

  The other person voice extraction unit 16 is a part that extracts the other person voice information indicating the voice of the other person other than the user of the communication terminal 1 from the sound information. Specifically, when the other person voice extraction unit 16 receives the sound information from which the user voice information is excluded from the user voice extraction unit 15, the other person voice extraction unit 16 analyzes the sound information and extracts the other person voice information, The voice information is transmitted to the emotion analysis unit 17.

  In this embodiment, the “other” is a person who is present around the user at such a distance that the sound collecting unit 11 can pick up the voice, and is directly talking to the user or the user's A person who has a conversation with another person in the surrounding area and can affect the psychological state of the user.

  The emotion analysis unit 17 identifies the emotion level of the user of the communication terminal 1 based on the user voice information extracted by the user voice extraction unit 15, and based on the other person voice information extracted by the other person voice extraction unit 16, Identify the emotional levels of others around the user. Specifically, the emotion analysis unit 17 identifies the emotion level using information such as the volume, waveform, pitch, and phoneme of the voice information. In this embodiment, four patterns of “joy”, “anger”, “sorrow”, and “easy” are set as the emotion level, and regarding “anger”, “violent anger”, “anger”, It is subdivided into three patterns of “inner anger”. It should be noted that a known technique may be used for emotion estimation in the emotion analysis unit 17, and such a technique is described in, for example, International Publication No. WO00 / 62279.

  The output information storage unit 18 holds output information corresponding to the emotion level of the user and the emotion level of the other person. In other words, the output information storage unit 18 defines a rule as to what kind of output is appropriate when the user's emotion and the surrounding human emotion are multiplied.

  A specific configuration of the output information storage unit 18 will be described with reference to FIGS. FIG. 3 is a diagram illustrating an example of the configuration of the output information storage unit 18. As shown in FIG. 3, the output information storage unit 18 includes one of the user's four levels of emotion levels (“joy”, “anger”, “sorrow”, “easy”) and the other level of emotion levels. A total of 16 patterns of output information corresponding to one of these are held. Also, FIG. 4 shows the situation of the place where the user is placed and the basis for selecting the output information, which are assumed according to the emotion level between the user and others in each of the 16 patterns of output information shown in FIG. FIG. In this example, when the emotion level of the user is “pleasant”, if the emotion level of the other person is “pleasant”, it is based on the ground that “the user and the surroundings are delighted, so they are happy” (see FIG. 4). ), The output information is “good”. Also, if the emotion level of the other person is “angry”, the output information is “based on the ground that“ the user is happy but the surroundings are angry, so alert the user who seems to be ecstatic ”. Keep calm! " In this way, the output information storage unit 18 can output different outputs according to the situation of each place where the user is placed when the emotion level of the other person is different even if the emotion level of the user is the same. Information is configured to be selected.

  FIG. 5 is a diagram showing output information when the emotion level is further subdivided for the portion 31 in which the emotion level of both the user and the other person is “angry” in FIG. 3. As shown in FIG. 5, regarding the emotion level “anger”, the output information storage unit 18 further has three patterns of emotion levels of the user and others (“violent anger”, “anger”, “inner anger”). A total of nine patterns of output information are held in accordance with the emotional levels of the user and others. Also, FIG. 6 is assumed based on the emotion level between the user and the other person in each of the nine patterns of output information of the portion 31 where the emotion level of the user and the other person is “angry” as shown in FIG. It is a figure which shows the grounds for selecting the output situation and the situation of the place where the user is placed. As shown in FIGS. 5 and 6, even when the user and others are both angry, the situation of the place where the user is placed is considered to be different depending on the degree of anger of each person, Different output information according to the situation of the place is selected.

  Returning to FIG. 1, the output information determination unit 19 determines output information to be provided to the user with reference to the output information storage unit 18 based on the emotion levels of the user and others identified by the emotion analysis unit 17.

  The information display unit 20 is a part that displays the output information determined by the output information determination unit 19 and presents it to the user. The information display unit 20 is generally the display 115 of the communication terminal 1.

  Next, processing executed in the communication terminal 1 of the present embodiment will be described using the flowcharts shown in FIGS. 7 and 8, and an information display method according to the present embodiment will be described. FIG. 7 is a flowchart showing sample voice registration processing executed in the communication terminal 1 of the present embodiment. This sample voice registration process is mainly executed by the voice registration unit 12.

  First, when the sample voice registration process is started by the voice registration unit 12 (S101), for example, an instruction for prompting the user to speak is displayed on the display 115 to register the user's voice. Then, sound collection is started by the sound collection unit 11 (S101), and sound information including the collected user's voice is transmitted to the sound registration unit 12.

  Next, the voice registration unit 12 extracts voice information from the sound information received from the sound collection unit 11 (S103), and confirms whether or not the extracted voice is one type (S104). If the extracted voice is not one type, it cannot be registered as the user's voice, and the process returns to step S103 to redo the sound collection. When the extracted voice is one type, the extracted voice is registered in the user voice storage unit 13 as sample voice for extracting the user voice (S105).

  FIG. 8 is a flowchart showing information display processing executed in the communication terminal 1 of the present embodiment. This information display process is performed after the sample voice registration process shown in FIG. 7 is executed and the user's sample voice is registered in the user voice storage unit 13.

  First, the sound information around the communication terminal 1 for the latest 5 minutes collected by the sound collection unit 11 is acquired from the sound information storage unit 14 by the user sound extraction unit 15 and the sound information is extracted from the sound information ( The voice information is compared with the sample voice information held in the user voice storage unit 13 (S201: sound collection step) (S202).

  Next, based on the comparison result of step S202, it is confirmed whether or not the user voice is included in the sound information in the sound information storage unit 14 (S203). If the user information is included in the sound information, the process proceeds to step S204. If the user information is not included in the sound information, the process returns to step S201 and waits until the user sound extracting unit 15 detects again that the sound information in the sound information storage unit 14 includes the sound.

  When it is determined in step S203 that the user information is included in the sound information, the user sound information is extracted from the sound information by the user sound extraction unit 15 (S204: user sound extraction step), and the extracted user sound information is extracted. Is transmitted to the emotion analysis unit 17, and the sound information excluding the user sound information is transmitted to the other person sound extraction unit 16.

  Next, the other person voice extraction unit 16 extracts the other person voice information from the sound information from which the user voice information is excluded by the user voice extraction unit 15 (S205: Other person voice extraction step). Is transmitted to the emotion analysis unit 17.

  Furthermore, the emotion analysis unit 17 identifies the emotion level of the user of the communication terminal 1 based on the user voice information, and identifies the emotion level of the others present around the user based on the other person voice information ( S206: Emotion analysis step).

  Then, the output information determination unit 19 determines output information to be provided to the user with reference to the output information storage unit 18 based on the emotion levels of the user and others (S207: information determination step), and the information display unit 20 Thus, the output information determined by the output information determination unit 19 is displayed on the display 115 and presented to the user (S208: display step).

  As described above, according to the communication terminal 1 and the information display method according to the present embodiment, the emotion level of the user is identified based on the user's voice and the other person extracted from the sound information around the user. The other person's emotional level is also identified based on the voice. Then, information to be provided to the user is determined based on the emotion level of these users and the emotion level of others. For this reason, the situation where the user is placed by comparing the emotion level of the user himself with the emotion level of others around the user, and also taking into account the influence of the relative relationship between the user and others around the user As a result, it is possible to provide appropriate information according to the user's situation.

  In addition, since the sound collecting unit 11 always collects ambient sounds and obtains them as sound information, it is possible to recognize the situation where the user is placed even during a call, and to provide the user with more appropriate information. Can be provided.

  The communication terminal and the information display method according to the present invention have been described with reference to the preferred embodiments, but the present invention is not limited to the above embodiments. For example, in the above embodiment, a message presented on the display 115 as output information to be provided to the user is used, but in addition to this, a character conversion candidate or a pictograph in the character input function may be applied.

  In addition to the “joy”, “anger”, “sorrow”, and “easy” exemplified in the above embodiment, the emotion level of the user and others for determining the output information is, for example, “anger”, “anaerobic” Other emotions such as “favor” may be applied.

  Also, the sound information held in the sound information storage unit 14 is analyzed to determine the content of the conversation, the relationship between the user and others (friend, boss, subordinate, lover, family, etc.), the location of the conversation (company, It is possible to identify more detailed information such as home, department store, etc., and provide a plurality of output information storage units 18 that can be appropriately selected according to this information, and to provide output information further adapted to the psychological state of the user You may comprise so that it can do.

DESCRIPTION OF SYMBOLS 1 ... Communication terminal, 11 ... Sound collection part (sound collection means), 12 ... Voice registration part, 13 ... User voice storage part, 14 ... Sound information storage part, 15 ... User voice extraction part (user voice extraction means), 16 ... other person voice extraction unit (other person voice extraction unit), 17 ... emotion analysis unit (emotion analysis unit), 18 ... output information storage unit, 19 ... output information determination unit (information determination unit), 20 ... information display unit ( Display means).

Claims (3)

Sound collecting means for collecting surrounding sound and obtaining it as sound information;
User voice extraction means for extracting user voice information indicating the user's voice from the sound information acquired by the sound collection means;
Another person voice extraction means for extracting other person voice information indicating the voice of another person other than the user from the sound information acquired by the sound collecting means;
Emotion that identifies the user's emotion level based on the user voice information extracted by the user voice extraction means and also identifies the emotion level of the other person based on the other person's voice information extracted by the other person's voice extraction means Analysis means;
Information determining means for determining information to be provided to the user based on the emotion level of the user identified by the emotion analyzing means and the emotion level of the other;
Display means for displaying the information determined by the information determination means and presenting it to the user;
A communication terminal comprising:   The communication terminal according to claim 1, wherein the sound collecting unit constantly collects surrounding sounds and obtains them as sound information. A sound collection step for collecting ambient sound and acquiring it as sound information;
A user voice extraction step for extracting user voice information indicating a user voice from the sound information acquired in the sound collection step;
Other person voice extraction step of extracting other person voice information indicating the voice of another person other than the user from the sound information acquired in the sound collecting step;
Emotion that identifies the user's emotion level based on the user voice information extracted in the user voice extraction step and also identifies the emotion level of the other person based on the other person's voice information extracted in the other person's voice extraction step An analysis step;
An information determination step for determining information to be provided to the user based on the emotion level of the user identified in the emotion analysis step and the emotion level of the other person,
A display step of displaying the information determined in the information determination step and presenting it to the user;
An information display method comprising:

Images