JP4871552B2 - Information processing terminal - Google Patents

Description

  The present invention relates to an information processing terminal that notifies an emotion of a person who utters a voice estimated from the voice.

A technique for estimating a person's emotion from a voice uttered by the person, specifically, from the volume of the voice, the waveform of the voice, the pitch of the voice, or the phoneme is disclosed (Patent Document 1). Also, based on the emotion estimated using the technique of Patent Document 1, etc., the expression of the face model formed on the 3D computer graphic simulating a human face is displayed in three dimensions, and the emotion is displayed. The technique to express is disclosed (patent document 2). With these technologies, it is possible to visually read the emotion of the person who uttered the sound only from the sound.
International Publication Number WO00 / 62279 Patent publication number 2002-298155

  However, when it is estimated that a certain voice includes a plurality of emotions, for example, six factors such as joy, sadness, surprise, disgust, anger, and fear are often used as the main factors of emotion information. Presenting emotional results in real time has the problem of directly communicating emotions that you do not want to show to the other party when communicating.

  Conventionally, a voice call by telephone has a problem that there is no way to look back on the call status at a later time, just to record the call time without leaving the conversation status at that time like email. .

  The present invention has been made in view of the above circumstances, and expresses the emotions of the person who uttered the voice directly in real time, thereby expressing that the expression he does not want to express is conveyed directly to the other party. It is an object to provide an information processing terminal that can be used.

  It is another object of the present invention to provide an information processing terminal that can be looked back later to grasp the overall picture of a call situation.

The information processing terminal according to the present invention includes a voice input unit for inputting voice, a parameter generation unit for generating emotion parameters from the input voice, and information on emotions corresponding to the parameters generated by the parameter generation unit. A display means for displaying; an input operation means for instructing a user operation; a storage means for storing data; and a call means for making a call. The voice input means inputs voice during a call by the call means. The parameter generation means generates emotion parameters during the call from the input voice during the call, and the storage means generates the emotion parameters from the voice during the call in the order in which the parameter generation means generates the emotion parameters. in association, stores parameters of the emotion during a call generated by said parameter generating means, the input operation unit, said emotion Nitsu It instructs the display mode or the output mode of the call result information about the emotional information of Te, the display means, based on the order of generating the parameter, before SL after the call is terminated by the calling means, the input operation unit The information about the emotion corresponding to the parameter stored in the storage means is displayed in accordance with the display form of the information about the emotion or the output form of the call result of the information about the emotion .

According to this configuration, the emotion of the person who uttered the voice is directly shown in real time, so that another expression means that an expression that the user does not want to express can be directly transmitted to the other party. Further, according to the information processing terminal of the present invention, it is possible to look back later and grasp the overall picture of the call status. In addition, since the emotion of the other party can be read in time series, it becomes easier to grasp the other party's emotion, and as a result, smooth communication can be realized. In addition, since the call partner or the emotion of the person in the call can be confirmed every time the call ends, smooth communication can be realized. In addition, since it is possible to display the call partner or the person's emotion in each call in time series, it is possible to more easily recall the contents of the conversation made in each call.

The information processing terminal of the present invention, the display means displays a variety of information, said storage means, for each call by the call unit, stores the parameters of the emotion, the display means, call history when displaying, for each call item in the call history, including those that display information about emotion corresponding to a parameter stored in the storage means.

  According to this configuration, since the call partner or the person's emotion in each call can be displayed in the call history item, the contents of the conversation made in each call can be easily recalled.

The information processing terminal of the present invention, the display means displays a variety of information, said storage means, for each call by the call unit, stores the parameters of the emotion, the display means, the storage The information about the emotion corresponding to the specific parameter among the emotions corresponding to the parameter stored in the means is displayed in time series.

  According to this configuration, since the call partner or the person's emotion in each call can be displayed in time series in the call history item, the contents of the conversation made in each call can be recalled more easily.

Further, in the information processing terminal according to the present invention, the display means stores information about emotions of each of the callers, one of the callers, or the entire callers corresponding to the parameters stored in the storage means. Includes what to display .

  According to this configuration, it is possible to visually recognize the temporal change of emotion during the incoming call, the temporal change of emotion during the incoming call of the other party, and the temporal change of emotion as the whole conversation. Moreover, the content of the whole conversation can be recalled from the information about the emotions arranged in time series after the call.

In the information processing terminal according to the present invention, the storage unit stores face image information in association with a person who has captured the face image, and the display unit on the other hand, or the information for both parties overall emotion, including those for displaying the face image of the party read out from said memory means image processing to.

  According to this configuration, the user of the mobile phone can easily visually recognize who the called party is without having to confirm the display of the name of the called party. It is possible to easily distinguish the face image in the sequence order and the face image in the time sequence order of the image associated with the partner.

An information processing terminal according to the present invention is an information processing terminal comprising voice input means for inputting voice, parameter generation means for generating emotion parameters from the input voice, and notification means for notifying various information. In addition, an emotion specifying unit that specifies an emotion represented by a characteristic parameter among the generated parameters is provided, and the display unit includes a unit that displays information about the specified emotion.

  According to this configuration, by identifying a characteristic emotion from the input voice, it is possible to prevent the emotion of the person who emitted the voice from being erroneously read as another emotion.

  The information processing terminal according to the present invention further includes storage means for storing data, the storage means stores emotion parameters generated by the parameter generation means, and the emotion identification means includes the stored parameters. Including those that specify emotions represented by characteristic parameters.

  According to this configuration, by storing the emotion parameter generated from the input voice, it is possible to specify the characteristic parameter at any time while reading the stored emotion parameter.

  The information processing terminal according to the present invention includes a calling unit that performs a call, and the voice input unit inputs a voice during a call by the calling unit.

  According to this configuration, the voice can be easily input to the information processing terminal by inputting the voice during the call.

  The information processing terminal of the present invention includes a voice recording unit that records a voice during a call by the calling unit, and the voice input unit inputs a voice recorded by the voice recording unit.

  According to this configuration, the mobile phone user can determine not only the fact that there is a recorded voice but also the content and urgency of the recorded voice at the same time. It becomes possible to take communication.

The information processing terminal of the present invention, the display means displays a variety of information, said storage means, a sentiment that the identified stored for each call, if the display means displays the call history And displaying information about the identified emotion according to the call history.

  In the information processing terminal according to the present invention, the storage unit stores a part of the voice reflecting the specified emotion, and the display unit has the same time as the specified emotion according to the call history. Including the part of the voice.

  According to this configuration, when the call history is confirmed, it is possible to confirm the emotion of the caller in the call by displaying the characteristic emotion at the time of each call in the call history. it can.

Further, the information processing terminal, the display means of the present invention, calls to display a variety of information, said storage means stores the emotion the identified for each caller, the display means is stored in the storage means When the personal information of the person is displayed, the information about the specified emotion is displayed in accordance with the personal information.

  According to this configuration, when the personal information is confirmed, by displaying a characteristic emotion at the time of a call with the individual, it is possible to determine what kind of emotion the individual was in the call with the information processing terminal user. Can be confirmed.

The information processing terminal of the present invention, the display means displays a variety of information, said storage means, said identified in correspondence to a date call emotions stored, the display means displays a calendar In some cases, information about the identified emotion is displayed in accordance with the calendar date.

  According to this configuration, it is possible to easily look back on the content of a call made on the corresponding date by displaying the emotions identified from the call for each date.

The information processing terminal of the present invention, the display means displays a variety of information, said storage means stores an image corresponding to the emotion, the display means is read out from said memory means, said Includes those that display images corresponding to the identified emotions.

  According to this configuration, by notifying a characteristic emotion by an image, the information processing terminal user who sees the image can easily confirm the emotion.

  In the information processing terminal of the present invention, the storage unit stores the appearance frequency of the emotion specified by the emotion specifying unit, and the display unit reads out the storage unit according to the appearance frequency of the emotion. And displaying an image corresponding to the identified emotion.

  According to this configuration, by changing the image corresponding to the characteristic emotion according to the appearance frequency of the emotion and displaying it, it is possible to enhance the entertainment when the emotion notification is executed, and to send the emotion notification over a long period of time. I can enjoy it.

  The information processing terminal according to the present invention includes a receiving unit that receives data from a server that records an image corresponding to an emotion, and the storage unit stores an image corresponding to the received emotion.

According to this configuration, by receiving the emotion notification operation by the display hands stage from a server provided on the information processing terminal outside, you can add what easily new emotions notification operation and enjoy the emotion notification for a long time be able to.

  The information processing terminal according to the present invention includes an information processing terminal in which the emotion specifying unit specifies an emotion having the highest parameter value among the emotion parameters stored by the storage unit.

  According to this configuration, the emotion with the highest numerical value from the beginning to the end of the input speech is set as a characteristic emotion, and the emotion is identified with emphasis on the content that is strongly impressed during the conversation. can do.

  In the information processing terminal according to the present invention, the emotion specifying unit specifies the emotion having the highest sum of the numerical values of the parameters generated from the start to the end of the voice among the emotion parameters stored by the storage unit. including.

  According to this configuration, it is possible to identify an emotion with an emphasis on the content of the entire conversation by making the emotion having the largest sum total from the beginning to the end of the voice a characteristic emotion.

  The information processing terminal according to the present invention includes an information processing terminal in which the emotion specifying unit specifies the emotion having the highest numerical value of the parameter generated before the end of the voice among the emotion parameters stored by the storage unit.

  According to this configuration, it is possible to identify an emotion with emphasis on the afterglow of the conversation by making the emotion having the largest numerical value at the very end of the voice a characteristic emotion.

  Moreover, the information processing terminal of this invention contains what is provided with the transmission means which transmits the specified emotion to another information processing terminal.

  According to this configuration, by transmitting characteristic emotion information to another information processing terminal that cannot estimate the emotion of the person who uttered the sound from the audio signal, the other information processing terminal user can also receive the information. Like the processing terminal user, it is possible to easily determine the emotion of the person who uttered the voice.

  According to the information processing terminal of the present invention, it is possible to express, by using another expression means, that the emotion of the person who uttered the voice is directly shown in real time, so that facial expressions that the user does not want to express can be directly transmitted to the other party.

  Further, according to the information processing terminal of the present invention, it is possible to look back later and grasp the overall picture of the call status.

  Hereinafter, the form of an information processing terminal for implementing the present invention will be described in detail by taking a mobile phone as an example.

(First embodiment)
A mobile phone according to a first embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 shows a configuration diagram of a mobile phone according to a first embodiment of the present invention. The mobile phone according to the first embodiment of the present invention includes a call device 10 that controls a voice call and the emotion of the mobile phone user, the emotion of the other party of the call based on the voice during the call input from the call device 10 Emotion notification device 20 that identifies both emotions, and a device that notifies the mobile terminal user of the emotion identified by emotion notification device 20, such as display unit 30, LED 40, vibration 50, speaker 80, or scent generating device 90 , And.

  In the communication device 10, the audio signal output unit 102 emotionally receives an audio signal received from the other party via the communication unit 101 that transmits and receives an audio signal, or an audio signal of the mobile phone user collected by the microphone 104. Output to the notification device 20. The audio signal output unit 102 outputs the voice received from the other party to the speaker 103, and outputs the call voice of the mobile phone user collected by the microphone 104 to the communication unit 101.

  In the emotion notification device 20, when the emotion estimation unit 201 is notified from the communication unit 101 of the call device 10 that the call has started, the emotion estimation unit 201 transmits the emotion of the person who uttered the voice from the voice signal input from the call device 10. presume. At this time, the voice signal received from the other party, the voice signal of the mobile phone user, or both voice signals are input to the emotion estimation unit 201. The mobile phone user may select an audio signal input from the call device 10 or may be set to switch an audio signal input from the call device 10 at regular intervals.

  The emotion estimation unit 201 estimates the emotion of the person who has emitted the voice from the voice signal input from the call device 10. FIG. 2 shows an example of emotion estimation by the mobile phone according to the first embodiment of the present invention. The emotion estimation unit 201, for example, for each factor of emotion information composed of affection, joy, anger, sadness, and neutral shown in FIG. None, 1: Emotional weakness, 2: Emotional strong) is continuously estimated from the start of the call to the end of the call, and each estimated value is sequentially output to the emotion accumulation unit 202.

  The emotion accumulation unit 202 accumulates the numerical value for each factor input from the emotion estimation unit 201 in correspondence with the input time or order.

  The emotion identification unit 203 reads the numerical value for each factor accumulated from the start of the call to the end of the call in the emotion accumulation unit 202, identifies the characteristic emotion from the read numerical value, and the characteristic emotion Is output to the emotion notification operation instruction unit 205. By identifying the emotion that had the largest number from the beginning of the call to the end of the call as a characteristic emotion, it is possible to identify emotions with emphasis on strong and memorable content during the call, In addition, by setting the emotion with the largest total from the start of the call to the end of the call as the characteristic emotion, the emotion can be specified with emphasis on the content of the entire call. By making the greatest emotion a characteristic emotion, it is possible to identify the emotion with emphasis on the afterglow of the conversation.

  Also, the emotion identification unit 203 clips the voice for a certain period (for example, 2 seconds) reflecting the identified characteristic emotion, and describes the clipped voice in the call history when the call history is later reviewed. In addition, it is stored in the emotion storage unit 202 as characteristic audio information.

  The emotion identifying unit 203 continuously inputs characteristic numerical values for each factor that the emotion estimating unit 201 sequentially outputs without using the emotion accumulation unit 202 to continuously identify characteristic emotions. May be. Thereby, characteristic emotions can be identified in synchronization with the call.

  When a characteristic emotion is input, the emotion notification operation instruction unit 205 responds to the characteristic emotion from the emotion content storage unit 204 that stores the emotion notification operation by a device such as the display unit 30, the LED 40, or the vibration 50. The emotion notification operation is read and an operation instruction is output to these devices. A device preset by the mobile phone user may be made to execute the emotion notification operation, or a plurality of devices may be made to simultaneously execute the emotion notification operation to enhance the expressiveness of the emotion notification. . Hereinafter, an operation example of the emotion notification operation of the display unit 30 and the LED 40 will be described with reference to the drawings.

  FIG. 3 shows an image display example of the display unit of the mobile phone according to the first embodiment of the present invention. When the emotion notification operation is executed by the display unit 30, the emotion notification operation instruction unit 205 reads an image corresponding to the characteristic emotion from the emotion content storage unit 204 when a characteristic emotion is input, and the read image Is output to the display unit 30. When the display unit 30 displays the image shown in FIG. 3 corresponding to the characteristic emotion, the user of the mobile phone can easily determine the emotion of the person who uttered the sound visually. The display unit 30 is a sub liquid crystal display unit (which has a smaller display area than the main liquid crystal display unit provided in recent mobile phones. It is often provided on the back of the main liquid crystal display unit of the mobile phone, and is also referred to as a rear liquid crystal. Included). As another display example by the display unit 30, the background color of information displayed on the display unit 30 is changed according to emotions, or a series of pictograms, images, pictograms, etc. are continuously displayed (motion Graphics), pre-captured images and stored images are displayed.

  FIG. 4 shows a lighting example of the LED of the mobile phone according to the first embodiment of the present invention. When the emotion notification operation is executed by the LED 40, the emotion notification operation instruction unit 205 reads color information corresponding to the characteristic emotion from the emotion content storage unit 204 when the characteristic emotion is input, and uses the read color. The LED 40 is instructed to emit light. The LED 40 emits light in the color shown in FIG. 4 corresponding to the characteristic emotion, so that the mobile phone user can easily determine the emotion of the person who has uttered the voice visually. Note that these LEDs also include backlights for operation keys provided in the mobile phone, and the lighting method of these LEDs is the color or intensity of lighting light, lighting range, flashing rhythm (depending on emotion). The flashing interval may be changed, or the sound output from the speaker 80 described later or flashing in accordance with the vibration of the vibration described later may be changed. In addition, when there are a plurality of LEDs, each LED may be lit in a different color, or a plurality of LEDs are regarded as dots and characters or marks expressing emotions are expressed by the dots. May be.

  When the emotion notification operation is executed by the vibration 50, the emotion notification operation instruction unit 205 reads and reads the vibration information of the vibration corresponding to the characteristic emotion from the emotion content storage unit 204 when the characteristic emotion is input. The vibration 50 is instructed to vibrate at the vibration frequency and vibration strength included in the vibration information. The vibration 50 vibrates at a vibration frequency and intensity corresponding to characteristic emotions (for example, if the fluctuation of emotion is small, the vibration intensity is reduced, and if it is large, the vibration intensity is increased. ), The mobile phone user can easily determine the emotion of the person who uttered the voice. The vibration 50 may vibrate according to the sound output from the speaker 80 described later. As a method of combining the vibration generated by the vibration 50 and the sound output from the speaker 80, for example, if the fluctuation of emotion is small, only the vibration 50 is vibrated with a weak vibration, and if the fluctuation is large, the frequency corresponds to the sound frequency. For example, it may vibrate at a vibration frequency and with a strong vibration intensity.

  When the emotion notification operation is executed by the speaker 80, the emotion notification operation instruction unit 205 reads the sound information corresponding to the characteristic emotion from the emotion content storage unit 204 when the characteristic emotion is input, and the read sound The speaker 80 is instructed to output information. The speaker 80 outputs sound information corresponding to characteristic emotions (for example, different music or melody is set for each emotion. As this melody, a ringing melody that the mobile phone rings when receiving an incoming call is used. The mobile phone user can easily determine the emotion of the person who uttered the sound easily. In addition, although the configuration that sets different music and melody for each emotion was described, it is possible to set music and melody common to multiple emotions and change the tone of the music and melody for each emotion. good.

  When the emotion notification operation is executed by the scent generating device 90, the emotion notification operation instruction unit 205 reads out and reads the scent information corresponding to the characteristic emotion from the emotion content storage unit 204 when the characteristic emotion is input. The scent generating device 90 is instructed to emit scent corresponding to the scent information. When the scent generating device 90 produces a fragrance corresponding to a characteristic emotion, the mobile phone user can easily determine the emotion of the person who uttered the sound in an olfactory manner. In addition, although the structure which makes a different fragrance produce for every emotion may be sufficient, you may set so that the fragrance common to a several emotion may be set and the density | concentration of the fragrance may be changed for every emotion.

  The mobile phone of the present invention may be configured to use any one of devices such as the display unit 30, the LED 40, the vibration 50, the speaker 80, or the scent generating device 90, or may be configured to operate in combination. There may be.

  The emotion notification operation instruction unit 205 instructs the mail transmission unit 60 of FIG. 1 to transmit an emotion notification operation corresponding to the characteristic emotion read from the emotion content storage unit 204 to another mobile phone. By sending emotion notification operation information corresponding to characteristic emotions by e-mail to other mobile phones that are not equipped with the emotion notification device 20 and cannot estimate the emotion of the person who uttered the voice from the voice signal. Other mobile phone users can easily determine the emotion of the person who uttered the voice, as with the mobile terminal user. The mobile terminal user may send an email after confirming the emotion notification operation, or may automatically send it to the mobile phone of the other party.

  Further, the emotion notification operation instruction unit 205 specifies the call history 701 recorded in the storage unit 70 of FIG. 1 and indicating a call state such as a call partner, a call time, and a call time in a past call by the emotion specifying unit 203. The recording unit 70 is instructed to record the characteristic emotion and the call state in the call specifying the emotion. After recording the call history 701, when the mobile phone user displays the call history 701 on the display unit 30 and confirms the emotion recorded in the call history, it is recorded in the call history from the emotion content storage unit 204. The emotion notification operation corresponding to the emotion is read, and an operation instruction is output to these devices.

  Conventionally, even if the call history is recorded, it is difficult to check the past conversation status by checking the transmission / reception history like the mail transmission / reception history, and it is difficult to look back on the past call contents from the call history. However, according to the mobile phone of the first embodiment of the present invention, since the emotion specified from a certain call and the call state in the call are recorded in the call history, when the call history is confirmed, It is possible to confirm the feeling of the mobile phone user or the other party, and it is possible to easily look back on the content of the call in the call.

  When the emotion notification operation instruction unit 205 reads the emotion notification operation corresponding to the characteristic emotion from the emotion content storage unit 204, the emotion notification device 20 determines the internal frequency of the emotion notification device 20 and the like. Factors or the date and time when the emotion notification operation is read, the call time of the mobile phone, or the current location specified by the location information acquisition function provided in the mobile phone, etc. You may make it read the emotion notification operation | movement different from the emotion notification operation | movement normally read when a specific factor satisfy | fills specific conditions. At this time, in the case of the emotion notification frequency, the specific condition is that when the emotion specifying unit 203 specifies a certain number of emotions more than a certain number of times, in the case of a date, the date is a date such as New Year, Christmas, or daytime, night When the current position is coincident with or included in a time zone such as midnight, the current position is included or close to a place such as an amusement park or the sea.

  FIG. 5 shows categories of images displayed on the display unit of the mobile phone according to the first embodiment of the present invention. Category 1 is an image that is displayed on the display unit 30 without any conditions, category 2 is an image that is displayed when a certain emotion is identified a certain number of times, and category 3 is an emergency that is displayed when a certain emotion is identified. The categories 1, 2, and 3 are classified so that the display frequency decreases from category 1 to category 3 for each internal factor. Category 4 is an image displayed when the current position of the mobile terminal is included in the predetermined location, and category 5 includes the current position of the mobile terminal in the predetermined location, and the mobile phone The images are displayed when the call time exceeds a predetermined time. Categories 4 and 5 are mainly classified according to external factors. By setting the display frequency for each image in this manner, the mobile phone user can improve the entertainment when the mobile phone performs emotion notification, and can enjoy emotion notification over a long period of time.

  Further, the characteristic emotion specified by the emotion specifying unit 203 and the call state in the call specifying the emotion are recorded in the call history 701 of the storage unit 70, but read from the emotion content storage unit 204. You may make it record the emotion notification operation | movement corresponding to a characteristic emotion, and the call state in the telephone call which specified the emotion. With this configuration, when the mobile phone user confirms the emotion recorded in the call history 701, the emotion notification operation instruction unit 205 corresponds to the emotion recorded in the call history from the emotion content recording unit 204. Since the emotion notification operation can be read directly from the communication history without searching for the notification operation, the time required from the confirmation of the emotion recorded in the call history to the operation of the device can be reduced.

  Next, the flow of processing from voice input to emotion notification and recording in the mobile phone according to the first embodiment of the present invention will be described. FIG. 6 is a flowchart showing the flow of processing by the mobile phone according to the first embodiment of the present invention.

  When the emotion estimation unit 201 is notified by the communication unit 101 that a call has started (S600), an audio signal is input from the audio signal output unit 102 (S601a to S601c). At this time, the voice signal received from the other party, the voice signal of the mobile phone user, or both voice signals are input. The emotion estimation unit 201 estimates the emotion of the person who uttered the voice from the input voice signal (S602), and sequentially outputs a numerical value for each major factor of the estimated emotion information to the emotion accumulation unit 202. When the emotion estimation unit 201 is notified from the communication unit 101 that the call has ended (S603), the emotion identification unit 203 determines the characteristic from the numerical values for each factor from the start of the call to the end of the call in a call. The emotion is specified (S604), and the characteristic emotion is output to the emotion notification operation instruction unit 205. When a characteristic emotion is input, the emotion notification operation instruction unit 205 reads an emotion notification operation corresponding to the characteristic emotion from the emotion content storage unit 204, and operates on a device such as the display unit 30, the LED 40, or the vibration 50. An instruction is output (S605).

  If the mobile phone user instructs to record the characteristic emotion or the emotion notification action corresponding to the characteristic emotion and the call state in the call history 701 (S607, YES), the characteristic is recorded (S608). If an instruction to send an emotion notification operation corresponding to a specific emotion is sent to another mobile phone (S609), the emotion notification operation information is sent to the other mobile phone (S610).

  In the flowchart of FIG. 6, the cellular phone according to the first embodiment of the present invention having the configuration shown in FIG. 1 performs the process of estimating the emotion of the person who uttered the voice from the voice signal input during the call. A process for estimating the emotion of the person who uttered the voice from the voice signal input by the above means is also conceivable. Here, as an example of another means, an answering machine function of a recent mobile phone (when a mobile phone user cannot respond to an incoming voice call, the mobile phone automatically responds to the incoming voice call and In the following explanation, it is assumed that the voice of the other party is stored in the storage device of the mobile phone), and the emotion is estimated from the voice signal recorded by the answering machine function. Processing to be performed will be described. FIG. 14 shows a configuration diagram of another example of the mobile phone according to the first embodiment of the present invention, and FIG. 15 shows a flowchart showing another example of the flow of processing by the mobile phone according to the first embodiment of the present invention. . 14 and 15, the functions and processes of the respective units to which the same reference numerals as those in FIGS. 1 and 6 are assigned are the same as described above, and thus description thereof is omitted.

  14 is obtained by adding a guidance sound storage unit 105 and a voice storage unit 106 to the configuration of FIG. The guidance sound storage unit 105 indicates that the mobile phone user cannot respond to an incoming voice call when the mobile device automatically responds to the incoming voice call when the mobile phone user cannot respond to the incoming voice call. A voice signal for outputting a voice is stored. The voice storage unit 106 stores a voice signal received from the call partner after the call device 10 automatically responds to the incoming voice call.

  Next, the flow of processing for notifying emotions from voices recorded using the answering machine function in the mobile phone according to the first embodiment of the present invention will be described with reference to FIG.

  When receiving an incoming voice call (step S1401), the communication unit 101 determines whether the mobile phone is set to use the answering machine function (step S1402). If the communication unit 101 determines that the answering machine function is set to be used (in the case of a mobile phone that uses the answering machine function when it does not respond to the incoming call for a predetermined time from the start of reception of the incoming call, In addition, a criterion of whether or not the time measured from the start of reception of incoming calls exceeds a predetermined time is added to the criterion of step S1402.Is it set to use the answering machine function as in this case? In response to the incoming voice call, the voice signal output unit 102 outputs the voice signal read from the guidance sound storage unit 105 as the transmission voice (S1403). After step S1403, the audio signal output unit 102 stores the received voice in the voice storage unit 106 when the communication unit 101 inputs the received voice from the other party. The emotion estimation unit 201 estimates the emotion of the person who has emitted the voice from the voice signal stored in the voice storage unit 106 input via the voice signal output unit 102. Thereafter, when the call ends in the process of step S603, the voice signal output unit 102 ends the process of storing the received voice in the voice storage unit 106, and performs various emotion notification operations based on the emotion estimated by the emotion estimation unit 201. Will cause the device to run.

  In the conventional answering machine function, only the information that the answering machine was recorded was notified by the notification means (display unit, LED, vibration, etc.), so the contents and urgency of the recorded voice itself are recorded. I didn't know until I played the voice. For this reason, mobile phone users knew the fact that there was recorded audio, but it was not possible to determine that the content was urgent, and as a result, the recorded audio was not played back. Communication with the other party who was recording the voice was interrupted. As with the mobile phone of the first embodiment of the present invention, the voice of the other party is recorded using the answering machine function, the emotion is estimated from the voice signal recorded by the answering machine function, and information about the emotion is recorded The mobile phone user can determine not only the fact that there is a recorded voice but also the content and urgency of the recorded voice at the same time. It becomes possible to take communication. In particular, such a configuration cannot respond to an incoming call for some reason, but it can be known in a relatively short time that there is a voice being recorded or a voice that has been recorded (for example, in a train). Smooth communication can be realized more effectively when it is a case or during a meeting.

  Next, the flow of processing for confirming a call history in the mobile phone according to the first embodiment of the present invention will be described. FIG. 7 shows a call history confirmation flowchart in the mobile phone of the first embodiment of the present invention, and FIG. 8 shows a call history display example of the display unit in the mobile phone of the first embodiment of the present invention.

  The mobile phone user presses the call history confirmation button in FIG. 8A to display the call history shown in FIG. 8B on the display unit 30 and selects one in the call history (S700). When it is attempted to confirm the emotion at the time of the call (S701, YES), the emotion notification operation instruction unit 205 reads the characteristic emotion recorded in the call history 701, and the characteristic emotion is stored in the emotion content storage unit 204. Is read and the display unit 30 is instructed to display the image (S702), and the display unit 30 outputs the image shown in FIG. 8D (S703).

  In this way, by recording the emotion identified from the call and the call state in that call in the call history, when the call history is confirmed, what kind of emotion the mobile phone user or the other party is in the call? It is possible to check the contents of the call in the call.

  In the mobile phone according to the first embodiment of the present invention, it has been described that the identified characteristic emotion is confirmed using the call history. However, the phone is not limited to the call history and is used as a function of the mobile phone. You may make it confirm using a book or a scheduler. Hereinafter, the flow of the telephone directory confirmation process and the scheduler confirmation process in the cellular phone according to the first embodiment of the present invention will be described.

  A flow of a telephone directory confirmation process in the mobile phone according to the first embodiment of the present invention will be described. FIG. 9 shows a flowchart of phone book confirmation in the mobile phone according to the first embodiment of the present invention, and FIG. 10 shows a phone book display example of the display unit in the mobile phone according to the first embodiment of the present invention.

  In step 608 shown in FIG. 6, after the characteristic emotion or the emotion notification operation corresponding to the characteristic emotion is recorded in the telephone directory 702 together with the call time as personal information of the other party registered in the telephone directory, the cellular phone is used. The person presses the telephone book selection button in FIG. 10A to display the telephone book on the display unit 30 and selects one person in the telephone book as shown in FIG. 10C (S900). If the emotion at the time of the call is to be confirmed (S901, YES), the emotion notification operation instruction unit 205 reads the characteristic emotion recorded in the telephone directory 702 (S902), and the characteristic is stored in the emotion content storage unit 204. The image corresponding to the emotion is read out and the display unit 30 is instructed to display the image in time series based on the call time (S903). The display unit 30 is shown in FIG. And it outputs the image (S904).

  In this way, by recording the emotion identified from the call in the phone book as the personal information of the other party registered in the phone book, when the phone book is confirmed, the emotion of the person registered in the phone book is It is possible to easily confirm how the call has changed due to a call with the user.

  The flow of the scheduler confirmation process in the mobile phone according to the first embodiment of the present invention will be described. FIG. 11 shows a flowchart of scheduler confirmation in the mobile phone according to the first embodiment of the present invention, and FIG. 12 shows a scheduler display example of the display unit in the mobile phone according to the first embodiment of the present invention.

  The mobile phone user presses the scheduler selection button in FIG. 12A to display the scheduler on the display unit 30 and selects one date on the calendar as shown in FIG. 12B (S1100). When an attempt is made to confirm the emotion during the date call (S1101, YES), the emotion notification operation instruction unit 205 reads the characteristic emotion recorded in the call history 701 at the corresponding date (S1102), and the emotion content storage unit 204 Then, an image corresponding to the characteristic emotion is read out and the display unit 30 is instructed to display the image for each call partner (S1103), and the display unit 30 outputs the image shown in FIG. S1104).

  In this way, by reading out emotions identified from a call for each date, it is possible to easily look back on the content of a call made on that date.

  In the mobile phone according to the first embodiment of the present invention, the emotion read by using the call history, the phone book or the scheduler is displayed on the display unit for confirmation, but the LED or vibration is activated. You may make it confirm.

(Second Embodiment)
A mobile phone according to a second embodiment of the present invention will be described in detail with reference to the drawings. FIG. 13 shows a configuration diagram of a system including the mobile phone according to the second embodiment of the present invention. The cellular phone according to the second embodiment of the present invention includes an emotion content storage server 2 that stores emotion notification operations by devices such as the display unit 30, the LED 40, or the vibration 50 in addition to the configuration of the cellular phone according to the first embodiment. An emotion content acquisition unit 206 that acquires emotion content is included. In addition, the thing of the same code | symbol as FIG. 1 has shown the same structure, and abbreviate | omits description.

  The emotion content acquisition unit 206 outputs the emotion notification operation acquired from the emotion content storage server 2 to the emotion content storage unit 204. The emotion content storage unit 204 newly adds and stores the emotion notification operation input from the emotion content acquisition unit 206.

  By acquiring the emotion notification operation from a server provided outside the mobile phone, the mobile phone user can easily add a new emotion notification operation using a display unit, LED, or vibration, etc. You can enjoy feelings notifications.

(Third embodiment)
FIG. 16 schematically shows the configuration of the mobile phone according to the third embodiment of the present invention. Many parts of the configuration of the cellular phone 11 has the same configuration as the cellular phone 1 of the first embodiment described, will be mainly described locations configuration is different in the following.

  In the emotion notification device 20 of the mobile phone 11, the voice input switching control unit 200 selectively controls the voice input to be input to the emotion estimation unit 201. Depending on the operation of the input operation unit 95 of the mobile phone user, setting is made as to whether the received voice, the transmitted voice, or both voices are input, and the setting is stored in the control setting storage unit 206. . When the voice input switching control unit 200 is notified by the communication unit 101 of the inputted call device 10 that the call has started, the voice input switching control unit 200 selectively selects the voice signal input from the call device 10 based on the voice input setting. Output to. The output voice signal is input to the emotion estimation unit 201, and the emotion of the person who has emitted the voice is estimated.

  The emotion accumulation unit 202 temporarily accumulates the estimated emotion.

  When the emotion estimation unit 203 outputs a characteristic emotion from the accumulated estimated emotion, the emotion estimation unit 203 stores it as the emotion data 703 in the storage unit 70 in association with the time of the characteristic emotion call.

  The emotion content storage unit 204 stores output control data 704 and image data of emotion notification operations corresponding to characteristic emotions from the emotion content storage unit 204 that stores emotion notification operations by devices such as the display unit 30, the LED 40, or the vibration 50. 705 is stored.

  When the call history data 701 stored in the storage unit 70 is referred to, the emotion notification operation instruction unit 205 reads the output control data 704 and the image data 705 with reference to the emotion data 703, and sends the device to the output control unit 80. Instructs the user to output the emotion notification operation.

  The output control unit 80 performs operation control on these devices.

  In addition, regarding the output of the emotion notification operation to a device such as the display unit 30, the LED 40, or the vibration 50, the emotion notification operation may be executed by one device set by the mobile phone user using the input operation unit 95. A plurality of devices may simultaneously execute the emotion notification operation. When the operation of a plurality of devices is performed, for example, the expressiveness of emotion notification is enhanced, and the mobile phone user has an effect that emotions during a call can be transmitted more effectively. Further, although the display unit 30 and the LED 40 are different from each other in output method, they are both devices that perform visual output, so the LED 40 is preferably located near the display unit 30. If the display unit 30 and the LED 40 enter the visual field at the same time when they are located close to each other, the simultaneous output to the display unit 30 and the LED 40 has an effect that the visual stimulus becomes larger than the output of each independently.

  The output control data 704 includes emission color control data of the display unit 30, blinking pattern control data of the LED 40, vibration pattern control data of the vibration 50, and the like. The output control data is not limited to these. For example, when an audio output device is provided as a device, the audio output device can be controlled if the corresponding audio pattern control data is provided.

  The emotion content storage unit 204 stores image data 705 together with output control data 704. The image data 705 is data for displaying an image on the display unit 30 in particular. The image data 705 includes bitmap format, still image format image data such as GIF format and JPEG format, and moving image format image data such as Flash format and Motion JPEG format.

  Here, the configuration of data stored in the storage unit 70 and the emotion content storage unit 204, and the display of each data in the correlation and the call history display of each data will be described.

  The call history data 701 includes at least (for example, timed by a time measuring unit 208 (not shown)) time information such as a call time, time information such as a call time, and number information specified by a call partner such as a telephone number. The phone book data 703 includes at least number information for identifying a call partner such as a telephone number, call partner information such as a name and a nickname associated with the number information, and further includes call partner related information (associated with a call partner). Character, image, sound, etc.) may be added. When the call history is displayed, the call history data 701 and the phone book data are associated with each other by the number information specified by the other party such as a telephone number, and the time when the call was made and the communication partner information are displayed. In addition, in the call history display with the call partner without the phone book data, the call history data 701 and the phone book data 703 cannot be associated with each other, so that the call partner such as the call time and the phone number can be specified. Number information to be displayed.

  Emotion data 703 is composed of at least the time when the call was made, the type of input voice (whether the voice is the other party's voice or one's own voice), and characteristic emotions. Moreover, the emotion data 703 may further include an index that can identify which time zone within the call time of a call at a certain call time. The image data 705 includes at least characteristic emotion and image data associated with the characteristic emotion. When the call history is displayed, the call history data 701, the emotion data 703, and the image data 705 are associated with each other depending on the time of the call, and at least the image data associated with the time of the call and the characteristic emotion. Is displayed.

  Moreover, since the emotion data 705 and the phone book data 703 are associated with each other as described above, for example, when the call history is displayed, the time when the call was made, the communication partner information, and the characteristic emotion can be displayed together. . Other combinations of information can also be displayed by the association between the data.

  When settings for emotion display are stored in the control setting storage unit 206, emotions are displayed as follows after the call ends or according to a call history browsing instruction of the mobile phone user through the input operation unit 95. The emotion notification operation control unit 205 associates the call history data 701 with the output control data 704 based on the stored emotion data 703, instructs the output control unit 80 to output, and the output control unit 207 performs emotion display control on the device. Do.

Next, emotion estimation by the mobile phone user and initial setting of the estimation result display will be described with reference to FIG.
The mobile phone user performs input for emotion estimation by an input operation by the input operation unit 95 ( S221 , Yes). When emotion estimation is not performed ( S221 , No), the initial setting is terminated. Next, for the input voice that is the target of emotion estimation, either the input of the transmitted voice, the input of the received voice only, the mixed input of the voice of the transmitted and received voices, or the input of the voice of the transmitted and received voices individually (S 222 ), and subsequently, regarding the display form of the emotion estimation result, the icon image is displayed in time series, the representative icon image is displayed, the moving image is displayed on a large screen, and the moving image is reduced in size. Any one of the screens to be added is selected ( S223 ). The initial setting is completed by performing the above.

Next, an emotion identification procedure during a call will be described with reference to the flowchart of FIG.
When the call is started, the time measuring unit 208 operates (S 231 ). The voice input switching unit 200 switches the voice input to the emotion estimation unit 201 (S 232 ). The time measuring unit 208 starts measuring the unit time ( S233 ). Once it passed the unit time (S 234, Yes), the emotion estimation unit 201 outputs the parameters of each emotion in the unit time (S 235). The emotion accumulating unit 202 accumulates the output emotion parameter of the unit time (S 236 ). The emotion identifying unit 203 identifies the characteristic emotion in the unit time from the emotion parameter of the unit time ( S237 ). The emotion accumulation unit 202 accumulates the characteristic emotions in the specified unit time (S 238 ). If the call has not ended (S 239 , No), the process returns to S 233 to start emotion estimation for the next unit time call. If the call has ended ( S239 , Yes), emotion estimation ends. By specifying emotions by this procedure, emotion parameters and characteristic emotions are accumulated in the emotion accumulation unit 203 in a time series obtained by dividing a call into unit times.

Hereinafter, the procedure for determining the emotion will be specifically described with reference to the drawings.
FIG. 20 is a graph in which emotion strength for each emotion type output by the emotion identification unit 203 is continuously represented in a unit time series. In FIG. 20, the horizontal axis represents time, and the vertical axis represents the intensity of each emotion type. The emotion identification unit 203 does not extract one specific emotion, but outputs strength (an index value to the extent that strong emotional expression is performed) for each of the emotion types described above, such as affection, joy, anger, sadness, and neutral. FIG. 21 is a graph of the graph shown in FIG. For each unit time, the intensity for each emotion type in the unit time is indicated by a numerical value. Among emotion types of the unit time, an emotion showing the maximum value is extracted, and if the maximum value is equal to or greater than a predetermined threshold, the emotion is determined as the emotion of the unit time. For example, since the intensity of love is 80, joy is 30, anger is 20, sadness is 10, and neutral is 50 in the first unit time, the one that shows the maximum emotion in the first unit time is love and Become. Moreover, since the intensity of affection is 80 and exceeds the intensity threshold value 50, it is determined that the emotion in the first unit time is affection. In addition, since the intensity of love is 30, joy is 20, anger is 10, sadness is 10, and neutral is 10 in the 8th unit time, what shows the maximum emotion in the 8 unit time is love. . However, since the intensity of love is 30 and does not exceed the intensity threshold of 50, it is determined that the emotion in the eighth unit time is not judged. At the bottom of FIG. 21, the emotion types that have the maximum value in unit time are shown as the emotion types that are most characteristic in the unit time.

Next, the procedure for emotion determination after the call ends will be described with reference to the flowchart of FIG.
When the call ends, the call time is divided into sections by a predetermined number of divisions. Here, it is assumed that the number of divisions of the call time is N (N is a natural number). The call time is divided by the section division number N and the unit time to calculate the number of unit times per section (one section division band) (S191). The emotion parameters per unit time accumulated in the emotion accumulation unit 203 are reduced so as to belong to each time division zone. The emotion parameter per unit time made difficult for each time division zone is recalculated for each emotion in each time division zone, and the characteristic emotion in each time division zone is determined. The estimated emotion of the section determined as described above is stored in the storage unit 70 in association with the call time.

  Here, the emotion determination procedure will be described in detail with reference to the drawings. FIG. 22 shows an example of recalculating the emotion parameters per unit time shown in FIG. The number obtained by adding 1 to the integer value not exceeding the number obtained by dividing the call time by the section division number N and the unit time is defined as the number of unit times. In the example of FIG. 21, the number of unit times in the call time is 11, and the number of section divisions is 4, so the number of unit hours in one section is 3. Therefore, emotion parameters for the first to third unit times are included in the first interval. Similarly, a unit time parameter is entered in the second interval, and emotion parameters for the seventh to ninth unit times are entered in the third interval. For the fourth unit section, for example, the parameters of the 10th and 11th unit times may be included, or the 9th to 11th emotions may be included. Subsequently, the value of each emotion parameter in each section is added and accumulated. The determination of the emotion from the accumulated value of each emotion parameter is performed by the same method as the determination of the emotion in unit time.

  When the emotion is determined, output is performed on the device based on the output control data.

  Next, emotion output in the call history will be described. FIG. 23 shows an example of icons when the characteristic emotion data 703 stored in the storage unit 70 is graphically displayed on the display unit 30. The left column of FIG. 23 shows the types of emotions, and the right column shows icons corresponding to the types of emotions. Each icon is stored as image data 705 in the emotion content storage unit 204. By replacing the emotion of joy with, for example, an icon representing the expression of joy and displaying it graphically, the emotion can be intuitively recognized visually. In FIG. 23, neutral, joy, anger, surprise, and sadness are illustrated as emotions, but other emotions may be associated with icons as long as the emotion extraction specifying unit can extract them. Further, the icon to be associated is not limited to the illustrated one, and may be other image data stored in the emotion content storage unit 204.

  24 and 25 show examples of LED emission (flashing) patterns corresponding to the emission color and emotion intensity when the characteristic emotion data 703 stored in the storage unit 70 is output to the LED 40. FIG. The left column of FIG. 24 is the type of emotion, the right column is the emission color setting corresponding to the type of emotion, the left column of FIG. 25 is the intensity of emotion, and the right column is the LED ON / OFF time control setting. Each LED emission color and LED emission pattern are stored in the emotion content storage unit 204 as output control data 704. For example, by replacing the emotion of anger with a red LED emission color that represents anger and outputting it, the user of the mobile phone can intuitively recognize the emotion of anger from the emotion image of the red color. In FIG. 24, the emotions of neutral, joy, anger, surprise, and sadness are illustrated in association with light blue, pink, red, yellow, and blue, respectively. However, the emotion that can be extracted by the emotion extraction specifying unit 203 is illustrated. For example, other emotions may be associated with the LED emission color. The LED emission color to be associated is not limited to that shown in FIG. 24, but may be an LED emission color based on other output control data stored in the emotion content storage unit 204. In addition, by outputting that a certain emotion is strong (for example, when the emotion is angry, when the emotion is angry) by flashing the LED at high speed, the user of the mobile phone can intuitively understand that a certain emotion is strong by visual stimulation. The user of the mobile phone is weak in certain emotions by outputting that an emotion is weak (for example, if the emotion is anger, a little anger is moderate) by constantly lighting the LED. This can be intuitively recognized by visual stimulation. The LED light emission pattern to be associated is not limited to that shown in FIG. 25, but may be an LED light emission pattern based on other output control data stored in the emotion content storage unit 204. Further, in this embodiment, the LED 40 is exemplified as one that outputs a binary state of ON-OFF, but if an LED that outputs three or more states that can specify a plurality of ON states is used, the emotional strength is further increased. Can be output in a variety of ways, and the range of expressiveness of emotions by the light emission pattern of the LED can be expanded. Furthermore, the emotion intensity output by the light emission pattern can be used in combination with the emotion output by the light emission color of the LED 40, so that the range of expressiveness of the emotion can be further expanded, and the user of the mobile phone recognizes the emotion finely. can do.

[Incoming call output form 1]
A first output form of the incoming call history will be described with reference to the drawings.
FIG. 26 shows an example in which an incoming call history list is displayed on the display unit 30 of the mobile phone 11. One incoming call history display 301 in FIG. 26 includes an icon 302 representing an image corresponding to an emotion extracted from the received voice (partner), an incoming time 303, a call time 304, and a name 305 of the incoming party.

  With these displays as one incoming call history, two displays are displayed on the screen of the display unit 30 in FIG. When there are three or more incoming call history accumulations, another incoming call history can be displayed by operating the input operation unit 95. The number of displays per screen may be three or more, and the number of displays is not limited as long as the display does not impair the visibility of the incoming call history on the screen.

  The icons 302 are arranged from left to right in chronological order from the start of the call until the end of the call. By displaying the icons 302 arranged in time series, the user of the mobile phone can visually recognize the temporal change in the emotion of the other party who is receiving the call. In addition, the contents of the conversation can be recalled after a call by visual stimulation. Further, by displaying the incoming time 303 and the name 305 of the called party together, there is an effect that the user of the mobile phone can more surely recall the contents of the conversation, the flow of the conversation, and the impression of the other party during the conversation. .

  As the name 305 of the called party, the name registered in the phone book data 702 is displayed in association with the phone number by referring to the phone book data from the incoming phone number at the time of the incoming call. When the incoming telephone number is not registered, the incoming telephone number is displayed as it is instead of the name display (see the display of the other party's telephone number 306 in the display unit 33 in FIG. 26), or an indication that it is not registered is displayed. May be.

  Since the icon 302 and the partner's name 305 are personal information, the user of the mobile phone may want to keep the icon 302 and the partner's name 305 secret. In this case, the user of the mobile phone as the owner may make a setting for not displaying the icon 302 or the name 305 of the other party for the specific party and cancel the setting. As a setting for not displaying the icon 302 or the partner's name 305, for example, the icon 302 or the partner's name 305 is displayed as a blank, and replaced with another image or character string. In addition, incoming call history from a specific partner may be displayed in a predetermined character display color, background color, or the like. By displaying in this way, the user of the mobile phone can clearly distinguish the incoming call history of a specific partner from the incoming call history from other parties. In addition, it is possible to make it unclear who a specific person is to anyone other than the owner of the mobile phone, so that personal information can be prevented from leaking when the mobile phone is lost. Has an effect. In addition, the above-described effect can be obtained even if a setting is made so that a specific incoming call history display is not displayed regardless of the destination of the incoming call.

  The incoming call history may be extracted for each call partner and displayed on the screen. Further, the extracted histories may be rearranged in the order of the incoming time or the length of the call time and displayed on the screen. By extracting each call partner and displaying it on the screen, it is possible to easily recall when and what kind of conversation was made with respect to a conversation with a specific partner. Further, by arranging the incoming call history in the order of time or the length of the call time and displaying it on the screen, the user of the mobile phone can easily extract a desired incoming call history from many incoming calls.

  Moreover, although the output to the display part 30 was described in this output form, you may perform also the output by light emission of LED40 synchronizing. By combining the output to the display unit 30 and the output to the LED 40, the user of the mobile phone can recognize emotions more finely than when outputting only to the display unit 30.

[Outgoing call output form 2]
A second output form of the incoming call history will be described with reference to the drawings.
In the screen display of the incoming call history in the first output form, it is possible to display a more detailed call result for any one of a plurality of incoming call histories on the screen display. In FIG. 26, one incoming call history display 301 is displayed in reverse color in distinction from other incoming call history displays. When the call result display instruction key 307 is further selected from the display state by the input operation unit 95, The call result is displayed.

  FIG. 27 is an example in which the call result is displayed on the display unit 30. The difference from the first output form is that, in one call result display 321, the emotions extracted from the uttered voice (self) and the received voice (partner) are displayed from the left of the screen in order of the icons 302 of the partner and the partner. A history title (own display) 308 and a history title (display of the other party) 309 are displayed on the screen so that the point arranged on the right and displayed together with the icon 302 of the other party can be distinguished and recognized. The other screen displays are the same.

  In the second output form, since the change in the emotion of the other party is displayed in chronological order together with the temporal change in the emotion of the other party, the user of the mobile phone can display the side of the icon 302 along the side of the call. In addition to visually recognizing temporal changes in each emotion, the emotion at a certain point in the conversation can be visually recognized by the vertical arrangement of the icons 302. It has the effect of being able to recall the flow of the person, the impression of the other party and his / her psychology more reliably.

[Incoming history output form 3]
A third output form of the incoming call history will be described with reference to the drawings.
FIG. 28 is another example of the second output form in which the call result is displayed on the display unit 30. The difference from the second output mode is that only one emotion extracted from the uttered voice (self) is displayed on the screen by arranging icons 302 from the left to the right in the time series order in one call result display 331. It is a point.

  By displaying only the user's own icon 302 in time series, it is possible to clearly recognize the temporal change in emotion during the incoming call. In addition, the contents of the conversation can be visually recalled after a call from the icons 302 arranged in chronological order.

  FIG. 29 is another example of the second output form in which the call result is displayed on the display unit 30. The difference from the second output form is that, in one call result display 341, only emotions extracted from the received voice (the other party) are displayed on the screen by arranging icons 302 from the left to the right in the time series order. It is a point.

  By displaying only the partner's icon 302 in time-sequential order on the screen, it is possible to recognize clearly the temporal change of the emotion during the other party's incoming call. In addition, it is possible to recall the contents of a conversation after a call from icons arranged in time series.

  Further, the above-described display of the call result of only one person or only the other party may be switched between. By doing this, there is an effect that the time series display of the icon 302 in the same incoming call history can be mutually referred by switching.

[Outgoing call output form 4]
A fourth output form of incoming call display will be described with reference to the drawings.
FIG. 30 is another example of the second output form in which the call result is displayed on the display unit 30. The difference from the second output mode is that, in one call result display 351, the emotions extracted from the entire voice (self) and the received voice (partner) are arranged with icons 302 from the left to the right in order of time series. This is the point that is displayed on the screen.

  By displaying the icons 302 of the person and the other party on the screen in chronological order, it is possible to visually recognize the temporal change in emotion as the whole conversation. In addition, the contents of the entire conversation can be recalled after a call from the icons arranged in time series.

[Incoming call history output form 5]
A fifth output form of incoming call display will be described with reference to the drawings.
FIG. 31 is another example of the second output form in which the call result is displayed on the display unit 30. The difference from the second output form is that, in one call result display 361, the emotion extracted from the received voice (the other party) is displayed on the screen using an animation 310 that represents a video corresponding to the emotion. .

  By displaying the other party's emotion as the whole conversation on the screen as a moving image, it is possible to more visually recognize the emotion of the entire conversation than when the screen is displayed using a still image such as an icon. In addition, the content of the conversation can be recalled more clearly after the call by the video.

  In this output form, a video associated with the emotion extracted from the entire conversation of the other party is displayed on the screen. You may display the moving image matched with the emotion extracted from the whole. By displaying each moving image, the emotion of the entire conversation can be recognized more visually than the display using icons. In addition, the content of the conversation can be recalled more clearly after the call by the video.

[Incoming history output form 6]
A sixth output form of incoming call display will be described with reference to the drawings.
FIG. 32 is another example of the second output form in which the call result is displayed on the display unit 30. The difference from the second output form is that in the one call result display 371, the emotions extracted from the uttered voice (self) and the received voice (partner) are associated with each other instead of the icon 302. Are arranged in a time-series order display 311 and a time-series order display 312 of images associated with the other party, and are respectively displayed on the screen.

  Here, when viewing the display 312 of the images associated with the partner in chronological order, different images associated with the partner are arranged. Specifically, two types of images, an image 313 and an image 314, are arranged. In this output form, the image associated with the incoming call partner “Nomura” in FIG. 33A is an image when the emotion is neutral (image 313), and is stored in the storage unit 70. It is also used in 11 telephone directory displays. Image processing is not performed on the image 313. On the other hand, the image associated with the called party “Nomura” in FIG. 33B is an image when the emotion is sad (image 314), and the image processing unit 207 performs image processing on the image 313. Has been generated by In the present output form, the image processing that the image 314 has undergone is the superimposition of the auxiliary image 315 that represents tear drops on the image 313. Here, the auxiliary image 315 is an image associated with the emotion of sadness specified by the emotion specifying unit 203, and is stored in the emotion content storage unit 204. Therefore, in this output form, when displaying emotions, image processing is performed using a basic image and an auxiliary image to generate a display image. Therefore, an image associated with oneself or a partner expressing a large number of emotions is displayed. There is no need to store the data in the storage unit individually in advance, and the storage unit can be used effectively. On the other hand, an image that has been subjected to image processing may be stored in the storage unit. In this case, there is no need to perform image processing when displaying the screen, so that the screen can be displayed at high speed. As the image processing, in addition to the superimposition of the auxiliary image, the image associated with the other party may be deformed. For example, in order to express an angry emotion, if the image associated with the other party is a face image, there are image processing such as deformation that raises the eyebrows and eyes and color correction that makes the face color blush. The type of image processing is not limited as long as the image after image processing has an effect of recalling emotions.

  By arranging images associated with the user and the other party as the emotion screen display, the user of the mobile phone can visually determine who the called party is without having to confirm the display of the name 305 of the called party. In addition to being easily recognizable, it is possible to distinguish between the time series display 311 of the image associated with the user and the time series display 312 of the image associated with the partner without checking the history titles 308 and 309. be able to.

  In particular, when an image of the incoming call partner or his / her face is used as an image associated with the incoming call partner or himself / herself, there is an effect that emotions can be recognized more clearly. In addition, if the face image of the person or the other party who is having a conversation is used, the contents of the conversation can be recalled more clearly after the call.

  As described above, by using an image associated with an incoming call partner for the incoming call display, a different image is used for each incoming call partner, so that it is possible to visually identify the incoming call partner easily. Furthermore, the contents of the call can be clearly recalled from the incoming call history. Also, you can be more familiar with the other party.

  Note that, as in the third and fourth output forms, the image display may be a display of only oneself, a display of only the other party, or a display of the entire conversation in which the person and the other party are combined. The effects obtained at this time are the effects described in the third and fourth output forms in addition to the effects of the sixth embodiment.

(Fourth embodiment)
In the mobile phone according to the fourth embodiment of the present invention, the phone book update by associating the emotion data 705 and the phone book data 703 after the call in the mobile phone 11 will be described.

  If an image associated with the registrant is registered in the storage unit 70 for the registrant of the phone book data 703, the emotion notification operation instruction unit 205 performs image processing on the image associated with the registrant. The image processing unit 207 is instructed to do so. The image processing is processing for associating emotion with the original image associated with the registrant based on the emotion data 703. As the type of image processing, similar to the sixth output form of the third embodiment, the auxiliary image stored in the emotion content storage unit 204 is superimposed on the image associated with the registrant of the phone book, The registered image may be deformed, but the type of image processing is not limited as long as the image after image processing has an effect of recalling emotion.

  The image registered in the phone book may be updated every time the screen is displayed, or updated after the call and the updated image data may be stored in the storage unit. If the update is performed after a call, there is no need to perform image processing when the phone book is displayed on the screen, so that the screen can be displayed at high speed.

  As an image associated with the registrant of the phone book, an image stored in advance in the storage unit, an image captured by an imaging device, a data communication function, or an external storage medium is used. As the type of image, one that easily recalls the registrant from the image is preferable, and for example, a registrant's face image is suitable.

  FIG. 34 shows a normal phone book screen display (A) and a phone book screen display (B) of this embodiment for a phone book in which a registrant's face image is registered as an image associated with the registrant. It is shown. The difference between FIG. 34 (A) and FIG. 34 (B) is that the image associated with the registrant is image-processed based on the emotion determined in the latest call and displayed on the screen in FIG. 34 (B). Is a point. For example, in the registered image corresponding to the registrant “Nomura” in FIG. 34B, since the emotion of the registrant at the time of the last call is determined as sadness, image processing corresponding to sadness is performed. Is displayed on the screen.

  After making a call with the phone book registrant in this way, based on the emotions determined in the latest call for the image associated with the registrant along with the phone number and the name of the registrant in the phone book By performing image processing and updating, when the user of the mobile phone displays the phone book on the screen after referring to it, the phone book registrant and emotion are visually associated with each other to stimulate memory and register the phone book. The content of the previous call with the person can be easily recalled. This effect is the same as that the call history can be displayed on the screen and the contents of the call can be recalled. Also, if the user of the mobile phone refers to the phone book before the next call, the content of the previous call will be recalled, so the content of the conversation can be considered in advance before the call, It also has the effect of having a smooth conversation.

  The information processing terminal of the present invention can directly express in real time the emotion of the person who uttered the voice, and can express the expression that he / she does not want to express directly to the other party, It is useful for an information processing terminal that has an effect of being able to look back later and grasp the overall state of a call state, and that notifies the emotions of the person who utters the voice estimated from the voice.

1 is a configuration diagram of a mobile phone according to a first embodiment of the present invention. Example of emotion estimation by the mobile phone of the first embodiment of the present invention Image display example of display unit of mobile phone according to first embodiment of present invention Illumination example of LED of mobile phone according to first embodiment of the present invention Categories of images displayed on the display unit of the mobile phone according to the first embodiment of the present invention The flowchart which shows the flow of the process by the mobile telephone of 1st Embodiment of this invention. Flowchart of call history confirmation in the mobile phone of the first embodiment of the present invention Call history display example of the display unit in the mobile phone according to the first embodiment of the present invention Flowchart for confirming a phone book in the cellular phone according to the first embodiment of the present invention Example of phone book display on display unit in mobile phone of first embodiment of the present invention Flowchart of scheduler confirmation in the mobile phone of the first embodiment of the present invention Example of scheduler display of display unit in mobile phone according to first embodiment of present invention The block diagram of the system containing the mobile telephone of 2nd Embodiment of this invention The block diagram of the other example of the mobile telephone of 1st Embodiment of this invention The flowchart which shows another example of the flow of the process by the mobile telephone of 1st Embodiment of this invention. Configuration diagram of mobile phone of third embodiment of the present invention Flowchart for initial setting of emotion estimation by mobile phone user and display of the estimation result by the mobile phone according to the third embodiment of the present invention The flowchart which shows the procedure of the emotion specification during the telephone call by the mobile telephone of 3rd Embodiment of this invention. The flowchart which shows the procedure of the emotion determination after the telephone call end by the mobile telephone of 3rd Embodiment of this invention A graph that continuously expresses the strength of emotion for each emotion type output by the emotion identification section in a time series of unit time Table that digitizes the graph of FIG. Example of recalculating the emotion parameter of unit time shown in FIG. Example of icon display when characteristic emotion data is displayed graphically on the display Example of emission color when outputting characteristic emotion data to LED Example of LED emission (flashing) pattern corresponding to emotional intensity Example of displaying an incoming call history list on the display unit 30 of the mobile phone according to the third embodiment of the present invention. An example in which a call result is displayed on the display unit 30 of the mobile phone according to the third embodiment of the present invention. Another example of the second output form in which the call result is displayed on the display unit 30 of the mobile phone according to the third embodiment of the present invention. Another example of the second output form in which the call result is displayed on the display unit 30 of the mobile phone according to the third embodiment of the present invention. Another example of the second output form in which the call result is displayed on the display unit 30 of the mobile phone according to the third embodiment of the present invention. Another example of the second output form in which the call result is displayed on the display unit 30 of the mobile phone according to the third embodiment of the present invention. Another example of the second output form in which the call result is displayed on the display unit 30 of the mobile phone according to the third embodiment of the present invention. An image associated with the called party (A) is an image before image processing, and (B) is an image after image processing. Example of screen display by phone book in which the registrant's face image is registered as an image associated with the registrant (A) Normal phone book screen display (A), (B) Screen display (B)

Explanation of symbols

DESCRIPTION OF SYMBOLS 1,11 Mobile phone 10 Communication apparatus 101 Communication part 102 Audio | voice signal output part 103 Speaker 104 Microphone 105 Guidance sound storage part 106 Voice storage part 2 Emotion content storage server 20 Emotion notification apparatus 201 Emotion estimation part 202 Emotion accumulation part 203 Emotion identification part 204 emotion content storage unit 205 emotion notification operation instruction unit 206 emotion content acquisition unit 30 display unit 40 LED
50 Vibration 60 Mail transmission unit 70 Storage unit 80 Speaker 90 Scent generator 95 Input operation unit

Claims (20)

Voice input means for inputting voice;
Parameter generating means for generating emotion parameters from the input speech;
Display means for displaying information about the emotion corresponding to the parameter generated by the parameter generating means;
Input operation means for instructing user operations;
Storage means for storing data;
A call means for making a call,
The voice input means inputs voice during a call by the call means,
The parameter generating means generates emotion parameters during a call from the input voice during the call,
The storage means stores the emotion parameters during the call generated by the parameter generation means in correspondence with the order in which the parameter generation means generates the emotion parameters from the voice during the call,
The input operation means indicates a plurality of display forms of information about the emotion or a plurality of output forms of a call result of information about the emotion,
The display means, based on the order of generating the parameter, before SL after the call is terminated by the calling means, designated by the input operation means, call result information about the display form or the emotion information about the emotional The information processing terminal which displays the information about the emotion corresponding to the parameter stored in the storage means according to the output form . The information processing terminal according to claim 1,
The display means displays various information,
The storage means stores the emotion parameter for each call by the call means,
The display means is an information processing terminal that displays information about an emotion corresponding to a parameter stored in the storage means for each call item in the call history when displaying the call history. The information processing terminal according to claim 1,
The display means displays various information,
The storage means stores the emotion parameter for each call by the call means,
The information processing terminal that displays information about an emotion corresponding to a specific parameter among emotions corresponding to the parameter stored in the storage unit in a time series. The information processing terminal according to any one of claims 1 to 3,
The information processing terminal that displays information about emotions of each caller, one of the callers, or the entire caller, corresponding to the parameters stored in the storage means. An information processing terminal according to claim 4,
The storage means stores face image information in association with the person who captured the face image,
The display means displays information about the emotions of each caller, one of the callers, or the whole of the callers, and displays the face image of the caller read from the storage means after image processing. Processing terminal. An information processing terminal according to any one of claims 1 to 5,
An emotion specifying means for specifying an emotion represented by a characteristic parameter among the generated parameters;
The information processing terminal that displays information about the identified emotion. The information processing terminal according to claim 6,
Storage means for storing data;
The storage means stores emotion parameters generated by the parameter generation means,
The emotion identifying means is an information processing terminal that identifies an emotion represented by a characteristic parameter among the stored parameters. The information processing terminal according to claim 6 or 7,
It has a means for making calls,
The voice input means is an information processing terminal for inputting voice during a call by the call means. An information processing terminal according to claim 8,
Voice recording means for recording voice during a call by the calling means,
The voice input means is an information processing terminal for inputting the voice recorded by the voice recording means. An information processing terminal according to claim 8 or 9, wherein
The display means displays various information,
The storage means stores the identified emotion for each call,
The display means is an information processing terminal that displays information about the identified emotion in accordance with the call history when displaying the call history. The information processing terminal according to claim 10,
The storage means stores a part of sound reflecting the identified emotion,
The information processing terminal that displays the presence or absence of the partial voice at the same time as the identified emotion according to the call history. An information processing terminal according to claim 8 or 9, wherein
The display means displays various information,
The storage means stores the identified emotion for each caller,
The display means is an information processing terminal for displaying information on the specified emotion in accordance with the personal information when displaying the personal information of the caller stored in the storage means. An information processing terminal according to claim 8 or 9, wherein
The display means displays various information,
The storage means stores the identified emotion in correspondence with the date of the call,
The display means is an information processing terminal for displaying information about the specified emotion in accordance with the date of the calendar when displaying the calendar. An information processing terminal according to any one of claims 6 to 13,
The display means displays various information,
The storage means stores an image corresponding to emotion,
The information processing terminal that displays the image corresponding to the identified emotion read from the storage unit. The information processing terminal according to claim 14,
The storage means stores the appearance frequency of the emotion specified by the emotion specifying means,
The information processing terminal that displays the image corresponding to the identified emotion read from the storage unit according to the appearance frequency of the emotion. The information processing terminal according to claim 14 or 15,
A receiving means for receiving data from a server that records images corresponding to emotions;
The storage means is an information processing terminal that stores an image corresponding to the received emotion. The information processing terminal according to any one of claims 7 to 16, wherein
The emotion identifying means is an information processing terminal that identifies an emotion having the highest parameter value among emotion parameters stored by the storage means. The information processing terminal according to any one of claims 7 to 16, wherein
The emotion specifying means is an information processing terminal for specifying an emotion having the highest sum of numerical values of parameters generated from the start to the end of the voice among the emotion parameters stored by the storage means. The information processing terminal according to any one of claims 7 to 16, wherein
The emotion identifying means is an information processing terminal for identifying an emotion having the highest numerical value of a parameter generated before the end of the voice among emotion parameters stored by the storage means. An information processing terminal according to any one of claims 6 to 19,
An information processing apparatus comprising a transmission means for transmitting the specified emotion to another information processing terminal.