JP2015050470A - Terminal device and speech communication data processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To utilize speech communication data obtained during speech communication in a terminal device.SOLUTION: A terminal device records speech communication data of a speaker and extracts a time when the speaker is in a specific emotional state (a normal state or any emotional state) from the speaker's voice level (voice pitch and volume) recorded in the speech communication data. On the basis of the time when the speaker is in the specific emotional state during the speech communication and the voice level during the speaker is in the emotional state, the terminal device determines emotional degree of the speaker after the speech communication is over, associates the determined emotional degree with another speaker, and records it in a speech communication record. Moreover, the terminal device changes a display mode of a prescribed screen of application (application equipped with a telephone directory, speech communication history, a contact list, or the like) when the speaker is communicating with the another speaker, in accordance with the emotional degree recorded in the speech communication record.

Description

  The present invention relates to a terminal device, and more particularly to a terminal device and a call data processing method for processing call data.

  For mobile phones and smartphones, expansion of various call functions for improving communication has been proposed. For example, in Patent Document 1 below, character data is stored in a terminal, the level of an audio signal received by the mobile phone terminal is detected, and character animation is transmitted in response to the level of the audio signal. A technique for displaying on the screen is disclosed.

  Further, in Patent Document 2 below, when a mood mode call for adding a sound effect / music at the time of a voice call is selected, and when a mood mode call is performed, the user's detected from the biometric sensor or the character data of the mail A technique is disclosed in which an emotional state is analyzed to set a mood mode and a mood level, and sound effects or music are selected or changed according to the mood mode and the mood level.

JP 2005-64939 A JP 2012-4885 A

However, the techniques described in Patent Documents 1 and 2 have the following problems. First, it aims at improving the communication effect during a call, and therefore cannot be applied to scenes other than during a call. Further, even if a character is displayed or a sound effect is played in accordance with the call voice, it is not always effective for the user who is calling, and it is difficult to say that the communication improvement effect is always increased. In addition, many people use mobile phones to connect with someone quickly, but how much they are involved with that person depends on the application history such as call history and the number of mail / SMS (Short Message Service) exchanges. Can only be seen in units.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to utilize call data obtained during a call even after the call to improve communication.

In order to solve the above-described problem, a terminal device according to the present invention includes a call recording unit that records a caller's call voice and a caller's call voice recorded in the call record unit. Based on the state extraction unit that extracts the time in the emotional state, the time in the specific emotional state, and the voice level during the time in the emotional state, the emotional level of the caller is determined after the call ends A determining unit, and a presenting unit that changes and presents a display mode of a predetermined screen including information related to the other party of the call after the end of the call according to the emotion level determined by the determining unit;
And.

In the present invention, the voice level is a voice level and a loudness level that the caller makes during a call, the emotion level is a laughter level, and the predetermined screen is a phone book, a call It includes at least one of a history, a call history, a call history, and a contact list display screen.

  Furthermore, in the present invention, the storage unit that stores an image and the presenting unit extract an image corresponding to the emotion level from the storage unit based on the determination result of the emotion level, and display the image on the predetermined screen. An image extracting unit is included.

  According to the present invention, data obtained during a call can be used to confirm the communication status with another person after the call.

It is a figure which shows the concept of the basic operation | movement of the voice call terminal which concerns on embodiment of this invention. It is a figure which shows the hardware constitutions of the voice call terminal which concerns on embodiment of this invention. It is a figure which shows the functional block of the control part of the voice call terminal which concerns on embodiment of this invention. It is a figure which shows the concept of the laughter degree determination which concerns on embodiment of this invention. It is a figure which shows the display screen of the telephone directory application which concerns on embodiment of this invention. It is a figure which shows the display screen of the other application which concerns on embodiment of this invention.

  DESCRIPTION OF EMBODIMENTS Hereinafter, a mode for carrying out the present invention (hereinafter referred to as this embodiment) will be described in detail with reference to the accompanying drawings. Note that the same numbers are assigned to the same elements throughout the description of the present embodiment.

(Basic operation concept of embodiment)
FIG. 1 is a diagram showing a concept of basic operation of a voice call terminal which is an example of a terminal device according to an embodiment of the present invention. The “emotion level” in the present invention means the degree of emotional state (either normal state or emotional state) during the call of the caller. The main steps of the processing flow of the voice call terminal will be described in particular when “laughter” is selected from the degrees.

  As shown in the drawing, first, in step S1, during a call with the other party, the laughter of himself (the owner of the terminal) and the other party is recorded in the storage unit in the terminal.

  Next, in step S2, “the level of laughter” (the level and / or magnitude of the laughter) and “the time when laughter was detected” are extracted for each unit time during the call.

  In step S3, a graph (referred to as an emotional degree determination graph) as shown on the right of the figure with the level of laughter as the X axis and the time when laughter is detected as the Y axis is provided in advance, and the call extracted in step S2 The total value (Xa) of the level of laughter for each unit time and the total time (Ya) at which laughter was detected in the total call time are obtained and compared with the graph.

  Finally, in step S4, the degree of laughter (for example, 0 (standard), 1, 2, 3, 4) applied to the point (Xa, Ya) where the X axis and the Y axis intersect on the graph. A UI (User Interface) corresponding to () is associated with the other party and stored in the telephone directory.

  In this way, the phone book, which currently has only the role of registering the other party's information, can be used as a gateway to communicate with the other party and visualize how much contact with the other party is possible. The call function can be expanded. For example, by referring to the display of the phone book of the other party who has laughed for a long time, it is remembered that he had a pleasant conversation with the other party. Also, if you anger the other party in the previous call, it will also alert you when you next communicate.

(Configuration of the embodiment)
FIG. 2 is a diagram showing a hardware configuration of the voice call terminal according to the embodiment of the present invention. As illustrated, the voice call terminal 10 according to the present embodiment exemplifies a mobile phone or a smartphone. The voice call terminal 10 includes a control unit 11 including a CPU 12 (Central Processing Unit) and a storage unit 13 (memory), a display unit 14 (display) having a display screen, and a key operation unit 15 that receives user key operations. A wireless communication unit 16 that controls wireless communication, and an audio input / output unit 17 including a microphone and a speaker. In addition, an imaging unit 18 (camera) and various sensors 19 may be provided.

  The control unit 11 controls the overall operation of the voice call terminal 10 as a whole. The control unit 11 includes a computer (microprocessor) that executes processing based on a program (OS, application program, etc.) stored in the storage unit 13, and performs processing according to a procedure instructed in this program.

  Various data used for processing in the CPU 12 is stored in the storage unit 13. Specifically, in addition to the program executed by the CPU 12, various data such as temporary data used in the process of the program are held. The storage unit 13 is, for example, a nonvolatile storage device (a nonvolatile semiconductor memory such as an SSD (Solid State Drive) or an SD memory card (Secure Digital memory card)) or a non-volatile random accessible storage device (for example, an SRAM). , DRAM). Specific operations of the control unit 11 according to the present embodiment will be described later.

  The display unit 14 is configured using a display device such as a liquid crystal panel or an organic EL (Electro-Luminescence), and displays an image corresponding to the video signal generated by the control unit 11. The key operation unit 15 and the display unit 14 described above may be configured by a touch panel in which the operation unit and the display unit are integrally formed.

  The key operation unit 15 includes keys or buttons to which various functions are assigned, such as a power key, a call key (calling / calling), a volume key, a numeric key, a character key, a selection key, and a determination key. doing. For example, when a numeric key is operated by a user, a signal corresponding to the operation content is generated and input to the control unit 11. The key operation unit 15 may be partially or entirely configured by a touch sensor. In this case, a signal corresponding to the touch operation is generated and input to the control unit 11.

  The wireless communication unit 16 performs wireless communication with a base station (not shown) connected to a communication network such as a voice communication network or a data communication network. For example, the transmission data supplied from the control unit 11 is subjected to a predetermined modulation process to be converted into a radio signal and transmitted via an antenna. In addition, a predetermined demodulation process is performed on a radio signal from the base station received by the antenna to convert it into reception data, which is output to the control unit 11.

The voice input / output unit 17 is mainly used in voice call, sound recording, and sound reproduction, and performs input / output processing of a voice signal input via a microphone and a voice signal output from a speaker. That is, the sound input from the microphone is amplified, subjected to analog-digital conversion, further subjected to signal processing such as encoding, converted into digital sound data, and output to the control unit 11. In addition, the audio data output from the control unit 11 is subjected to signal processing such as decoding, digital-analog conversion, and amplification, and is converted into an analog audio signal and output to a speaker. Moreover, you may provide the speech recognition function which converts the input audio | voice signal into a character text. Instead of notification by sound, it is also possible to notify an incoming mail or telephone call by vibration of the casing generated by a vibration generating unit (not shown).

  The imaging unit 18 (camera) can input characters other than key operations and voice recognition and various images by taking pictures of OCR and barcode as well as taking pictures and videos. The various sensors 19 may include, for example, an acceleration sensor, a proximity sensor, a temperature sensor, a biological sensor (perspiration sensor, pulse sensor, etc.), a direction sensor, and the like.

  The voice call terminal 10 described above is specifically composed of a peripheral LSI including a CPU 12 and a storage unit 13 (memory), and the functions of the control unit 11 serving as a control center are recorded in the memory by the CPU 12. This is realized by sequentially reading and executing the program. The functions of the display unit 14, the key operation unit 15, the wireless communication unit 16, and the voice input / output unit 17 are realized by a peripheral LSI that is programmably controlled by the CPU 12, and an input / output port (not shown) is provided. The functions of the control unit 11 are realized.

(Operation of the embodiment)

  Hereinafter, the operation of the mobile terminal device of the present embodiment will be described in detail with reference to FIGS. 3 and 4. FIG. 3 is a diagram showing functional blocks of the control unit of the voice call terminal according to the embodiment of the present invention. FIG. 4 is a diagram showing the concept of emotion level determination according to the embodiment of the present invention.

  As shown in FIG. 3, as will be described later, the control unit 11 is divided into functional blocks of a call recording unit 11a, a state extraction unit 11b, a determination unit 11c, a presentation unit 11d, and an image extraction unit 11e as processing units. Can do. Each functional block may be integrated or further divided. Further, the storage unit 13 included in the control unit 11 stores call voice data 13a, an emotion level determination table 13b, a call record 13c, an emotion level illustration 13d, and a user photo collection 13e, as will be described later.

  The call recording unit 11a receives a voice input from the microphone of the voice input / output unit 17 and the wireless communication unit 16, and records a voice call between the person who is the owner of the voice call terminal 10 (self) and the other party, Recorded in the storage unit 13 as call voice data 13a. At this time, in the case where various sensors 20 are equipped with sensors that detect the caller's biological information, such as a sweat sensor and a pulse sensor, the biological information may be separately recorded during the call time. In the call voice data 13a, the entire call may be recorded or may be recorded except for the silent part.

  The state extraction unit 11b analyzes the recorded call voice data 13a and first separates the utterance portion between the user and the call partner. Then, a specific emotional state during the call is detected from the voice level (pitch, loudness) uttered during the call for each speaker, and the start time and end time of the specific emotional state are recorded. In the detection of the emotional state, when information from a biosensor or user profile information (gender, age, etc.) is obtained, such information may be used. The specific emotional state refers to the emotional state selected as one of the emotional state as well as the state where the caller is in a normal emotional state. Although this emotional state detection is conceptually shown in FIG. 4A, a known technique may be used. However, reference signs A and B in the figure indicate the start time and end time of a specific emotional state (in this example, a state of comfort (laughing)), and reference character C indicates the time that the emotional state lasted. . In addition, you may make it refer the information from the biometric sensor mentioned above for extraction of an emotional state.

When the extraction process of the state extraction unit 11b is completed, the determination unit 11c is called, and the determination unit 11c obtains the time during which the extracted specific emotional state has been maintained, and the time during which the specific emotional state has been maintained. The voice level (pitch, loudness) is detected every unit time. Then, the total time (Xa) during which the specific emotion has been maintained over the entire call and the total value (Ya) of the voice level per unit time during that time are obtained. Then, the obtained total value (Xa, Ya) is determined with reference to the emotion level determination table 13b, and the emotion level, which is the degree of increase in the emotion, is determined after the call ends. The emotion level determined by the determination unit 11c is stored in the storage unit in association with the call partner information (call partner identification information). The emotion level is determined by a step value (level) such as an emotion level of 1 to 4 according to the rise of a specific emotion, with a normal state (standard) as an emotion level of 0, for example. The emotion level determination table 13b represents the emotion level determination graph described in FIG. 1 (reprinted in FIG. 4C) in a table format so that the computer can easily process it (FIG. 4B). reference). However, the curve in the graph of FIG. 4C is a true arc with a radius r _n (n = 0 to 4) for simplification, but in general, it may be an arbitrary straight line or combination of curves. .

  The presentation unit 11d serves to provide the emotion level determination result as a UI (user interface) to an application such as a telephone directory. For example, in the case of a phone book application, the provision of a UI refers to presenting a display mode on a predetermined screen including information on a call partner such as a telephone number in accordance with an emotion level determination result. Presenting the display mode includes, for example, adding an effective image, changing the character size such as a phone number and the name of the other party, font, character decoration (italic, bold, underline, etc.) It means changing the background color and displaying it on the application screen.

The presentation unit 11d includes an API (Application User Interface) and can be called from each application. In addition, if the emotion level is recorded in the call record 13c, the presenting unit 11d uses the emotion level illustration 13d prepared in advance according to the emotion level of the latest call record or the average value of the emotion level for a certain period. Then, an illustration image corresponding to the recorded emotion level is extracted. Alternatively, instead of the illustration image, it is stored in a storage unit (the storage unit 13 inside the terminal or an external server database may be used in combination), and from the user photo album 13e in which the other party is shown, to the call record 13c. You may make it extract the other party's face picture corresponding to the recorded feeling degree. The extracted illustration and face photograph are displayed on the display unit 14 (display screen) of the voice call terminal 10 by the presentation unit 11d or an application that calls the presentation unit 11d. In this way, the voice data obtained during the call can be applied not only to the call but also to other applications, which is more effective for improving communication.

  FIG. 5 is a diagram showing a display screen of the telephone directory application according to the embodiment of the present invention. In FIG. 5 (a), the illustration image prepared in advance is changed step by step according to the laughter level (feeling level) recorded in the call record, and the phone number character size and background color are changed step by step. 2 shows an example of a screen that is changed dynamically. FIG. 5C shows an example of a screen in which the face photo of the other party is extracted from the user photo collection according to the degree of laughter and displayed instead of the illustration image. The face photograph does not have to be a single photograph of the other party, but may be one in which the other party is reflected in a plurality of persons. It ’s best if there ’s a picture of the person and the other party ’s two-shot. Of course, if there is no image in the photobook that corresponds to the emotion level of the other party, such as an angry face, an illustration is used.

FIG. 6 is a diagram showing a display screen of another application according to the embodiment of the present invention. In this figure, as applications other than the phone book, FIG. 6A shows a call history (including an outgoing call history and an incoming call history that were not actually called), and FIG. 6B shows a contact used in a group conversation. An example of a list display screen is shown. In the communication list, not only the one-to-one call but also the emotion level of all members when talking in a group may be displayed.

(Effect of embodiment)
As explained above, when communicating with the other party, you can visualize how much contact you have with the other party when communicating with the other party. . For example, by changing the display mode of the phone book of the other party who has laughed for a long time, it is visualized that he / she talked happily with the other party. Also, if you have angered the other party in the previous call, it will help to alert you to remember to consider the next communication.

  In addition, the obtained voice data can be used not only during a call but also when using other applications (call history, contact list, etc. other than the phone book) after the call ends, which is more effective for improving communication. . That is, by reflecting the result of communication during a call in the UI of various application functions, it is expected that the user will have more opportunities to use the functions and the communication will deepen.

  In the description of the hardware configuration of the embodiment of the present invention in FIG. 2, a mobile phone or a smartphone is exemplified as the voice call terminal. However, the voice call function and the phone book function are not necessarily required to be a portable terminal. It may be a PC (Personal Computer), a tablet terminal, a PDA (Personal Digital Assistants), or the like.

The present invention may be applied to a videophone. Specifically, instead of the time during which the specific emotional state has been maintained in the above embodiment, the time when the smile has been maintained based on the image of the caller during the videophone call obtained by the image pickup unit by the extraction unit To extract. Then, the determination unit determines the smile level of the caller according to the time when the smile extracted by the extraction unit is maintained and the level of smile level at the time. Further, the presenting unit changes and presents a display mode of a predetermined screen including information on the caller based on the smile level determined by the determining unit.
Note that the audio and video during a videophone call may be recorded in the storage unit, or only the degree of smile determined during the videophone call and not recorded in the storage unit, and only the time during which the smile was maintained It is good.
According to the said aspect, there exists an effect which can be used for the confirmation of the communication condition with others after the telephone call of the data during a videophone call.

  As mentioned above, although preferred embodiment of this invention was explained in full detail, it cannot be overemphasized that the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above embodiment. Further, it is apparent from the description of the scope of claims that embodiments with such changes or improvements can also be included in the technical scope of the present invention.

  DESCRIPTION OF SYMBOLS 10 ... Voice call terminal, 11 ... Control part, 11a ... Call recording part, 11b ... State extraction part, 11c ... Determination part, 11d ... Presentation part, 11e ... Image extraction part 11e, 12 ... CPU, 13 ... Storage part, 13a Call voice data, 13b ... Emotion level determination table, 13c ... Call record, 13d ... Illustration according to emotion level, 13e ... User photo book, 14 ... Display unit, 15 ... Key operation unit, 16 ... Wireless communication unit, 17 ... Voice Output unit, 18 ... imaging unit, 19 ... various sensors

Claims (6)

A call recording unit for recording caller's call voice;
A state extraction unit that extracts a time during which the caller is in a specific emotion state from the call voice of the caller recorded in the call recording unit;
A determination unit that determines the emotion level of the caller based on the time in the specific emotional state and the voice level during the time in the emotional state;
A presenting unit that changes and presents a display mode of a predetermined screen including information on the other party of the call after the call ends according to the emotion level determined by the determination unit;
A terminal device comprising:   The terminal device according to claim 1, wherein the voice level is a voice level and a voice level that the caller makes during a call.   The terminal device according to claim 1, wherein the emotion level is a laughter level.   4. The terminal device according to claim 1, wherein the predetermined screen includes at least one of a phone book, a call history, a call history, a call history, and a contact list display screen. . A storage unit for storing images;
The presenting unit includes an image extracting unit that extracts an image corresponding to the emotion level from the storage unit based on the determination result of the emotion level and displays the image on the predetermined screen. The terminal device according to any one of 4. A call recording step for recording a caller's call voice;
A state extracting step of extracting a time during which the caller is in a specific emotion state from the call voice of the caller recorded in the call recording step;
A determination step of determining the emotion level of the caller after the call based on the time in the specific emotional state and the voice level during the time in the emotional state;
In accordance with the degree of emotion determined in the determination step, after the call is ended, a presentation step of changing and presenting a display mode of a predetermined screen including information about the call partner;
A call data processing method characterized by comprising:

Images