JP7169031B1 - Program, information processing device, information processing system, information processing method, information processing terminal - Google Patents

Abstract

A program, an information processing device, an information processing system, an information processing method, and an information processing terminal are provided for confirming what topic was communicated between speakers in a dialogue. In a system comprising a server connected via a network, a plurality of user terminals, a CRM system, and a voice server, a control unit of the server receives voice data relating to dialogue, and from the received voice data, extracting a plurality of segmental audio data, identifying one or a plurality of segmental audio data related to a first topic related to a predetermined topic among the plurality of segmental audio data, and identifying one or more segmental audio data among the plurality of segmental audio data Alternatively, a topic processing unit that executes topic presentation processing for generating summary text summarizing text information included in one or more pieces of segmental speech data based on the plurality of pieces of segmental speech data and the first topic. [Selection drawing] Fig. 19

Description

The present disclosure relates to a program, an information processing device, an information processing system, an information processing method, and an information processing terminal.

Online interactive services between multiple users are known.
Patent Literature 1 discloses a method of assisting the realization of more efficient sales activities while considering objective indicators.

Retable 2020/184631

There is a problem that it is not possible to confirm what topic the speakers communicated about in the dialogue.
Accordingly, the present disclosure has been made to solve the above problems, and its purpose is to provide a technique for confirming what topics have been communicated between speakers in a dialogue.

A program, comprising a processor and a storage unit, for causing a computer to process information relating to a dialogue between a first user and a second user, the program comprising: a receiving step of receiving voice data regarding the dialogue to the processor; a speech extraction step of extracting a plurality of segmental speech data for each utterance segment from the speech data received in the step; included in the one or more section audio data based on the section identification step of identifying the section audio data, the one or more section audio data identified in the section identification step among the plurality of section audio data, and the first topic A program that causes a summary step to generate summary text that summarizes textual information.

According to the present disclosure, it is possible to identify what topic the speaker has communicated with in the dialogue service.

2 is a block diagram showing a functional configuration of system 1; FIG. 3 is a block diagram showing the functional configuration of the server 10; FIG. 2 is a block diagram showing the functional configuration of the first user terminal 20; FIG. 3 is a block diagram showing a functional configuration of a second user terminal 30; FIG. 2 is a block diagram showing the functional configuration of a CRM system 50; FIG. FIG. 10 is a diagram showing the data structure of a user table 1012; FIG. FIG. 10 is a diagram showing the data structure of an organization table 1013; FIG. FIG. 10 is a diagram showing the data structure of a dialogue table 1014; FIG. FIG. 10 is a diagram showing the data structure of a label table 1015; FIG. FIG. 10 is a diagram showing the data structure of a voice interval table 1016; FIG. 10 shows the data structure of a topic relevance table 1017. FIG. FIG. 10 is a diagram showing the data structure of an emotional condition master 1021; FIG. FIG. 10 is a diagram showing the data structure of speaker type master 1022. FIG. FIG. 10 is a diagram showing the data structure of a topic master 1023; FIG. It is a figure which shows the data structure of the customer table 5012. FIG. 4 is a flowchart showing the operation of emotion analysis processing; 4 is a flowchart showing the operation of impression analysis processing; 10 is a flowchart showing the operation of topic analysis processing; 10 is a flowchart showing the operation of topic presentation processing; It is an example of a screen which shows operation|movement of a topic presentation process. 2 is a block diagram showing the basic hardware configuration of computer 90. FIG.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all the drawings for explaining the embodiments, common constituent elements are given the same reference numerals, and repeated explanations are omitted. It should be noted that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Also, not all the components shown in the embodiments are essential components of the present disclosure. Each figure is a schematic diagram and is not necessarily strictly illustrated.

<Configuration of System 1>
A system 1 according to the present disclosure is an information processing system that provides an online dialogue service (online dialogue service) between a first user who is an operator and a second user who is a customer. Note that the system 1 according to the present disclosure may also provide an online interactive service between three or more users including the first user, the second user, and one or more other users.
The system 1 includes information processing devices connected via a network N, including a server 10 , a first user terminal 20 , a second user terminal 30 , a CRM system 50 and a voice server (PBX) 60 .
FIG. 1 is a block diagram showing the functional configuration of system 1. As shown in FIG.
FIG. 2 is a block diagram showing the functional configuration of the server 10. As shown in FIG.
FIG. 3 is a block diagram showing the functional configuration of the first user terminal 20. As shown in FIG.
FIG. 4 is a block diagram showing the functional configuration of the second user terminal 30. As shown in FIG.
FIG. 5 is a block diagram showing the functional configuration of the CRM system 50. As shown in FIG.

Each information processing device is composed of a computer having an arithmetic device and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the server 10, the first user terminal 20, the second user terminal 30, the CRM system 50, and the voice server (PBX) 60, descriptions overlapping the basic hardware configuration of the computer and the basic functional configuration of the computer to be described later will be omitted. .

<Configuration of Server 10>
The server 10 is an information processing device that provides a service of storing and managing data (dialogue data) related to dialogue between a first user and a second user.
The server 10 has a storage unit 101 and a control unit 104 .

<Configuration of Storage Unit 101 of Server 10>
The storage unit 101 of the server 10 includes an application program 1011, an emotion evaluation model 1031, an impression evaluation model 1032, a first impression evaluation model 1033, a second impression evaluation model 1034, a summary model 1035, a user table 1012, an organization table 1013, and a dialogue table. 1014, a label table 1015, a voice section table 1016, a topic relevance table 1017, an emotional condition master 1021, a speaker type master 1022, and a topic master 1023.

The application program 1011 is a program for causing the control unit 104 of the server 10 to function as each functional unit.
Application programs 1011 include applications such as web browser applications.

The emotion evaluation model 1031 is a model for outputting numerical strengths and numerical values for each of a plurality of emotional states, using voice data, video data, and text data relating to user utterances in voice data or video data as input data. .

The impression evaluation model 1032 is a model for outputting numerical strengths and numerical values for each of a plurality of impressions, using voice data, video data, and text data relating to user utterances in voice data or video data as input data.

The first impression evaluation model 1033 is a model for outputting a dialogue feature amount related to the manner of speaking of a speaker, using voice data, moving image data, and text data related to user utterances in voice data or moving image data as input data. The dialogue feature amount is a feature amount relating to at least one of the speaker's speaking speed, intonation, the number of polite expressions, the number of fillers, and the number of grammatical utterances.

The second impression evaluation model 1034 is a model for outputting numerical strengths and numerical values for each of a plurality of impressions using dialogue feature amounts as input data.

The user table 1012 is a table that stores and manages information on member users (hereinafter referred to as users) who use the service. By registering to use the service, the user's information is stored in a new record in the user table 1012 . This enables the user to use the service according to the present disclosure.
The user table 1012 is a table having user ID, CRM ID, organization ID, user name, and user attribute columns with user ID as a primary key.
FIG. 6 is a diagram showing the data structure of the user table 1012. As shown in FIG.

User ID is an item that stores user identification information for identifying a user. User identification information is an item in which a unique value is set for each user.
CRMID is an item that stores user identification information for identifying a user in the CRM system 50 . The user can receive CRM services by logging into the CRM system 50 using the CRM ID. A user ID in the server 10 is associated with a CRMID in the CRM system 50 .
The organization ID is an item that stores organization identification information for identifying an organization.
The user name is an item that stores the name of the user. Any character string such as a nickname may be set as the user name instead of the full name.
The user attribute is an item that stores information related to user attributes such as age, gender, hometown, dialect, occupation (sales, customer support, etc.) of the user. The user attributes may include information about the user's personal attributes, as well as information about the company's attributes such as the organization, company, group, etc. to which the user belongs.

The organization table 1013 is a table that stores and manages information (organization information) regarding organizations to which users belong. Organizations include arbitrary organizations and groups such as companies, corporations, corporate groups, circles, and various organizations. Organizations may also be defined by more detailed sub-groups such as company departments (sales department, general affairs department, customer support department).
The organization table 1013 is a table having columns of organization ID, organization name, and organization attribute with organization ID as a primary key.
FIG. 7 is a diagram showing the data structure of the organization table 1013. As shown in FIG.

The organization ID is an item that stores organization identification information for identifying an organization. Organization identification information is an item in which a unique value is set for each organization information.
The organization name is an item that stores the name of the organization. Any character string can be set for the organization name.
The organization attribute is an item that stores information related to organization attributes such as organization type (company, corporate group, other organization, etc.) and industry (real estate, finance, etc.).

The dialogue table 1014 is a table for storing and managing information (dialogue information) related to dialogue between the user and the customer.
The dialogue table 1014 is a table having columns of dialogue ID, user ID, customer ID, dialogue category, reception/transmission type, audio data, and video data, with the dialogue ID as a primary key.
FIG. 8 is a diagram showing the data structure of the dialogue table 1014. As shown in FIG.

The dialogue ID is an item that stores dialogue identification information for identifying a dialogue. Dialogue identification information is an item in which a unique value is set for each piece of dialogue information.
The user ID is an item that stores user identification information for identifying the user in the interaction between the user and the customer. A plurality of user IDs may be associated with each piece of dialogue information.
The customer ID is an item that stores user identification information for identifying a customer in a dialogue between the user and the customer. A plurality of customer user IDs may be associated with each piece of dialogue information.
The dialogue category is an item that stores the type (category) of dialogue between the user and the customer. Interaction data is classified by interaction category. Values such as telephone operator, telemarketing, customer support, and technical support are stored in the dialogue category according to the purpose of the dialogue between the user and the customer.
The reception/transmission type is an item that stores information for distinguishing whether the dialogue between the user and the customer is transmitted by the user (outbound) or received by the user (inbound). . Also, when three or more users interact, the reception/transmission type of room is stored.
Audio data is an item that stores audio data collected by a microphone. It is also possible to store reference information (paths) to audio data files located in other locations. The audio data format may be any data format such as AAC, ATRAC, mp3, mp4.
The voice data may be data in a format in which the user's voice and the customer's voice are individually identifiable as identifiers. In this case, the control unit 104 of the server 10 can perform independent analysis processing on the user's voice and the customer's voice. Also, the user ID of the user or customer can be specified based on the voice data of the user or customer.
In the present disclosure, video data including audio information may be used instead of audio data. Also, audio data in the present disclosure includes audio data included in moving image data.
Moving image data is an item that stores moving image data captured by a camera or the like. It is also possible to store reference information (paths) for moving image data files located in other locations. Any data format such as MP4, MOV, WMV, AVI, and AVCHD may be used as the format of moving image data.
The moving image data may be data in a format in which identifiers are set so that the moving images of the user and the moving images of the customer are independently identifiable. In this case, the control unit 104 of the server 10 can perform independent analysis processing on the user's moving image and the customer's moving image. Also, the user ID of the user and the customer can be specified based on the video data of the user and the customer.

The label table 1015 is a table for storing and managing information about labels (label information).
The label table 1015 is a table having columns of dialogue ID and label data.
FIG. 9 is a diagram showing the data structure of the label table 1015. As shown in FIG.

The dialogue ID is an item that stores dialogue identification information for identifying a dialogue.
Label data is an item that stores label information for managing interactions. Label information is additional information for managing dialogue information, such as classification names, labels, classification labels, and tags.
The label data may be a character string indicating the name of label information, or may be a label ID or the like for referring to the name of label information stored in another table.
Label data includes classification information according to the speaker's emotional state in a particular dialogue. The classification data includes classification information for classifying whether the speaker's response is good or bad in a specific dialogue.

The speech segment table 1016 is a table for storing and managing information (speech segment information) regarding a plurality of speech segments included in dialogue information.
The speech section table 1016 uses the section ID as a main key, and includes section ID, dialogue ID, speaker ID, start date and time, end date and time, section speech data, section video data, section reading text, emotion data, impression data, and topic ID. is a table with columns of
FIG. 10 is a diagram showing the data structure of the voice interval table 1016. As shown in FIG.

The section ID is an item that stores section identification information for identifying a speech section. The section identification information is an item in which a unique value is set for each piece of speech section information.
The dialog ID is an item that stores dialog identification information for identifying a dialog associated with speech segment information.
The speaker ID is an item that stores speaker identification information for identifying a speaker associated with speech segment information. Specifically, the speaker ID is an item that stores the user IDs of multiple users who have participated in the dialogue.
The start date and time is an item for storing the start date and time of the audio segment and the moving image segment.
The end date and time is an item for storing the end date and time of the audio segment and the video segment.
The section sound data is an item for storing sound data included in the sound section. It is also possible to store reference information (paths) to audio data files located in other locations. Also, based on the start date and time and the end date and time, reference to the voice data in the period from the start date and time to the end date and time of the voice data in the dialog table 1014 may be stored. Also, the section audio data may include the audio data included in the section video data.
The audio data format may be any data format such as AAC, ATRAC, mp3, mp4.
Section moving image data is an item for storing moving image data included in a voice section. It is also possible to store reference information (paths) for moving image data files located in other locations. Also, based on the start date and time and the end date and time, reference to the video data in the period from the start date and time to the end date and time of the video data in the interaction table 1014 may be stored.
Any data format such as MP4, MOV, WMV, AVI, and AVCHD may be used as the format of moving image data.
The section reading text is an item that stores text information of the content uttered by the speaker in the section speech data included in the speech section. Specifically, the section reading text may be generated manually, using arbitrary machine learning, deep learning, or other learning models based on section audio data and section video data.
Emotion data is an item that stores the emotional state of the speaker in the speech period. Emotional data is a multidimensional measure (emotional vector) of a speaker's multiple emotional states, such as interest/excitement, joy, surprise, anxiety, anger, disgust, contempt, fear, shame, guilt, and so on. Emotion data quantitatively expresses what kind of emotional state the speaker is in during the dialogue section as the intensity and numerical value for each of a plurality of emotional states (dimensions). The emotion data may be configured to calculate and store an emotion scalar indicating the intensity of one-dimensional emotion based on the emotion vector.
The impression data is an item that stores the speaker's impression in the speech period. Impression data is a multidimensional scale (vector) of multiple different impressions given by a speaker: like, dislike, picky, hard to hear, polite, unclear, timid, nervous, intimidating, violent and sexual. It quantitatively expresses what kind of impression the speaker gives in the dialogue interval as the intensity and numerical value for each of multiple impressions (dimensions).
The topic ID is an item that stores topic identification information associated with a voice segment in the voice segment.

The topic relevance table 1017 is a table for storing and managing information on topic relevance (topic relevance information) for each speech segment.
The topic relevance table 1017 is a table having columns of section ID, topic ID, and relevance.
FIG. 11 is a diagram showing the data structure of the topic relevance table 1017. As shown in FIG.

The section ID is an item for storing section identification information of a target speech section.
Topic ID is an item that stores topic identification information for identifying a topic.
The degree of relevance is an item that stores information relating to the degree of relevance for each topic identification information specified by a topic ID in a speech segment included in dialogue information. This is an item in which a numerical value indicating the degree of relevance to a topic specified by a topic ID is stored for one voice segment. The greater the degree of relevance, the stronger the relevance between the dialogue information and the topic.

Emotional condition master 1021 is a table for storing and managing information on emotional conditions (emotional condition information).
The emotional condition master 1021 is a table having columns of emotional conditions and label data.
FIG. 12 is a diagram showing the data structure of the emotional condition master 1021. As shown in FIG.

The emotional condition is an item that stores conditions related to emotional data. Specifically, conditions for threshold values, average values, regression coefficients when regression analysis is performed, and the like of emotion data are stored.
Label data is an item that stores label information associated with emotional conditions.

The speaker type master 1022 is a table for storing and managing information on impression conditions (impression condition information).
The speaker type master 1022 is a table having columns of impression condition and speaker type.
FIG. 13 shows the data structure of the speaker type master 1022. As shown in FIG.

The impression condition is an item that stores conditions related to impression data. Specifically, conditions for threshold values, average values, regression coefficients when performing regression analysis, etc. of impression data are stored.
The speaker type is an item that stores the speaker type associated with the impression condition. The speaker type classifies the impression that the speaker gives to the interlocutor, such as assertive, modest, serious, friendly, positive, and emotional.

The topic master 1023 is a table for storing and managing information on topics (topic information).
The topic master 1023 is a table having topic ID and keyword columns with topic ID as a primary key.
FIG. 14 is a diagram showing the data structure of the topic master 1023. As shown in FIG.

Topic ID is an item that stores topic identification information for identifying a topic. Topic identification information is an item in which a unique value is set for each topic information.
A keyword is an item that stores a plurality of keywords with which a topic is associated. Specifically, a plurality of keywords are associated with one topic.

<Configuration of Control Unit 104 of Server 10>
The control unit 104 of the server 10 includes a user registration control unit 1041 , an emotion analysis unit 1042 , an impression analysis unit 1043 , a topic processing unit 1044 and a learning unit 1051 . Control unit 104 implements each functional unit by executing application program 1011 stored in storage unit 101 .

The user registration control unit 1041 performs processing for storing information of users who wish to use the service according to the present disclosure in the user table 1012 .
The information stored in the user table 1012 is transmitted to the server 10 by the user opening a web page or the like operated by the service provider from any information processing terminal, entering information in a predetermined input form, and transmitting the information to the server 10 . The user registration control unit 1041 stores the received information in a new record of the user table 1012, and user registration is completed. As a result, the users stored in the user table 1012 can use the service.
Prior to registration of user information in the user table 1012 by the user registration control unit 1041, the service provider may perform a predetermined examination to limit whether or not the user can use the service.
The user ID may be any character string or number that can identify the user, any character string or number desired by the user, or any character string or number automatically set by the user registration control unit 1041. .

Emotion analysis unit 1042 executes emotion analysis processing. Details will be described later.

The impression analysis unit 1043 executes impression analysis processing. Details will be described later.

The topic processing unit 1044 executes topic definition processing, topic analysis processing, and topic presentation processing. Details will be described later.

The learning unit 1051 executes learning processing.

<Configuration of first user terminal 20>
The first user terminal 20 is an information processing device operated by a first user who uses the service. The first user terminal 20 may be, for example, a stationary PC (Personal Computer), a laptop PC, or a mobile terminal such as a smart phone or a tablet. Moreover, it may be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch type terminal.
The first user terminal 20 includes a storage unit 201 , a control unit 204 , an input device 206 and an output device 208 .

<Configuration of Storage Unit 201 of First User Terminal 20>
The storage unit 201 of the first user terminal 20 has a first user ID 2011 and an application program 2012 .

The first user ID 2011 stores the user identification information of the first user. The user transmits the first user ID 2011 from the first user terminal 20 to the voice server (PBX) 60 . The voice server (PBX) 60 identifies the first user based on the first user ID 2011 and provides the service according to the present disclosure to the first user. The first user ID 2011 includes information such as a session ID temporarily assigned by the voice server (PBX) 60 for identifying the user using the first user terminal 20 .

The application program 2012 may be stored in the storage unit 201 in advance, or may be downloaded from a web server or the like operated by the service provider via the communication IF.
Application programs 2012 include applications such as web browser applications.
The application program 2012 includes an interpreted programming language such as JavaScript (registered trademark) that runs on a web browser application stored in the first user terminal 20 .

<Configuration of the control unit 204 of the first user terminal 20>
The control unit 204 of the first user terminal 20 has an input control unit 2041 and an output control unit 2042 . Control unit 204 implements each functional unit by executing application program 2012 stored in storage unit 201 .

<Configuration of Input Device 206 of First User Terminal 20>
The input device 206 of the first user terminal 20 has a camera 2061 , a microphone 2062 , a position information sensor 2063 , a motion sensor 2064 and a keyboard 2065 .

<Configuration of output device 208 of first user terminal 20>
The output device 208 of the first user terminal 20 has a display 2081 and a speaker 2082 .

<Configuration of Second User Terminal 30>
The second user terminal 30 is an information processing device operated by a second user who uses the service. The second user terminal 30 may be, for example, a mobile terminal such as a smart phone or tablet, a stationary PC (Personal Computer), or a laptop PC. Moreover, it may be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch type terminal.
The second user terminal 30 includes a storage section 301 , a control section 304 , an input device 306 and an output device 308 .

<Configuration of Storage Unit 301 of Second User Terminal 30>
The storage unit 301 of the second user terminal 30 has an application program 3012 and a telephone number 3013 .

The application program 3012 may be stored in advance in the storage unit 301, or may be downloaded from a web server or the like operated by the service provider via the communication IF.
Application programs 3012 include applications such as web browser applications.
The application program 3012 includes an interpreted programming language such as JavaScript (registered trademark) executed on a web browser application stored in the second user terminal 30 .

<Configuration of the control unit 304 of the second user terminal 30>
The control unit 304 of the second user terminal 30 has an input control unit 3041 and an output control unit 3042 . Control unit 304 implements each functional unit by executing application program 3012 stored in storage unit 301 .

<Configuration of Input Device 306 of Second User Terminal 30>
The input device 306 of the second user terminal 30 has a camera 3061 , a microphone 3062 , a position information sensor 3063 , a motion sensor 3064 and a touch device 3065 .

<Configuration of output device 308 of second user terminal 30>
The output device 308 of the second user terminal 30 has a display 3081 and a speaker 3082 .

<Configuration of CRM System 50>
The CRM system 50 is an information processing device managed and operated by a company (CRM company) that provides CRM (Customer Relationship Management, second user relationship management) services. CRM services include SalesForce, HubSpot, Zoho CRM, Kintone, and the like.
The CRM system 50 has a storage unit 501 and a control unit 504 .

<Configuration of Storage Unit 501 of CRM System 50>
The storage unit 501 of the CRM system 50 has an application program 5011 and a customer table 5012 .

The application program 5011 is a program for causing the control unit 504 of the CRM system 50 to function as each functional unit.
Application programs 5011 include applications such as web browser applications.

The customer table 5012 is a table for storing and managing user information (customer information) on customers.
The customer table 5012 is a table having customer ID, user ID, name, telephone number, and speaker type columns with customer ID as a primary key.
FIG. 15 is a diagram showing the data structure of the customer table 5012. As shown in FIG.

The customer ID is an item that stores customer user identification information. User identification information is an item in which a unique value is set for each customer.
User ID is an item for storing user identification information of a user who manages a customer.
The name is an item for storing the customer's name.
The phone number is an item that stores the customer's phone number.
The user accesses the website provided by the CRM system, selects the customer to whom he/she wants to make a call, and performs a predetermined operation such as "call" to make a call from the first user terminal 20 to the customer's telephone number. I can make a call.
The speaker type is an item that stores the user's speaker type specified by the customer ID.

<Configuration of Control Unit 504 of CRM System 50>
The control unit 504 of the CRM system 50 has a user registration control unit 5041 . Control unit 504 implements each functional unit by executing application program 5011 stored in storage unit 501 .

The user registration control unit 5041 performs processing for storing customer information in the customer table 5012 in the service according to the present disclosure.
The information stored in the customer table 5012 is sent to the CRM system 50 by the user opening a web page or the like operated by the service provider from any information processing terminal, entering information in a predetermined input form, and sending the information to the CRM system 50 . The user registration control unit 5041 stores the received information in a new record in the customer table 5012, completing customer registration. As a result, the customer information is stored in association with the user ID of the user who manages the customer.
The customer ID may be any character string or number that can identify the user, any character string or number desired by the user, or any character string or number automatically set by the user registration control unit 5041. .

<Configuration of voice server (PBX) 60>
The voice server (PBX) 60 is an information processing device that functions as an exchange that enables dialogue between the first user terminal 20 and the second user terminal 30 by connecting the network N and the telephone network T to each other. .
The voice server (PBX) 60 has a storage unit 601 .

<Configuration of storage unit 601 of voice server (PBX) 60>
The storage unit 601 of the voice server (PBX) 60 has an application program 6011 .

The application program 6011 is a program for causing the control unit 604 of the voice server (PBX) 60 to function as each functional unit.
Application programs 6011 include applications such as web browser applications.

<Operation of System 1>
Each process of the system 1 will be described below.
FIG. 16 is a flow chart showing the operation of emotion analysis processing.
FIG. 17 is a flowchart showing the operation of impression analysis processing.
FIG. 18 is a flowchart showing the operation of topic analysis processing.
FIG. 19 is a flowchart showing the operation of topic presentation processing.
FIG. 20 is a screen example showing the operation of topic presentation processing.

<Outgoing process>
The calling process is a process of making a call (calling) from a user (first user) to a customer (second user).

<Outline of call processing>
The calling process is a series of processes in which the user selects a customer who wishes to make a call from among a plurality of customers displayed on the screen of the first user terminal 20 and performs a calling operation to make a call to the customer. . In the present disclosure, a case of selecting the second user as a customer will be described as an example.

<Details of outgoing processing>
A call processing of the system 1 when a call is made from a user to a customer will be described.

When a user calls a customer, the system 1 performs the following processes.

By operating the first user terminal 20 , the user activates the web browser and accesses the CRM service website provided by the CRM system 50 . The user can display a list of his/her own customers on the display 2081 of the first user terminal 20 by opening the customer management screen provided by the CRM service.
Specifically, the first user terminal 20 transmits a CRM ID 2013 and a request to display a list of customers to the CRM system 50 . When the CRM system 50 receives the request, it searches the customer table 5012 and sends information about the user's customer, such as customer ID, name, telephone number, customer attributes, customer organization name, customer organization attributes, etc., to the first user terminal 20. do. The first user terminal 20 displays the received customer information on the display 2081 of the first user terminal 20 .

The user presses and selects a customer (second user) to whom a call is desired from the customers listed on the display 2081 of the first user terminal 20 . With the customer selected, pressing the "call" button or phone number button displayed on the display 2081 of the first user terminal 20 sends a request including the phone number to the CRM system 50 . The CRM system 50 that has received the request transmits the request including the telephone number to the server 10 . The server 10 that has received the request transmits a call origination request to the voice server (PBX) 60 . Upon receiving the call request, the voice server (PBX) 60 makes a call (call) to the second user terminal 30 based on the received telephone number.

Along with this, the first user terminal 20 controls the speaker 2082 and the like to ring to indicate that the voice server (PBX) 60 is making a call (calling). Also, the display 2081 of the first user terminal 20 displays information indicating that the voice server (PBX) 60 is making a call to the customer. For example, the display 2081 of the first user terminal 20 may display the words "calling".

The customer picks up the receiver (not shown) on the second user terminal 30 or presses the "receive" button displayed on the input device 306 of the second user terminal 30 when receiving an incoming call. becomes possible. Along with this, the voice server (PBX) 60 transmits information indicating that the second user terminal 30 has responded (hereinafter referred to as a “response event”) to the first Send to the user terminal 20 .
As a result, the user and the customer can interact using the first user terminal 20 and the second user terminal 30, respectively, so that the user and the customer can interact with each other. Specifically, the user's voice collected by the microphone 2062 of the first user terminal 20 is output from the speaker 3082 of the second user terminal 30 . Similarly, the customer's voice collected from the microphone 3062 of the second user terminal 30 is output from the speaker 2082 of the first user terminal 20 .

When the display 2081 of the first user terminal 20 becomes ready for interaction, it receives the response event and displays information indicating that interaction is taking place. For example, the display 2081 of the first user terminal 20 may display the characters "answering".

<Incoming processing>
Incoming call processing is processing in which the user receives a call (receives a call) from a customer.

<Overview of Incoming Call Processing>
The incoming call process is a series of processes in which the user receives an incoming call when the customer calls the user while the user has launched an application on the first user terminal 20 .

<Details of incoming call processing>
Incoming call processing of the system 1 when the user receives an incoming call (receives a call) from a customer will be described.

When a user receives an incoming call from a customer, the system 1 performs the following processing.

By operating the first user terminal 20 , the user activates the web browser and accesses the CRM service website provided by the CRM system 50 . At this time, it is assumed that the user logs in to the CRM system 50 with his own account on the web browser and waits. It is sufficient for the user to be logged in to the CRM system 50, and the user may be performing other work related to the CRM service.

The customer operates the second user terminal 30 , inputs a predetermined telephone number assigned to the voice server (PBX) 60 , and makes a call to the voice server (PBX) 60 . The voice server (PBX) 60 receives the outgoing call from the second user terminal 30 as an incoming call event.

A voice server (PBX) 60 sends an incoming call event to the server 10 . Specifically, the voice server (PBX) 60 transmits an incoming call request including the customer's telephone number 3011 to the server 10 . The server 10 transmits an incoming call request to the first user terminal 20 via the CRM system 50 .
Along with this, the first user terminal 20 controls the speaker 2082 and the like to ring to indicate that the voice server (PBX) 60 is receiving an incoming call. The display 2081 of the first user terminal 20 displays information indicating that the voice server (PBX) 60 has received an incoming call from a customer. For example, the display 2081 of the first user terminal 20 may display the characters "Incoming Call".

The first user terminal 20 receives a user's response operation. The response operation is, for example, by lifting the handset (not shown) of the first user terminal 20 or by operating the mouse 2066 to press the button labeled "answer the call" on the display 2081 of the first user terminal 20. It is realized by the operation of
Upon receiving the response operation, the first user terminal 20 transmits a response request to the voice server (PBX) 60 via the CRM system 50 and the server 10 . The voice server (PBX) 60 receives the transmitted response request and establishes voice communication. This allows the first user terminal 20 to interact with the second user terminal 30 .
The display 2081 of the first user terminal 20 displays information indicating that dialogue is taking place. For example, the display 2081 of the first user terminal 20 may display the characters "interacting".

<Modified example of outgoing call processing and incoming call processing>
The method for allowing the first user to interact with the second user is not limited to outgoing call processing and incoming call processing, and any method for realizing interaction between the first user and the second user can be used. I don't mind. For example, a virtual dialogue space called a room for dialogue between a first user and a second user is created on the server 10, and the first user and the second user enter the room. A method of enabling interaction by accessing via a web browser or an application program stored in the terminal 20 or the second user terminal 30 may also be used. In this case, the voice server (PBX) 50 becomes unnecessary.
Specifically, the first user, who is the organizer of the dialogue, operates the input device 206 of the first user terminal 20 to transmit a request for holding the dialogue to the server 10 . Upon receiving the request, the control unit 104 of the server 10 issues room identification information such as a unique room ID, and transmits a response to the first user terminal 20 . The first user transmits the received room identification information to the second user who is the conversation partner by any means of communication such as e-mail or chat. The first user can enter the room by operating the input device 206 of the first user terminal 20, accessing the URL of the server 10 providing room-related services using a web browser, etc., and entering the room identification information. Similarly, the second user can enter the room by operating the input device 306 of the second user terminal 30, accessing the URL of the server 10 providing room-related services using a web browser, etc., and entering the room identification information. . As a result, the first user and the second user can have a dialogue via the first user terminal 20 and the second user terminal 30, respectively, in a virtual dialogue space called a room associated with the room identification information. can be done.
By inputting room identification information, in addition to the first and second users, one or more other users can enter one room. As a result, three or more users can interact via their respective user terminals in a virtual interaction space called a room associated with the room identification information.

<Video dialogue>
The system 1 in the present disclosure may provide an online interactive service (video interactive service) including moving image data. For example, the control unit 204 of the first user terminal 20 and the control unit 304 of the second user terminal 30 send video data captured by the camera 2061 of the first user terminal 20 and the camera 3061 of the second user terminal 30 to the server, respectively. 10.
Based on the received moving image data, the server 10 transmits the moving image data captured by the camera 2061 of the first user terminal 20 to the second user terminal 30, and transmits the moving image data captured by the camera 3061 of the second user terminal 30 to the first user terminal 30. Send to the user terminal 20 . The control unit 204 of the first user terminal 20 displays the received video data captured by the camera 3061 of the second user terminal 30 on the display 2081 . The control unit 304 of the second user terminal 30 displays the received video data captured by the camera 2061 of the first user terminal 20 on the display 3081 .
The server 10 may transmit video data of some or all of the multiple users participating in the online dialogue to the first user terminal 20 and the second user terminal 30 . In this case, the control unit 204 of the first user terminal 20 arranges the video data of some or all of the received plural users participating in the online dialogue on one screen and displays it on the display 2081 of the first user terminal 20. . Thereby, it is possible to confirm the conversation status of a plurality of users participating in the online conversation. A similar process may be executed in the second user terminal 30 as well.

<Dialogue Amnestics>
The interaction storage process is the process of storing data relating to interactions between users and customers.

<Outline of dialogue memory processing>
The dialogue storage process is a series of processes for storing data related to dialogue in the dialogue table 1014 when dialogue is started between the user and the customer.

<Details of dialogue memory processing>
When a dialogue is started between the user and the customer, the voice server (PBX) 60 records voice data relating to the dialogue between the user and the customer and transmits the data to the server 10 . Upon receiving the voice data, the control unit 104 of the server 10 creates a new record in the dialogue table 1014 and stores data relating to the dialogue between the user and the customer. Specifically, the control unit 104 of the server 10 stores the user ID, the customer ID, the dialogue category, the reception/transmission type, and the contents of the voice data in a new record of the dialogue table 1014 .

The control unit 104 of the server 10 acquires the first user ID 2011 of the first user from the first user terminal 20 in the outgoing call process or the incoming call process, and stores it in the user ID item of the new record in the dialogue table 1014 .
The control unit 104 of the server 10 makes an inquiry to the CRM system 50 based on the telephone number in the outgoing call processing or incoming call processing. The CRM system 50 obtains the customer ID by searching the customer table 5012 by telephone number, and transmits it to the server 10 . The control unit 104 of the server 10 stores the acquired customer ID in the customer ID field of the new record of the interaction table 1014 .
The control unit 104 of the server 10 stores the dialogue category value set in advance for each user or customer in the dialogue category field of the new record in the dialogue table 1014 . The dialogue category may be stored by the user selecting and inputting a value for each dialogue.
The control unit 104 of the server 10 identifies whether the ongoing dialogue is originated by the user or originated by the customer, and adds outbound (from the user Outgoing) or inbound (originating from the customer).

The control unit 104 of the server 10 stores the voice data received from the voice server (PBX) 60 in the voice data item of the new record of the interaction table 1014 . The voice data may be stored as a voice data file in another location, and the reference information (path) to the voice data file may be stored after the dialogue is finished. Also, the control unit 104 of the server 10 may be configured to store voice data after the dialogue is finished.

In the video dialogue service, the control unit 104 of the server 10 stores the moving image data received from the first user terminal 20 and the second user terminal 30 in the moving image data item of the new record of the dialogue table 1014. . The moving image data may be stored as a moving image data file in another location, and reference information (path) to the moving image data file may be stored after the dialogue is finished. Also, the control unit 104 of the server 10 may be configured to store the moving image data after the dialogue ends.

<Emotion Analysis Processing>
Emotion analysis processing analyzes dialogue information such as voice and video of online dialogue conducted by multiple users, identifies the emotional state of users participating in the dialogue, and identifies label information based on the emotional state. , are stored in association with dialogue information.

<Overview of Emotion Analysis Processing>
When an online dialogue between users is detected, the emotion analysis process stores dialogue information related to the dialogue, and converts the audio data and video data included in the dialogue information into segment data such as segment audio data and segment video data for each utterance segment, respectively. This is a series of processes of dividing, calculating the emotion feature amount for each section data, identifying label information based on the emotion feature amount, and storing the label information in association with dialogue information.

<Details of emotion analysis processing>
Details of the emotion analysis processing will be described below.

In step S101, an online dialogue between the user and the customer is started through the already-described outgoing call processing, incoming call processing, room, and the like.

In step S102, the emotion analysis unit 1042 of the server 10 executes a receiving step of receiving voice data regarding dialogue.
Specifically, the first user terminal 20 transmits the first user ID 2011 , audio data collected from the microphone 2062 , and moving image data captured by the camera 2061 to the server 10 by the dialogue storage process. The control unit 104 of the server 10 stores the received first user ID 2011, audio data, and moving image data in the user ID, audio data, and moving image data items of the new record in the interaction table 1014, respectively.
Similarly, the second user terminal 30 transmits the second user ID 3011 , audio data collected from the microphone 3062 , and video data captured by the camera 3061 to the server 10 . The control unit 104 of the server 10 stores the received second user ID 3011, audio data, and moving image data in the user ID, audio data, and moving image data items of the new record in the interaction table 1014, respectively.
Along with this, a new dialogue ID is numbered and stored in the dialogue ID field of the new record in the dialogue table 1014 .

In step S103, the emotion analysis unit 1042 of the server 10 executes a voice extraction step of extracting a plurality of segmental voice data for each utterance segment from the voice data received in the receiving step.
Specifically, the emotion analysis unit 1042 of the server 10 acquires (accepts) the dialogue ID, audio data, and video data stored in the dialogue table 1014 in step S102. The emotion analysis unit 1042 of the server 10 detects a section (speech section) in which speech exists from the acquired (accepted) audio data and video data, and generates audio data and video data for each speech section. It is extracted as section audio data and section video data. The segment audio data and segment video data are associated with the user ID of the speaker, the start date and time of the utterance segment, and the end date and time of the utterance segment for each utterance segment.
The emotion analysis unit 1042 of the server 10 performs text recognition on the utterance contents of the extracted segmental voice data and segmental video data, thereby transforming the segmental voice data and segmental video data into segmental reading text, which is characters (text). Convert to and transcribe. Note that the specific method of text recognition is not particularly limited. For example, it may be converted by signal processing technology, machine learning using AI (artificial intelligence), deep learning, or the like.

The emotion analysis unit 1042 of the server 10 processes the dialogue ID to be processed, the speaker's user ID (first user ID 2011 or second user ID 3011), start date and time, end date and time, segment audio data, segment video data, segment reading text are stored in the dialog ID, speaker ID, start date/time, end date/time, segment voice data, segment video data, and segment reading text of new records in the voice segment table 1016, respectively.

In the voice segment table 1016, the segment reading text for each utterance segment of the voice data is associated with the start date and time and the speaker and stored as continuous time-series data. By checking the read-aloud text for the section stored in the speech section table 1016, the user can check the content of the dialogue as text information without checking the content of the speech data.

In addition, during the text recognition process, information such as fillers included in the text that is meaningless in terms of understanding the dialogue between the user and the customer is removed from the text in advance, and the speech recognition information is converted to speech segments. It may be configured to be stored in the table 1016 .

In step S104, the emotion analysis unit 1042 of the server 10 calculates a plurality of emotional feature amounts relating to the emotional state of the speaker in the segmental speech data corresponding to each of the plurality of segmental speech data extracted in the speech extraction step. Perform calculation steps. The emotion calculation step calculates an emotion feature amount as output data by applying the interval speech data extracted in the speech extraction step as input data to a learning model.
Specifically, the emotion analysis unit 1042 of the server 10 acquires the segment voice data, segment video data, and segment reading text stored in the voice segment table 1016 in S103, and applies them to the emotion evaluation model 1031 as input data. The emotion evaluation model 1031 outputs an emotion feature quantity corresponding to the input data as output data.

In step S104, the emotion calculation step executes a step of calculating an emotion vector indicating intensity related to multidimensional emotion corresponding to each of the plurality of segment voice data extracted in the voice extraction step.
Specifically, the emotion analysis unit 1042 of the server 10 acquires the segment voice data, segment video data, and segment reading text stored in the voice segment table 1016 in S103, and applies them to the emotion evaluation model 1031 as input data. The emotion evaluation model 1031 outputs, as output data, emotion vectors quantitatively represented as intensities and numerical values for each of a plurality of emotional states (dimensions) according to input data.

In the emotion calculation step, based on the calculated emotion vector, a step of calculating an emotion scalar representing a one-dimensional emotion-related intensity corresponding to each of the plurality of segmental speech data extracted in the speech extraction step is executed.
The emotion analysis unit 1042 of the server 10 applies a principal component analysis, a learning model such as a deep learning model, a calculation for each component of the emotion vector, etc. to the emotion vector, thereby obtaining an emotion that indicates the intensity of one-dimensional emotion. Compute a scalar. For example, the emotion scalar is an index that quantitatively expresses the degree of positivity or negativity of the speaker's emotional state in the speech segment information, normalized to a range of values from +1 (positive) to -1 (negative). Numerical data may also be used.

The emotion analysis unit 1042 of the server 10 stores the emotion vector and emotion scalar, which are the calculated emotion feature amounts, in the emotion data item of the record to be analyzed in the voice segment table 1016 . Either an emotion vector or an emotion scalar may be stored in the item of emotion data.

In step S104, the emotion analysis unit 1042 of the server 10 retrieves the user ID of the user table 1012 based on the speaker ID of the analysis target record of the speech segment table 1016, and acquires the user attribute.

In step S105, the emotion analysis unit 1042 of the server 10 executes a label identification step of identifying label information for the dialogue based on the plurality of emotion feature amounts calculated in the emotion calculation step.
Specifically, the emotion analysis unit 1042 of the server 10 searches for the dialogue ID in the voice segment table 1016 based on the dialogue ID, and acquires the item of emotion data. The emotion analysis unit 1042 of the server 10 searches the emotion condition master 1021 for the presence or absence of a record corresponding to the emotion condition based on the emotion data, and acquires the item of the label data of the corresponding record.
In the present disclosure, the emotion analysis unit 1042 of the server 10 calculates each of a plurality of speech segment information extracted from one piece of dialogue information, and calculates a plurality of emotions corresponding to a plurality of stored emotion data. A configuration may be adopted in which label data is specified and acquired by using the feature amount as the emotion condition.

In step S105, the label identification step executes a step of identifying label information for the dialogue based on the plurality of emotion scalars calculated in the emotion calculation step.
Specifically, the emotion analysis unit 1042 of the server 10 calculates each of the plurality of voice segment information extracted for one piece of dialogue information, and calculates the emotion scalar included in the plurality of stored emotion data as follows: Label data may be specified as the emotional condition.

In step S105, the label identification step executes a step of identifying label information for the dialogue based on the plurality of emotion vectors calculated in the emotion calculation step.
Specifically, the emotion analysis unit 1042 of the server 10 calculates each of the plurality of voice segment information extracted for one piece of dialogue information, and the emotion vectors included in the plurality of stored emotion data are calculated as follows: Label data may be specified as the emotional condition. For example, the emotional condition may be specified by a range or the like for each element component of the emotion vector.

In step S105, the label identification step executes a step of identifying label information for the dialogue based on the number of emotion feature amounts equal to or greater than or equal to a predetermined threshold among the plurality of emotion feature amounts calculated in the emotion calculation step. .
Specifically, it is assumed that a predetermined threshold value and the number of pieces of information equal to or greater than the threshold value (predetermined number) are stored in the emotional condition item of the emotional condition master 1021 . The emotion analysis unit 1042 of the server 10 compares the value of the emotion scalar corresponding to each of the plurality of speech segment information extracted for one piece of dialogue information with a predetermined threshold, and Count the number of (emotion scalar). It should be noted that the number below a predetermined threshold may be counted.
The emotion analysis unit 1042 of the server 10 determines that the emotion condition corresponds to the emotion condition when the number of counted voice segment information is larger than a predetermined number, and the label data associated with the emotion condition in the emotion condition master 1021 is stored. Get and identify an item.
For example, when the number of pieces of speech segment information (emotion scalar) equal to or greater than a predetermined threshold is greater than a predetermined number, label information indicating that the emotional state in the dialogue is positive is specified. Similarly, when the number of pieces of speech segment information (emotion scalar) below a predetermined threshold is greater than a predetermined number, label information indicating that the emotional state in the dialogue is negative is specified.

In step S105, the label identification step executes a step of identifying label information for the dialogue based on the ratio of the emotion feature amounts above or below a predetermined threshold among the plurality of emotion feature amounts calculated in the emotion calculation step. do.
Specifically, it is assumed that a predetermined threshold value and information on the ratio of the threshold value or more (predetermined ratio) are stored in the emotional condition item of the emotional condition master 1021 . The emotion analysis unit 1042 of the server 10 compares the value of the emotion scalar corresponding to each of the plurality of speech segment information extracted for one piece of dialogue information with a predetermined threshold, and Count the number of (emotion scalar). It should be noted that the number below a predetermined threshold may be counted.
The emotion analysis unit 1042 of the server 10, when the ratio of the counted number of voice segment information to the number of all voice segment information extracted for one piece of dialogue information is greater than a predetermined percentage, It is determined that the condition is met, and the item of label data associated with the emotional condition is acquired and specified in the emotion condition master 1021 .
For example, if the rate of speech segment information (emotion scalar) equal to or greater than a predetermined threshold is greater than a predetermined rate, label information indicating that the emotional state in the dialogue is positive is specified. Similarly, when the ratio of speech segment information (emotion scalar) below a predetermined threshold is greater than a predetermined ratio, label information indicating that the emotional state in the dialogue is negative is specified.

In addition, instead of the emotion scalar, one element component included in the emotion vector, an index calculated based on one or more element components included in the emotion vector, etc. are regarded as emotion feature amounts, and similar processing is performed. I don't mind.

In step S105, the label identification step executes a step of identifying label information for the dialogue based on the statistical values of the plurality of emotion feature quantities calculated in the emotion calculation step.
Specifically, it is assumed that predetermined threshold information is stored in the emotion condition item of the emotion condition master 1021 . The emotion analysis unit 1042 of the server 10 calculates the average value, the median value, the average value such as the mode value, the maximum value, Statistical values such as the minimum value are calculated, compared with a predetermined threshold value, and if it is equal to or greater than the predetermined threshold value, it is determined that the emotional condition is applicable, and the item of the label data associated with the emotional condition in the emotional condition master 1021 is selected. Obtain and identify. It should be noted that the condition may be the case where it is equal to or less than a predetermined threshold value.

In step S105, the label identification step executes a step of identifying label information for the dialogue based on time-series changes in the plurality of emotion feature quantities calculated in the emotion calculation step.
The label identification step includes a step of performing regression analysis on time-series changes in the multiple emotion feature quantities calculated in the emotion calculation step, and label information for the dialogue based on the regression coefficients obtained as a result of the regression analysis. identifying.
Specifically, it is assumed that the range of regression coefficients is stored in the emotion condition item of the emotion condition master 1021 . In the target dialogue data, for each of the plurality of speech segment information associated with the dialogue data, the start date and time of the speech segment information, the end date and time, the value of any date and time between the start date and time and the end date and time on the X axis, Regression analysis of Y=f(X) is performed when the Y axis is the value of the emotion scalar included in the emotion data of the speech segment information. Any regression analysis such as linear regression or quadratic regression may be applied as the regression analysis. A regression coefficient is calculated by performing a regression analysis, compared with the range of the regression coefficient, and if it is within the range of the regression coefficient, it is determined that the emotional condition corresponds, and the label associated with the emotional condition in the emotional condition master 1021. Acquire and identify an item of data.
For example, in the case of linear regression (first-order regression), when the intercept is negative and the slope is positive, label information indicating that the emotional state in the dialogue is improving is specified.
In addition, instead of the emotion scalar, one element component included in the emotion vector, an index calculated based on one or more element components included in the emotion vector, etc. are regarded as emotion feature amounts, and similar processing is performed. I don't mind.

In step S105, the emotion analysis unit 1042 of the server 10 identifies a first emotion group, which is a set of a plurality of emotion feature amounts corresponding to a plurality of time-series continuous segmental voice data extracted in the voice extraction step. to run. The emotion analysis unit 1042 of the server 10 executes a step of identifying a second emotion group, which is a set of a plurality of emotion feature amounts corresponding to a plurality of time-series continuous segmental voice data extracted in the voice extraction step.
Specifically, the emotion analysis unit 1042 of the server 10 divides a plurality of speech segment information extracted from one piece of dialogue information into segment groups each composed of a plurality of speech segment information, and for each segment group: may perform the label identification step already described. As a result, label information corresponding to each of the plurality of section groups is specified.
For example, the emotion analysis unit 1042 of the server 10 calculates an emotion scalar for each of a plurality of pieces of extracted speech segment information included in the segment group and stores it in emotion data. The label data may be specified using the emotion scalar included in the plurality of stored emotion data as the emotion condition.
For example, the emotion analysis unit 1042 of the server 10 calculates an emotion vector for each of a plurality of pieces of extracted voice segment information included in the segment group and stores it in emotion data. Emotion vectors included in the plurality of stored emotion data may be used as emotion conditions to identify label data.

In step S105, the label identification step includes a step of identifying first label information for the dialogue based on the plurality of emotion feature amounts included in the first emotion group; , identifying second label information for the interaction.
Specifically, the emotion analysis unit 1042 of the server 10 divides a plurality of speech segment information extracted from one piece of dialogue information into segment groups each composed of a plurality of speech segment information, and for each segment group: Label information corresponding to each of the plurality of section groups is specified by executing the label specifying step already described in .

In step S105, the emotion analysis unit 1042 of the server 10 executes a label presentation step of presenting the first label information and the second label information to the first user.
Specifically, the emotion analysis unit 1042 of the server 10 transmits the identified first label information and second label information to the first user terminal 20 . The control unit 204 of the first user terminal 20 displays the received first label information and second label information on the display 2081 of the first user terminal 20 to present them to the first user. Note that the first label information and the second label information may be presented to arbitrary users such as the second user, other administrators, and other users.

In step S105, the emotion analysis unit 1042 of the server 10 performs a selection receiving step of receiving a selection instruction from the first user to select at least one of the first label information and the second label information presented in the label presenting step. Run.
Specifically, the first user operates the input device 206 or the like of the first user terminal 20 to select either the first label information or the second label information presented on the display 2081 of the first user terminal 20. choose one. Note that the first user may select none of them. The control unit 204 of the first user terminal 20 transmits the selected label information to the server 10 . Emotion analysis unit 1042 of server 10 identifies the received label information.

In step S105, in the label specifying step, the plurality of emotion feature amounts calculated in the emotion calculation step, and the user attributes of the first user or the second user who uttered the segmental speech data corresponding to the plurality of emotion feature amounts. Based on this, perform the step of identifying label information for the interaction.
Specifically, when identifying the label information, the emotion analysis unit 1042 of the server 10 may consider the user attributes of the first user and the second user identified in step S104 to identify the label information. For example, the emotion conditions in the emotion condition master 1021 may include the user attributes of the first user and the second user as conditions.

In step S105, the label specifying step corresponds to the segmental voice data of the first user's utterance based on the plurality of emotion feature values corresponding to the segmental voice data of the second user's utterance calculated in the emotion calculating step. A step of identifying label information for a dialogue is performed without considering a plurality of emotion features.
Specifically, the emotion analysis unit 1042 of the server 10 excludes the voice segment information whose speaker ID is the first user ID 2011 from among the plurality of voice segment information extracted for one piece of dialogue information. The already-described label identification step may be executed based only on the voice section information whose ID is the second user ID 3011 .
This makes it possible to specify label information that takes into account only the customer's emotional state. Generally, the first user, such as an operator, is interested in the customer's emotional state rather than his or her own emotional state. By adopting such a configuration, it is possible to specify label information that particularly considers the customer's emotional state.

The emotion analysis unit 1042 of the server 10 excludes the speech section information whose speaker ID is the second user ID 3011 from among the plurality of speech section information extracted for one piece of dialogue information, The already-described label specifying step may be executed based only on the voice section information of the user ID 2011 .

The emotion analysis unit 1042 of the server 10 executes the already described label identifying step for each of the voice segment information whose speaker ID is the first user ID 2011 and the voice segment information whose speaker ID is the second user ID 3011. Then, a plurality of label information of the first label information and the second label information may be specified respectively.

In addition, the emotion analysis unit 1042 of the server 10 extracts a plurality of pieces of voice segment information extracted for one piece of dialogue information. Information may be excluded, and the already described label identification step may be performed based only on speech segment information for which the user identified by the speaker ID is not the host user.
This allows label information to be specified without considering the emotional state of the host of the dialogue. It is common for dialogue organizers to be more interested in the emotional state of the interlocutor than in their own emotional state. With such a configuration, it is possible to specify label information that takes into consideration the emotional state of the conversation partner.

In step S106, the emotion analysis unit 1042 of the server 10 executes a storage step of storing the label information identified in the label identification step in association with the dialogue.
Specifically, the emotion analysis unit 1042 of the server 10 stores the label information specified in step S105 in the label data item of the label table 1015 in association with the dialogue ID numbered in step S101.
In step S105, the specified label information may be presented to the first user, and the label information for which the selection instruction is received from the first user may be stored as label data in the label table 1015. FIG.

In step S106, the storing step executes a step of storing the first label information or the second label information specified in the label specifying step in association with the interaction. The storing step stores at least one of the first label information and the second label information in association with the dialogue based on the selection instruction received from the first user in the selection receiving step.
Specifically, label information for which a selection instruction is received from the first user may be stored as label data in the label table 1015 .

Further, the first user can display the label information stored in the label table 1015 from the server 10 on the display 2081 of the first user terminal 20 by operating the input device 206 of the first user terminal 20. .

<Regarding the execution timing of emotion analysis processing>
Steps S103 to S106 of the emotion analysis process may be configured to be executed after the online dialogue by a plurality of users is finished. As a result, after the online dialogue ends and the contents of the dialogue are determined, the label information corresponding to the emotional state of the user in the dialogue is identified and stored in association with the dialogue information.

Also, the emotion analysis processing may be configured to be executed after the start of online dialogue by a plurality of users and before the end of the dialogue.
In other words, it may be configured to be executed at an arbitrary timing during online dialogue between a plurality of users. Also, steps S103 to S106 may be configured to be periodically executed in real time during online dialogue. As a result, even during the online dialogue, the label information corresponding to the emotional state of the user in the previous dialogue may be identified and stored in association with the dialogue information.
As a result, the user can confirm the emotional state of the user participating in the online dialogue in real time during the online dialogue, and organize and manage the dialogue information based on the latest emotional state.

<Impression analysis processing>
Impression analysis processing analyzes dialogue information such as voice and video of online dialogue conducted by multiple users, identifies the impression state of the users participating in the dialogue, and informs the user of the impression state and speaker type. This is the processing to be presented.

<Overview of Impression Analysis Processing>
When an online dialogue between users is detected, the impression analysis process stores the dialogue information related to the dialogue, and converts the audio data and video data included in the dialogue information into segment data such as segment audio data and segment video data for each utterance segment, respectively. This is a series of processes of dividing, calculating the impression feature amount for each section data, identifying the speaker type based on the impression feature amount, and presenting the identified speaker type to the user.

<Details of impression analysis processing>
Details of the impression analysis processing will be described below.

In step S301, an online dialogue between the user and the customer is started through the already explained outgoing call processing, incoming call processing, room, and the like.

In step S302, the impression analysis unit 1043 of the server 10 executes a dialog acquisition step of acquiring dialog information from the second user regarding the interaction response with the first user.
Since step S302 is the same as step S102 in the emotion analysis process, the description is omitted.

In step S303, the impression analysis unit 1043 of the server 10 executes a voice extraction step of extracting a plurality of segment voice data for each utterance segment from the voice data of the second user received in step S302.
Since step S303 is the same as step S103 in the emotion analysis process, the explanation is omitted.

In step S304, the impression analysis unit 1043 of the server 10 calculates an impression feature amount relating to the impression given by the second user to other users in the dialogue based on the dialogue information of the second user acquired in the dialogue acquisition step. Perform calculation steps. In the impression calculation step, based on the dialogue information acquired from the second user in the dialogue acquisition step, at least any one of like, dislike, loud, hard to hear, polite, difficult to understand, timid, nervous, overbearing, violent and sexual. Alternatively, a step of calculating an impression feature quantity indicating the strength of one impression is executed.
In the impression calculation step, dialogue information obtained from the second user in the dialogue acquisition step is used as input data, and is applied to a learning model, thereby outputting an impression feature quantity relating to the impression given by the second user to other users in the dialogue. Execute the step of computing as data.
Specifically, the impression analysis unit 1043 of the server 10 acquires the segment voice data, segment video data, and segment reading text stored in the voice segment table 1016 in S303. The speech section information with the user ID 2011 is excluded, and only the speech section information with the speaker ID of the second user ID 3011 is applied to the impression evaluation model 1032 as input data. as the output data. Thereby, the impression given by the second user can be evaluated by the impression feature amount.
Note that the input data applied to the impression evaluation model 1032 excludes the speech section information whose speaker ID is the second user ID 3011 among the speech section information, and the speech section information whose speaker ID is the first user ID 2011. good. In this case, the impression given by the first user can be evaluated by the impression feature amount.

In step S304, the impression calculation step includes calculating a dialogue feature amount related to the second user's way of speaking in the dialogue based on the second user's dialogue information acquired in the dialogue acquisition step, and based on the calculated dialogue feature amount, and calculating an impression feature amount.
The impression calculation step is a step of applying the dialogue information of the second user acquired in the dialogue acquisition step as input data to the first learning model, thereby calculating, as output data, a dialogue feature quantity relating to the manner of speaking of the second user in the dialogue. and calculating an impression feature amount by applying the calculated dialogue feature amount as input data to a second learning model.
In the impression calculation step, based on the second user's dialogue information acquired in the dialogue acquisition step, at least the second user's speaking speed, intonation, the number of polite expressions, the number of fillers, and the number of grammatical utterances in the dialogue. It includes the step of calculating dialogue features for any one of the speaking styles.

Specifically, the impression analysis unit 1043 of the server 10 acquires the segment voice data, segment video data, and segment reading text stored in the voice segment table 1016 in S303. The speech section information with the user ID 2011 is excluded, and only the speech section information with the speaker ID of the second user ID 3011 is applied as input data to the first impression evaluation model 1033, and the first impression evaluation model 1033 It outputs the dialogue feature quantity obtained as output data.
The impression analysis unit 1043 of the server 10 applies the dialogue feature amount as input data to the second impression evaluation model 1034, and the second impression evaluation model 1034 outputs the impression feature amount according to the input data as output data. Thereby, the impression given by the second user can be evaluated by the impression feature amount.
Note that the input data applied to the impression evaluation model 1032 excludes the speech section information whose speaker ID is the second user ID 3011 among the speech section information, and the speech section information whose speaker ID is the first user ID 2011. good. In this case, the impression given by the first user can be evaluated using the impression feature amount.

In step S304, the impression analysis unit 1043 of the server 10 executes a storage step of storing the impression feature amount calculated in the impression calculation step in association with the second user.
Specifically, the impression analysis unit 1043 of the server 10 stores the calculated impression feature amount in the impression data item of the record to be analyzed in the voice segment table 1016 . As a result, the impression feature amount is stored in association with the second user via the speaker ID (second user ID) of the voice segment table 1016 . Note that the impression feature amount may be stored in association with the second user ID by providing a column for storing impression data (not shown) in the customer table 5012 of the CRM system 50 . Further, the impression feature amount may be stored in association with the second user ID by providing a column for storing impression data (not shown) in the user table 1012 of the server 10 .
By storing it in the customer table 5012 of the CRM system 50, it is possible to share the impression feature amount of the user specified in the target dialogue with members of other departments in the company. For example, efficient work can be performed according to the impression of the dialogue partner specified by the impression feature amount.

In step S305, the impression analysis unit 1043 of the server 10 performs an identification step of identifying the speaker type labeled with the impression given by the second user to other users based on the impression feature amount calculated in the impression calculation step. Run.
Specifically, the impression analysis unit 1043 of the server 10 searches for the dialogue ID in the voice segment table 1016 based on the dialogue ID, and acquires the items of impression data. Based on the impression data, the impression analysis unit 1043 of the server 10 searches the speaker type master 1022 for records corresponding to the impression condition, and acquires the speaker type item of the corresponding record.
In the present disclosure, the impression analysis unit 1043 of the server 10 calculates each of the plurality of speech segment information extracted for one piece of dialogue information, and calculates the impression feature amount related to the plurality of stored impression data. , the speaker type may be identified and acquired as the impression condition.

In step S305, the impression analysis unit 1043 of the server 10 executes a storage step of storing the speaker type specified in the specifying step in association with the second user.
Specifically, the impression analysis unit 1043 of the server 10 transmits the specified speaker type and second user ID to the CRM system 50 . The control unit 504 of the CRM system 50 stores the received speaker type and second user ID in the speaker type and user ID items of the customer table 5012, respectively. That is, the specified speaker type is stored in association with the user ID of the user who spoke in the dialogue.
By storing in customer table 5012 of CRM system 50, the user's speaker type identified in the conversation of interest can be shared with members of other departments within the company. For example, efficient reception work can be performed according to the speaker type of the conversation partner.
In the present disclosure, the user's speaker type is stored in the customer table 5012 of the CRM system 50, but may be stored in the user table 1012 of the server 10 in association with the second user.

In step S306, the impression analysis unit 1043 of the server 10 executes a presentation step of presenting to the first user the impression feature amount stored in association with the second user in the storage step.
Specifically, the impression analysis unit 1043 of the server 10 transmits the impression feature quantity specified in step S305 to the first user terminal 20 . The control unit 204 of the first user terminal 20 displays the received impression feature quantity on the display 2081 of the first user terminal 20 to present it to the first user. Note that the impression feature amount may be presented to any user such as the second user, other administrators, or other users.

In step S306, the impression analysis unit 1043 of the server 10 stores the first user in association with the second user in the storage step prior to the interaction between the first user and the second user. A presentation step of presenting the impression feature quantity is executed.
For example, when the first user or another user starts an online dialogue with the second user via outgoing call processing, incoming call processing, rooms, etc., the display 2081 of the first user terminal 20 displays , an outgoing call screen for making an outgoing call to the second user, an incoming call screen for receiving an incoming call from the second user, a room screen before starting the dialogue, etc., the second user stored in association with the second user in step S305. may be displayed and presented to the first user.
Thereby, the first user can prepare a response according to the second user's impression before starting the dialogue.

Note that, prior to the dialogue between the first user and the second user, the impression analysis unit 1043 of the server 10 asks the first user for the speaker stored in association with the second user in the storage step. A presenting step of presenting the type may be performed.
For example, when the first user or another user starts an online dialogue with the second user via outgoing call processing, incoming call processing, rooms, etc., the display 2081 of the first user terminal 20 displays , an outgoing call screen for making an outgoing call to the second user, an incoming call screen for receiving an incoming call from the second user, a room screen before starting the dialogue, etc., the second user stored in association with the second user in step S305. may be displayed and presented to the first user.
This allows the first user to prepare a response according to the second user's speaker type before starting the dialogue.

Before the dialogue between the first user and the second user ends, the impression analysis unit 1043 of the server 10 provides the first user with the impression feature amount stored in association with the second user in the storage step. may be performed.
For example, while the first user or another user is having an online dialogue with the second user, the interactive screen, room screen, etc. displayed on the display 2081 of the first user terminal 20 may display step S305 may display the second user's impression feature quantity stored in association with the second user in and present it to the first user. Note that the impression feature amount may be presented to any user such as the second user, other administrators, or other users.
Thereby, the first user can prepare a response according to the second user's impression during the dialogue.

Before the dialogue between the first user and the second user ends, the impression analysis unit 1043 of the server 10 provides the first user with the speaker type stored in association with the second user in the storage step. may be performed.
For example, while the first user or another user is having an online dialogue with the second user, the interactive screen, room screen, etc. displayed on the display 2081 of the first user terminal 20 may display step S305 may display the second user's speaker type stored in association with the second user in and present it to the first user. Note that the impression feature amount may be presented to any user such as the second user, other administrators, or other users.
Thereby, the first user can prepare a response according to the second user's speaker type during the dialogue.

In the impression calculation step, the impression analysis unit 1043 of the server 10 may execute a presentation step of presenting one or more of the plurality of dialogue feature quantities having a large degree of influence on the impression feature quantity.
Specifically, the impression analysis unit 1043 of the server 10 applies a plurality of dialogue feature amounts as input data to the second impression evaluation model 1034, and the second impression evaluation model 1034 outputs impression feature amounts according to the input data. When outputting as, one or more dialogue feature values that have a large impact on the output impression feature value are specified, and transmitted to the first user terminal 20, the second user terminal 30, other user terminals, etc. and may be presented to the user.
For example, the second impression evaluation model 1034 may output, as output data, one or a plurality of dialogue feature quantities that greatly affect the output impression feature quantity. As a result, it is possible to quickly acquire the dialogue feature that greatly affects the impression feature.

<Modified example of impression analysis processing>
The impression analysis process may be configured to identify the impression state of the first user who is the operator instead of the second user who is the customer.
Alternatively, the first user may receive the desired impression feature amount and the target speaker type that the first user wants to give to other users, calculate the dialogue feature amount to be improved by the first user, and present it to the first user. That is, a step of suggesting a preferred way of speaking to the first user may be included.
In this case, in steps S301 to S305 of the impression analysis process, only the second user is read as the first user, and the processing contents are the same, so description thereof will be omitted.

In step S306, the impression analysis unit 1043 of the server 10 executes a target receiving step of receiving a target speaker type that the first user should give to other users in dialogue.
Specifically, the first user accesses a predetermined web page provided by the server 10 by operating the input device 206 or the like of the first user terminal 20, and selects the target speaker type from the displayed list of speaker types. Select the desired speaker type (target speaker type). The control unit 204 of the first user terminal 20 identifies the selected target speaker type and transmits it to the server 10 . The server 10 receives and accepts the target speaker type. The target speaker type is a speaker type related to an impression state desired by the first user as an impression state to be given to other users. You may select according to a 1st user's job etc.

In step S306, the impression analysis unit 1043 of the server 10 executes a target receiving step of receiving a target impression feature amount that the first user should give to other users in the dialogue.
Specifically, the impression analysis unit 1043 of the server 10 retrieves the speaker type item of the speaker type master 1022 based on the received target speaker type, and acquires the impression condition. Based on the acquired impression conditions, the impression analysis unit 1043 of the server 10 identifies and receives impression feature amounts included in the range of the impression conditions as target impression feature amounts. The impression analysis unit 1043 of the server 10 may be configured to acquire and receive the target impression feature amount output by applying the target speaker type as input data to a learning model (not shown) or the like. Alternatively, the desired impression feature amount may be received from the first user via the input device 206 of the first user terminal 20 or the like.

In step S306, the impression analysis unit 1043 of the server 10 determines the dialogue feature amount to be improved by the first user based on the impression feature amount calculated in the impression calculation step and the target impression feature amount received in the target reception step. Carry out the calculated improvement steps.
Specifically, the impression analysis unit 1043 of the server 10 identifies and accepts the dialogue feature amount for obtaining the target impression feature amount as the target dialogue feature amount based on the identified target impression feature amount. The impression analysis unit 1043 of the server 10 may be configured to acquire and receive a target dialogue feature amount by applying the target impression feature amount as input data to a learning model (not shown) or the like.
Dialogue features to be improved by the first user include, for example, "faster speaking speed", "slower speaking speed", "higher intonation", and "lower intonation". Further, the dialogue feature amount to be improved by the first user may be a target dialogue feature amount (target dialogue feature amount).

The impression analysis unit 1043 of the server 10 compares the dialogue feature amount calculated in step S304 with the target dialogue feature amount. The impression analysis unit 1043 of the server 10 calculates the difference between the dialogue feature amount and the target dialogue feature amount as the dialogue feature amount to be improved by the first user. In addition, the impression analysis unit 1043 of the server 10 compares the dialogue feature amount and the target dialogue feature amount, and identifies the dialogue feature amount with a large divergence as the dialogue feature amount to be improved by the first user.
The impression analysis unit 1043 of the server 10 transmits to the first user terminal 20 the dialogue feature amount to be improved by the first user. The control unit 204 of the first user terminal 20 displays the received dialogue feature amount to be improved on the display 2081 of the first user terminal 20 to present it to the first user.
For example, out of the dialogue feature quantities such as the first user's speaking speed, intonation, the number of polite expressions, the number of fillers, and the number of grammatical utterances in the dialogue, the dialogue feature quantity to be improved by the first user is specified. , speech rate, intonation, the number of polite expressions, the number of fillers, etc., should be improved to the first user. As a result, the operator or the like can improve the impression given to others by specifically improving the manner of speaking.
Note that the dialogue feature amount may be presented to the second user or other users.

As a result, the impression analysis unit 1043 of the server 10 calculates the dialogue feature amount to be improved by the first user based on the speaker type calculated in the impression calculation step and the target speaker type received in the target reception step. improvement steps can be taken.
In other words, the user can grasp the dialogue feature amount to be improved according to the received target speaker type, and improve the speaking style based on the dialogue feature amount to be improved, so that the impression the user gives to others can be determined by the target speaker type. can be brought closer to

<Topic definition processing>
The topic definition process is a process in which a user registers and stores a topic associated with a plurality of keywords and related to a predetermined topic.

<Overview of topic definition processing>
Users can define and memorize new topics based on keywords such as multiple words, nouns, adjectives, and the like. Also, for an already stored topic, a keyword highly related to the topic is presented based on the dialogue information stored in the past, and the keyword is added to the keywords related to the topic and stored. It is a series of processes for expanding the keywords associated with the topic.

<Details of topic definition processing>
Details of the topic definition process will be described below.

Based on the speech data stored in the speech storage step and the plurality of keywords accepted in the keyword acceptance step, the topic processing unit 1044 of the server 10 selects one or more new keywords to be newly associated with the first topic by the first user. perform a keyword presenting step for presenting to
Specifically, the first user operates the input device 206 or the like of the first user terminal 20 to execute the application program 2012 and execute the browser application. The first user inputs a predetermined URL (Uniform Resource Locator) designating a predetermined web server provided by the server 10 in the browser application, thereby making a request to the server 10 for a page for defining a topic. Send.

Based on the first user ID 2011 included in the received request, the topic processing unit 1044 of the server 10 searches for the speaker ID item in the speech segment table 1016 and acquires segment reading text.
The topic processing unit 1044 of the server 10 extracts character strings such as nouns, adjectives, and keywords included in the segment reading text by executing processing such as morphological analysis on the segment reading text. At this time, the degree of importance of the character string may be calculated based on the appearance frequency of the character string for each piece of dialogue information and voice section information. tf-idf and the like are available as methods for calculating the degree of importance. The topic processing unit 1044 of the server 10 identifies a predetermined number of character strings with high importance as keyword candidates.

The topic processing unit 1044 of the server 10 acquires topic IDs and keywords from the topic master 1023, and a plurality of keywords associated with each of the plurality of topic IDs and co-occurrence relationships in one or a plurality of pieces of dialogue information or speech section information. , and is not associated with the topic ID, may be specified as a keyword candidate. In calculating the co-occurrence relationship, the degree of importance of each keyword and character string may be considered. When identifying keyword candidates, a predetermined number of character strings may be identified as keyword candidates in consideration of the degree of importance calculated based on appearance frequency or the like.

The topic processing unit 1044 of the server 10 transmits the identified keyword candidates to the first user terminal 20 . The control unit 204 of the first user terminal 20 displays the received keyword candidate on the display 2081 of the first user terminal 20 and presents it to the first user.

The topic processing unit 1044 of the server 10 executes a keyword acceptance step of accepting one or more keywords from the first user.
Specifically, the first user operates the input device 206 or the like of the first user terminal 20 to select a keyword to be newly associated with the topic from the keyword candidates displayed on the display 2081 of the first user terminal 20. do.
The control unit 204 of the first user terminal 20 transmits the one or more keyword candidates selected by the first user to the server 10 .

The keyword acceptance step executes a step of accepting one or more keywords selected by the first user from among the plurality of new keywords presented to the first user in the keyword presentation step.
Specifically, the topic processing unit 1044 of the server 10 receives and accepts one or more keyword candidates from the first user terminal 20 .

The topic processing unit 1044 of the server 10 executes a topic storage step of storing one or more keywords accepted in the keyword acceptance step in association with a first topic related to a predetermined topic.
Specifically, the topic processing unit 1044 of the server 10 associates a plurality of received keyword candidates with topic IDs and stores them in the topic master 1023 . One or a plurality of keyword candidates selected by the first user may be associated with a topic ID already stored in the topic master 1023, or a new topic ID may be generated and the newly generated topic It is good also as a structure associated with ID.
When storing in association with a topic ID already stored in the topic master 1023, the first user operates the input device 206 or the like of the first user terminal 20 to select the topic ID to be associated. perform an operation.

<Topic analysis processing>
Topic analysis processing analyzes dialogue information such as audio and video of online dialogue conducted by multiple users, calculates the degree of relevance between the dialogue information and one or more topics, and based on the degree of relevance, analyzes the dialogue information This is the process of associating and storing topics.

<Overview of topic analysis processing>
When an online dialogue between users is detected, the topic analysis process stores the dialogue information related to the dialogue, and converts the audio data and video data included in the dialogue information into segment data such as segment audio data and segment video data for each utterance segment, respectively. This is a series of processes of dividing, calculating the degree of relevance to a plurality of topics for each section data, identifying the topic for each section data, and storing representative topics as label information of dialogue information.

<Details of topic analysis processing>
Details of the topic analysis process will be described below.

In step S511, an online dialogue between the user and the customer is started through the already explained outgoing call processing, incoming call processing, room, and the like.

In step S512, the topic processing unit 1044 of the server 10 executes a receiving step of receiving voice data regarding dialogue. The topic processing unit 1044 of the server 10 executes a voice storage step of storing the voice data received in the reception step.
Since step S512 is the same as step S102 in the emotion analysis process, the description is omitted.

In step S513, the topic processing unit 1044 of the server 10 executes a voice extraction step of extracting a plurality of segment voice data for each utterance segment from the voice data received in the receiving step.
Since step S513 is the same as step S103 in the emotion analysis process, the explanation is omitted.

In step S513, the voice extracting step may execute a step of extracting a plurality of segmental voice data for each utterance segment from the voice data received in the receiving step before the dialogue ends.
In other words, the speech extraction step may be configured to be executed at arbitrary timing during online dialogue between a plurality of users.

In step S514, the topic processing unit 1044 of the server 10 performs a topic identification step of identifying a first topic associated with a plurality of keywords and related to a predetermined topic.
Specifically, the topic processing unit 1044 of the server 10 refers to the topic master 1023 to obtain and specify a topic ID registered in advance by the topic definition process and one or more keywords associated with the topic ID.

The degree-of-relevance calculation step executes a step of calculating the degrees of relevance for each of the plurality of topics identified in the topic identification step for each of the plurality of segmental speech data.
In this disclosure, one first topic and one or more keywords associated with the first topic will be mainly described for simplicity, but the topic is not limited to one and may include multiple topics (second topic, second topic, 3 topics . . . ).

In step S514, the topic processing unit 1044 of the server 10 executes a degree-of-relevance calculation step of calculating a first degree of relevance indicating the degree of relevance to the first topic identified in the topic identification step for each of the plurality of segmental audio data. .
Specifically, the topic processing unit 1044 of the server 10 generates a first association index indicating the degree of association with the first topic according to the association between the speech section information acquired in S513 and the keyword associated with the first topic. Calculate degrees.

An example of the method for calculating the first degree of association will be described below. The topic processing unit 1044 of the server 10 creates a high-dimensional vector (topic vector) as distributed representation (embedded representation) based on the keyword associated with the first topic. In addition, the topic processing unit 1044 of the server 10 performs processing such as morphological analysis on the segmented reading text included in the plurality of pieces of speech segment information, thereby identifying nouns, adjectives, keywords, etc. included in the segmented reading text. is extracted, and a high-dimensional vector (speech segment vector) is created as a distributed representation based on the extracted character string. Note that a technique called Word2vec is known as a method for creating a distributed representation. The topic processing unit 1044 of the server 10 calculates the first relevance by calculating the cosine similarity between the topic vector and the speech segment vector. For the first degree of association, any algorithm for calculating the distance between arbitrary multidimensional vectors, such as Euclidean distance, Mahalanobis distance, Manhattan distance, Chebyshev distance, Minkowski distance, etc., may be applied.
The first degree of relevance calculated in this way reflects the general tendency of similarity between the plurality of keywords associated with the first topic and the character strings included in the plurality of speech segment information. As a result, the character strings included in the speech segment information are not determined to be different words with the same meaning due to differences in paraphrasing expressions and notations of the keywords included in the topic, so that the keywords included in the first topic and the semantic content are different. A higher degree of relevance is obtained for speech segment information that is highly relevant.
In the present disclosure, the calculation of the first degree of relevance indicating the degree of relevance to the first topic has been described, but the calculation of the degree of relevance between an arbitrary topic and the speech section information is the same.

The degree-of-relevance calculating step is a step of calculating a first degree of relevance indicating the degree of relevance to the first topic identified in the topic identifying step for each segmental audio data included in the plurality of segmental audio data before the dialogue ends. may be executed.
In other words, it may be configured to be executed at an arbitrary timing during online dialogue between a plurality of users. As a result, even in the middle of an online dialogue, it is possible to calculate the degree of relevance to each topic for speech segment information in the dialogue up to that point.

In the degree-of-relevance calculation step, among the plurality of keywords associated with the first topic, the more keywords included in the plurality of segmental speech data extracted in the speech extraction step, the smaller the weight given to the degree of relevance. A degree of matching that takes into consideration the weighting of a plurality of keywords associated with the first topic for each piece of segmental audio data may be calculated as the first degree of relevance indicating the degree of relevance with the first topic.
Specifically, different weights may be applied to the importance of each of the plurality of keywords associated with the first topic when calculating the degree of association. For example, for a plurality of pieces of speech segment information extracted for one piece of dialogue information, the importance and weight of keywords frequently appearing in many pieces of speech segment information may be reduced so as to reduce the degree of influence on the degree of relevance. may be set to a value smaller than the number of keywords. As a result, it is possible to prevent overestimation of the degree of relevance to topics associated with common keywords that frequently appear in a lot of speech segment information.
In the present disclosure, calculation of the first degree of relevance, which indicates the degree of relevance to the first topic, has been described.

In the degree-of-relevance calculation step, among the plurality of keywords associated with the first topic, a large number of the plurality of section-speech data up to a predetermined number in chronological order from the target section-sound data for which the first degree of relevance is to be calculated. The degree of association with the first topic is indicated by weighting the degree of relevance for each of the plurality of segmental audio data, taking into consideration the weighting of the plurality of keywords associated with the first topic, for each of the plurality of segmental audio data. It may be calculated as the first degree of association.
For example, instead of all of a plurality of speech segment information extracted from one piece of dialogue information, many of the speech segment information up to a predetermined number in chronological order from the target segment speech information to be calculated are extracted. The importance and weight may be set to values smaller than those of other keywords so that the degree of influence of the keywords frequently appearing in the speech section information on the relevance is small. This makes it possible to more accurately calculate the degree of relevance between the most recent speech segment information and the topic even at any timing during the dialogue before the dialogue ends.
In the present disclosure, calculation of the first degree of relevance, which indicates the degree of relevance to the first topic, has been described.

The topic processing unit 1044 of the server 10 calculates the degree of relevance calculated for each of the plurality of topics with respect to the plurality of voice segment information extracted from one piece of dialogue information, the segment ID specifying the voice segment information, the topic and the calculated degree of relevance are respectively stored in the section ID, topic ID, and degree of relevance items of a new record in the topic relevance table 1017 .

In step S515, among one or a plurality of topics having a degree of relevance equal to or greater than a predetermined value in each speech segment information, the topic with the highest degree of relevance is specified as a topic related to the predetermined topic referred to by the speech segment information. Note that the topic does not necessarily have to be specified. The topic processing unit 1044 of the server 10 stores the topic ID of the identified topic in the topic ID field of the record identified by the section ID of the speech section information for which the degree of association is to be calculated in the speech section table 1016 . As a result, the speech section information is stored in association with the topic with a high degree of relevance.

In step S516, the topic processing unit 1044 of the server 10 executes a label identification step of identifying label information for the dialogue based on the degree of relevance for each of the plurality of topics calculated in the degree of relevance calculation step. The topic processing unit 1044 of the server 10 executes a storage step of storing the label information identified in the label identification step in association with the dialogue.
Specifically, in step S515, the topic processing unit 1044 of the server 10 aggregates the topic IDs stored for each of the plurality of voice segment information extracted for one piece of dialogue information, and the aggregated topic One or a plurality of topic IDs are specified in descending order of IDs as topics that characterize the one piece of dialogue information. Note that one or a plurality of topic IDs whose total number of topic IDs is equal to or greater than a predetermined number may be specified as the topic that characterizes the one item of dialogue information.
The topic processing unit 1044 of the server 10 identifies the topic name of the identified topic ID and the name of the topic, such as a label, as label information. Note that a table (not shown) or the like may be referred to, and arbitrary label information may be specified based on the specified topic ID.
The specified label information and the dialogue ID of the one piece of dialogue information are stored in the label data and dialogue ID fields of a new record in the label table 1015 . As a result, the dialogue information and the topic characterizing the dialogue information are associated and stored as label information, which can be conveniently used when retrieving the dialogue information.

<Regarding the execution timing of topic analysis processing>
Steps S513 to S516 of the topic analysis process may be configured to be executed after the online dialogue by a plurality of users is finished. As a result, after the online dialogue ends and the contents of the dialogue are determined, the topic related to the dialogue is identified and stored in association with the dialogue information.

Also, the topic analysis processing may be configured to be executed after the start of online dialogue by a plurality of users and before the end of the dialogue.
In other words, it may be configured to be executed at an arbitrary timing during online dialogue between a plurality of users. Also, steps S513 to S516 may be configured to be periodically executed in real time during online dialogue. As a result, even in the middle of an online dialogue, a topic corresponding to the dialogue up to that point may be identified and stored in association with the dialogue information.
As a result, the user can confirm the topics mentioned by the users participating in the online dialogue in real time during the online dialogue, and can organize and manage the dialogue information based on the latest topics.

<Topic presentation processing>
The topic presentation process visually visualizes dialogue information such as audio and video of online dialogue conducted by multiple users and presents it to the user, as well as presenting the topic associated with the dialogue information to the user. be. The user can confirm the dialogue information and the topic related to the dialogue information at a glance, and can intuitively grasp the outline of the dialogue contents.

<Overview of topic presentation processing>
A voice that accepts the dialogue information to be presented from the user, acquires the dialogue information, acquires the section data and the topic for each section data, analyzes the dialogue information, and visually confirms the utterance status of each speaker. This is a series of processes of presenting a graph to the user, superimposing it on the speech graph, and presenting the topic for each utterance section to the user by superimposing it on the speech graph.

<Details of topic presentation processing>
Details of the topic presentation process will be described below.

In step S521, the first user selects dialogue information whose topic he wants to check.
Specifically, the first user operates the input device 206 or the like of the first user terminal 20 to execute the application program 2012 and execute the browser application. The first user inputs a predetermined URL (Uniform Resource Locator) designating a predetermined web server provided by the server 10 in the browser application, thereby making a request to the server 10 for a page for presenting a topic. Send.
The topic processing unit 1044 of the server 10 searches the user ID item of the dialogue table 1014 based on the first user ID 2011 included in the received request, and acquires the dialogue ID. The topic processing unit 1044 of the server 10 transmits the acquired one or more dialogue IDs to the first user terminal 20 . The control unit 204 of the first user terminal 20 presents the received one or more interaction IDs to the first user by displaying them on the display 2081 of the first user terminal 20 .
The first user selects a predetermined interaction ID from the presented interaction IDs by operating the input device 206 of the first user terminal 20 or the like. The control unit 204 of the first user terminal 20 transmits the selected predetermined interaction ID to the server 10 . The server 10 receives and accepts the dialogue ID.

Note that, when the first user is having a dialogue using the online dialogue service according to the present disclosure, the dialogue information during the dialogue may be selected. In other words, the topic presentation process may be executed on the dialog screen displayed on the display 2081 of the first user terminal 20 during the dialog.

In step S522, the topic processing unit 1044 of the server 10 searches the dialogue ID item in the dialogue table 1014 based on the received dialogue ID, and searches for the user ID, customer ID, dialogue category, transmission/reception type, voice data, and video data. Acquire dialogue information such as

In step S523, the topic processing unit 1044 of the server 10 searches the dialogue ID item in the voice segment table 1016 based on the received dialogue ID, and acquires the segment ID, start date/time, end date/time, and topic ID items. The topic processing unit 1044 of the server 10 searches the item of the section ID in the topic relevance table 1017 based on the obtained section ID, and obtains the topic ID and the relevance.
That is, the topic processing unit 1044 of the server 10 acquires a plurality of pieces of speech segment information associated with the dialogue ID, the topic ID for each piece of speech segment information, and the degree of association.

In step S<b>524 , the topic processing unit 1044 of the server 10 outputs a speech graph showing the chronological transition of the utterance state of the speaker based on the dialogue information acquired in step S<b>522 , and transmits the speech graph to the first user terminal 20 . The control unit 204 of the first user terminal 20 displays the received speech graph on the display 2081 of the first user terminal 20 to present it to the first user. An example screen 70 containing an audio graph presented to the first user is shown in FIG.
Note that the voice graph may be presented to any user such as the second user, other administrators, or other users.

The speech graph is a graph in which the horizontal axis is the dialogue time, the vertical axis (upper) is the first user's speech output amount, and the vertical axis (lower) is the second user's speech output amount. The voice of the user is shown, and the dashed line L2 represents the voice of the second user.
Looking at the solid line L1 and the dashed line L2, basically, while the first user is speaking (speaking), the second user is not speaking (listening silently). It can be seen that while the second user is speaking (speaking), the first user is not speaking (listening silently). Here, the location indicated by Z3 is a state in which both of them are uttering voices at the same time (a state of overlap), and there is a possibility that the first user started speaking before the second user finished speaking. . The locations indicated by Z1 and Z2 are times when neither of them utters a sound (silence time). The locations indicated by P1 and P2 are locations where a given keyword appears.

In step S525, the topic processing unit 1044 of the server 10 selects, from among the plurality of segmental audio data, one or more pieces of segmental audio data whose first relevance calculated in the relevance calculating step is equal to or greater than a predetermined value. Execute an interval group identification step for identifying an interval group.
Specifically, in the topic analysis processing, the topic processing unit 1044 of the server 10 determines that the first relevance calculated for each of the plurality of speech segment information extracted for one piece of dialogue information is equal to or greater than a predetermined value. When it is determined that one or more pieces of speech segment information refer to a topic related to the first topic, one or more pieces of speech segment information including the one or more pieces of speech segment information are identified as a first segment group. For example, the association with topics of a plurality of chronologically continuous speech section information is section 1: topic A, section 2: topic A, section 3: no topic, section 4: topic A, section 5: no topic, Section 6: Topic B, Section 7: Topic B, Section 8: Topic B, Section 1 to Section 4 are specified as the group of sections related to Topic A, and Section 6 to Section 8 are specified as the group of sections related to Topic B. do. Even if the section of topic A includes a speech section related to another topic, such as section 3, section 1 to section 4 as a whole can be considered to refer to the topic of topic A. In this case, Sections 1 to 4 may be collectively identified as a group of sections related to Topic A.

In the present disclosure, the first segment group is identified, but one or more segmental audio data related to a first topic related to a predetermined topic may be identified among a plurality of segmental audio data. Alternatively, the input operation of the first user or the second user may be used to select one or a plurality of segment audio data and the first segment group.

In step S525, the topic processing unit 1044 of the server 10 associates the first segment group identified in the segment group identifying step with the first topic, and executes the presentation step of presenting it to the first user or the second user. In the presenting step, in the speech graph showing the time-series transition of the utterance situation of the speaker, which is obtained by analyzing the speech data received in the receiving step, the first segment group identified in the segment group identifying step is regarded as the speech graph. Presenting on the same chronological axis and associating the first topic with the first section group and presenting it to the first user or the second user.
Specifically, in the speech graph of FIG. 20, the topic processing unit 1044 of the server 10 processes the first section group T1 associated with the first topic, the second section group T2 associated with the second topic, and the third topic The associated third segment group T3 is presented as a drawing object overlaid on the speech graph. For example, the first section group T1, the second section group T2, and the third section group T3 may be drawn as drawing objects in different colors assigned to each topic. Thereby, the first user can associate the section group with the related topic and visually superimpose it on the speech graph. This allows the first user to visually confirm at a glance which part is talking about what topic in the speech graph.
Note that the topic processing unit 1044 of the server 10 presents the first segment group specified in the segment group specifying step to any user such as an administrator other than the first user and the second user, other users, etc. Also good.

In step S525, the section group identification step includes calculating a moving average based on the first relevance calculated for each of the plurality of section audio data arranged in time series, and a step of identifying the segment audio data having a predetermined value or more as the first segment group.
Specifically, when specifying a segment group, the topic processing unit 1044 of the server 10 arranges the speech segment information obtained from the topic relevance table in chronological order based on the start date and time of the speech segment information. The topic processing unit 1044 of the server 10 calculates, as a moving average, the average of the latest N degrees of association with predetermined speech segment information. N is any integer. The calculated moving average is regarded as a new degree of relevance with respect to the predetermined speech segment information, and the speech segment information with the relevance greater than or equal to a predetermined value is specified as the first segment group associated with the first topic.
In the present disclosure, the moving average for the degree of relevance of one first topic has been mainly described for the sake of simplicity, but the number of topics is not limited to one, and similar processing may be performed for a plurality of topics.
As a result, even when the topic with a high degree of relevance changes in a short period of time for each utterance section, by smoothing the relevance of the topic, a group of sections referring to the topic can be collectively identified. In the online dialogue service, it becomes easier for the user to confirm what topic the speaker spoke about.

In step S525, in the segment group identification step, among the plurality of segment audio data arranged in chronological order, a plurality of consecutive segment audio data having a calculated first degree of association equal to or greater than a predetermined value are selected as the first segment group. You may perform the steps identified as
Specifically, when specifying a segment group, the topic processing unit 1044 of the server 10 arranges the speech segment information obtained from the topic relevance table in chronological order based on the start date and time of the speech segment information. The topic processing unit 1044 of the server 10 identifies a plurality of pieces of continuous speech segment information with degrees of association equal to or greater than a predetermined value as the first segment group associated with the first topic.
In the present disclosure, the moving average for the degree of relevance of one first topic has been mainly described for the sake of simplicity, but the number of topics is not limited to one, and similar processing may be performed for a plurality of topics.
As a result, it is possible to collectively identify the segment audio data continuously having a high degree of relevance to a specific topic as a segment group referring to the topic. In the online dialogue service, it becomes easier for the user to confirm what topic the speaker spoke about.

In step S525, the topic processing unit 1044 of the server 10 generates one or more segmental audio data based on the one or more segmental audio data among the plurality of segmental audio data and the first topic identified in the topic identification step. Perform a summarization step that generates summary text that summarizes the textual information contained in the data. The summarizing step includes extracting only text information included in one or more pieces of segmental speech data that are highly relevant to the first topic identified in the topic identification step, so that the text information is included in one or more pieces of segmental speech data Execute the step of generating a summary text that summarizes the textual information.

In step S525, the summarization step generates a summary text by applying text information included in one or more segmental speech data and a plurality of keywords associated with the first topic as input data to a learning model. Execute the step.
Specifically, section data including at least one of section audio data, section video data, and section reading text, and a plurality of keywords associated with the topic of the section data as input data, the summary model 1035 , and obtains a summary text, which is text information that summarizes the text information included in the section data, as output data. As a result, it is possible to extract only portions that are particularly highly relevant to the topic from among the text information included in the section data, and obtain a summary text that summarizes the text information included in the section data.

In step S525, the summarizing step includes one or more segment audio data included in the first segment group identified in the segment group identifying step and the first topic identified in the topic identifying step. A step of generating a summary text summarizing the text information contained in the segmental speech data is executed.
Specifically, one or more section data included in the section group and a plurality of keywords associated with the topic of the section group are applied as input data to the summary model 1035, and the text contained in the section group is Obtain summary text, which is text information that summarizes information, as output data. As a result, it is possible to extract a portion of the text information included in the section data that is particularly highly relevant to the topic, and obtain a summary text that summarizes the text information included in the section data.

In step S525, the topic processing unit 1044 of the server 10 executes a presentation step of presenting the summary text generated in the summary step in association with one or more pieces of segmental speech data.
In step S525, the topic processing unit 1044 of the server 10 executes a presentation step of presenting the summary text generated in the summary step in association with the just-one segment group identified in the segment group identification step.
Specifically, in the speech graph of FIG. 20, the topic processing unit 1044 of the server 10 presents the summary text 701 regarding the first topic of the first section group T1 in association with the first section group T1. Note that the topic processing unit 1044 of the server 10 may present the summary text 701 in association with any one or a plurality of voice segments instead of the segment group.
Note that the topic processing unit 1044 of the server 10 presents the first segment group specified in the segment group specifying step to arbitrary users such as the first user, the second user, other administrators, and other users. It may be configured.

<Learning processing>
Learning processing of the emotion evaluation model 1031, the impression evaluation model 1032, the first impression evaluation model 1033, and the second impression evaluation model 1034 will be described below.

<Learning processing of emotion evaluation model 1031>
The learning process of the emotion evaluation model 1031 is a process of learning the learning parameters of the deep neural network included in the emotion evaluation model 1031 by deep learning.

<Overview of Learning Processing of Emotion Evaluation Model 1031>
In the learning process of the emotion evaluation model 1031, the input data (input vector) is the segmental voice data, the segmental video data, and the segmental reading text, and the emotion vector or the emotion scalar, which is the emotional feature quantity, is output data (teacher data). Second, it is a process of learning the learning parameters of the deep neural network included in the emotion evaluation model 1031 by deep learning.
From the input data of the emotion evaluation model 1031, any one of the segment voice data, the segment moving image data, and the segment reading text may be omitted.

<Details of the learning process of the emotion evaluation model 1031>
The learning unit 1051 of the server 10 prepares learning data so that a predetermined emotional feature amount becomes output data (teacher data) with input data (input vectors) such as interval audio data, interval video data, interval reading text, and the like. do.
The learning unit 1051 of the server 10 creates data sets such as training data, test data, and verification data for learning the deep neural network of the emotion evaluation model 1031 based on the learning data.
The learning unit 1051 of the server 10 learns the learning parameters of the deep neural network included in the emotion evaluation model 1031 by deep learning based on the created data set.

<Learning processing of the impression evaluation model 1032>
The learning process of the impression evaluation model 1032 is a process of learning the learning parameters of the deep neural network included in the impression evaluation model 1032 by deep learning.

<Overview of Learning Processing of Impression Evaluation Model 1032>
In the learning process of the impression evaluation model 1032, the impression evaluation model 1032 receives the input data (input vector) from the segment audio data, the segment video data, and the segment reading text, and outputs the impression feature amount as the output data (teaching data). This is the process of learning the learning parameters of the included deep neural network by deep learning.
From the input data of the impression evaluation model 1032, any one of the segment voice data, the segment video data, and the segment reading text may be omitted.

<Details of Learning Processing of Impression Evaluation Model 1032>
The learning unit 1051 of the server 10 creates learning data so that a predetermined impression feature amount becomes output data (teacher data) using the segment audio data, segment video data, segment reading text, etc. as input data (input vector). do.
The learning unit 1051 of the server 10 creates data sets such as training data, test data, and verification data for learning the deep neural network of the impression evaluation model 1032 based on the learning data.
The learning unit 1051 of the server 10 learns the learning parameters of the deep neural network included in the impression evaluation model 1032 by deep learning based on the created data set.

<Learning processing of the first impression evaluation model 1033>
The learning process of the first impression evaluation model 1033 is a process of learning the learning parameters of the deep neural network included in the first impression evaluation model 1033 by deep learning.

<Overview of Learning Processing of First Impression Evaluation Model 1033>
In the learning process of the first impression evaluation model 1033, the input data (input vectors) are the segmental voice data, the segmental video data, and the segmental reading text, and the dialogue feature amount is output data (teacher data). This is a process of learning the learning parameters of the deep neural network included in the evaluation model 1033 by deep learning.
From the input data of the first impression evaluation model 1033, any one of the segment voice data, the segment video data, and the segment reading text may be omitted.

<Details of the learning process of the first impression evaluation model 1033>
The learning unit 1051 of the server 10 creates learning data such that the input data (input vector) is the segmental audio data, the segmental video data, the segmental reading text, etc., and the predetermined dialogue feature amount is output data (teaching data). do.
The learning unit 1051 of the server 10 creates data sets such as training data, test data, and verification data for learning the deep neural network of the first impression evaluation model 1033 based on the learning data.
The learning unit 1051 of the server 10 learns the learning parameters of the deep neural network included in the first impression evaluation model 1033 by deep learning based on the created data set.

<Learning processing of the second impression evaluation model 1034>
The learning process of the second impression evaluation model 1034 is a process of learning the learning parameters of the deep neural network included in the second impression evaluation model 1034 by deep learning.

<Overview of Learning Processing of Second Impression Evaluation Model 1034>
The learning process of the second impression evaluation model 1034 uses the deep neural network included in the second impression evaluation model 1034 so that the dialogue feature amount is input data (input vector) and the impression feature amount is output data (teaching data). This is a process of learning the learning parameters of by deep learning.

<Details of the learning process of the second impression evaluation model 1034>
The learning unit 1051 of the server 10 creates learning data so that the conversation feature amount and the like are input data (input vector) and a predetermined impression feature amount is output data (teacher data).
The learning unit 1051 of the server 10 creates data sets such as training data, test data, and verification data for learning the deep neural network of the second impression evaluation model 1034 based on the learning data.
The learning unit 1051 of the server 10 learns the learning parameters of the deep neural network included in the second impression evaluation model 1034 by deep learning based on the created data set.

<Details of Learning Processing of Summary Model 1035>
The learning unit 1051 of the server 10 receives the input data ( Learning data is generated so that output data (teacher data) is summarized text, which is text information that summarizes the text information included in the section data, as an input vector).
The learning unit 1051 of the server 10 creates data sets such as training data, test data, and verification data for learning the deep neural network of the summary model 1035 based on the learning data.
The learning unit 1051 of the server 10 learns the learning parameters of the deep neural network included in the summary model 1035 by deep learning based on the created data set.

<Basic computer hardware configuration>
FIG. 21 is a block diagram showing the basic hardware configuration of the computer 90. As shown in FIG. The computer 90 includes at least a processor 901, a main storage device 902, an auxiliary storage device 903, and a communication IF 991 (interface). These are electrically connected to each other by a communication bus 921 .

The processor 901 is hardware for executing an instruction set described in a program. The processor 901 is composed of an arithmetic unit, registers, peripheral circuits, and the like.

The main storage device 902 is for temporarily storing programs and data processed by the programs. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

Auxiliary storage device 903 is a storage device for storing data and programs. For example, flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, semiconductor memory, and the like.

The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard.
The network is composed of various mobile communication systems constructed by the Internet, LAN, wireless base stations, and the like. For example, networks include 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), wireless networks (for example, Wi-Fi (registered trademark)) that can be connected to the Internet through predetermined access points, and the like. When connecting wirelessly, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), and the like. In the case of wired connection, the network includes direct connection using a USB (Universal Serial Bus) cable or the like.

It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration to a plurality of computers 90 and connecting them to each other via a network. Thus, the computer 90 is a concept that includes not only the computer 90 housed in a single housing or case, but also a virtualized computer system.

<Basic Functional Configuration of Computer 90>
A functional configuration of the computer realized by the basic hardware configuration of the computer 90 (FIG. 21) will be described. The computer includes at least functional units of a control section, a storage section, and a communication section.

Note that the functional units included in the computer 90 can also be implemented by distributing all or part of each functional unit to a plurality of computers 90 interconnected via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

The control unit is implemented by the processor 901 reading out various programs stored in the auxiliary storage device 903, developing them in the main storage device 902, and executing processing according to the programs. The control unit can implement functional units that perform various information processing according to the type of program. Thereby, the computer is implemented as an information processing device that performs information processing.

A storage unit is realized by the main storage device 902 and the auxiliary storage device 903 . The storage unit stores data, various programs, and various databases. Also, the processor 901 can secure a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 according to a program. In addition, the control unit can cause the processor 901 to execute addition, update, and deletion processing of data stored in the storage unit according to various programs.

A database refers to a relational database, and is used to manage tabular tables structurally defined by rows and columns, and data sets called masters in association with each other. In a database, a table is called a table, a master is called a column, and a row is called a record. In a relational database, relationships between tables and masters can be set and associated.
Normally, each table and each master has a primary key column for uniquely identifying a record, but setting a primary key to a column is not essential. The control unit can cause the processor 901 to add, delete, and update records in specific tables and masters stored in the storage unit according to various programs.

Note that the database and master in the present disclosure may include any data structure (list, dictionary, associative array, object, etc.) in which information is structurally defined. The data structure also includes data that can be regarded as a data structure by combining data with functions, classes, methods, etc. written in any programming language.

A communication unit is implemented by the communication IF 991 . The communication unit implements a function of communicating with another computer 90 via a network. The communication section can receive information transmitted from another computer 90 and input it to the control section. The control unit can cause the processor 901 to execute information processing on the received information according to various programs. Also, the communication section can transmit information output from the control section to another computer 90 .

<Appendix>
The items described in the above embodiments will be added below.

(Appendix 1)
A program comprising a processor and a storage unit, and causing a computer to process information relating to a dialogue between a first user and a second user, wherein the program causes the processor to receive voice data relating to the dialogue (S512). a voice extracting step (S513) for extracting a plurality of segmental voice data for each utterance segment from the voice data received in the receiving step; Alternatively, based on the segment identification step (S525) of identifying a plurality of segment audio data, one or a plurality of segment audio data identified in the segment identification step among the plurality of segment audio data, and the first topic, 1 Alternatively, a program for executing a summarizing step (S525) of generating a summary text summarizing text information contained in a plurality of segmental speech data.
As a result, the user can confirm at a glance what topic the speaker has communicated about in the dialogue.

(Appendix 2)
The summarizing step (S525) extracts the text information contained in the one or more segmental speech data by extracting the portion highly relevant to the first topic from the text information contained in the one or more segmental speech data. 10. The program of Clause 1, the step of generating a condensed summary text.
As a result, the user can confirm at a glance what topic the speaker has communicated about in the dialogue.

(Appendix 3)
The summarizing step (S525) is a step of generating a summary text by applying text information contained in one or more segmental speech data and a plurality of keywords associated with the first topic as input data to a learning model. The program according to Supplementary Note 1, wherein
As a result, the user can confirm at a glance what topic the speaker has communicated about in the dialogue.

(Appendix 4)
The program causes the processor to execute a degree-of-relevance calculation step (S514) of calculating a first degree of relevance indicating the degree of relevance to a first topic for each of a plurality of pieces of section speech data, and a section identification step (S525) of , a step of identifying a first segment group including one or a plurality of segmental audio data whose first relevance calculated in the relevance calculating step is equal to or greater than a predetermined value among the plurality of segmental audio data, and a summarizing step ( S525) summarizes the text information contained in the one or more segmental audio data based on the one or more segmental audio data contained in the first segment group identified in the segment identifying step and the first topic. 10. The program of Clause 1, the step of generating summary text.
As a result, the user can confirm at a glance what topic the speaker has communicated about in the dialogue.

(Appendix 5)
1. The program according to appendix 1, wherein the program causes the processor to perform a presenting step (S525) of presenting the summarized text generated in the summarizing step in association with one or more segmental speech data.
As a result, the user can confirm at a glance what topic the speaker has communicated about in the dialogue.

(Appendix 6)
5. The program according to appendix 4, wherein the program causes the processor to execute a presenting step (S525) of presenting the summary text generated in the summarizing step in association with the just-one group of sections identified in the section identifying step (S525).
As a result, the user can confirm at a glance what topic the speaker has communicated about in the dialogue.

(Appendix 7)
4. The program according to appendix 4, wherein the program causes the processor to execute a presenting step (S525) of presenting the first segment group identified in the segment identifying step in association with the first topic (S525).
As a result, the user can confirm at a glance what topic the speaker has communicated about in the dialogue.

(Appendix 8)
The presenting step (S525) presents the first section group identified in the section identifying step in the speech graph showing the chronological transition of the utterance situation of the speaker, which is obtained by analyzing the speech data received in the receiving step. 8. The program according to appendix 7, wherein the step of presenting on the same chronological axis as the graph and presenting the first topic in association with the first section group.
As a result, the user can confirm at a glance what topic the speaker has spoken about by superimposing the situation of the speaker's utterance in the dialogue on the speech graph showing the chronological order.

(Appendix 9)
The program stores, in the processor, a keyword receiving step (S502) of receiving one or more keywords from a first user, and the one or more keywords received in the keyword receiving step in association with a first topic related to a predetermined topic. The program according to appendix 1, causing the topic storage step (S503) to be executed.
With this, it is possible to confirm at a glance what topic the speaker has communicated about in the dialogue based on the topic that the user has stored in advance in association with the keyword.

(Appendix 10)
The program instructs the processor to store a voice data received in the receiving step (S512), and one or more new keywords to be newly associated with the first topic based on the voice data stored in the voice storing step. and a keyword presentation step (S501) of presenting to the first user, and a keyword reception step (S502) includes, among the plurality of new keywords presented to the first user in the keyword presentation step, 10. The program of Clause 9, the step of accepting one or more keywords selected by the first user.
Thereby, the user can receive presentation of one or more new keywords that are preferably newly associated with the topic, based on the keywords used in the past dialogue information. Users can easily define and memorize topics.

(Appendix 11)
The voice extracting step (S513) is a step of extracting a plurality of segment voice data for each utterance segment from the voice data received in the receiving step before the dialogue ends. 4. The program according to Supplementary Note 4, which is a step of calculating a first degree of relevance indicating a degree of relevance to the first topic for each segmental audio data included in the plurality of segmental audio data before the end of.
As a result, it is possible to calculate the degree of relevance between the segmental audio data and the topic in real time during the dialogue. For example, during business negotiations, it is possible to confirm what topic the speaker is communicating about.

(Appendix 12)
The degree-of-relevance calculation step (S514) is a step of calculating the degrees of relevance for each of a plurality of topics associated with a plurality of keywords for each of the plurality of segmental audio data. A memo specifying step (S516) of specifying a response memo to the dialogue based on the calculated degrees of relevance for each of the plurality of topics, and a storing step (S516) of storing the response memo identified in the memo identifying step in association with the dialogue. and the program according to appendix 4.
Accordingly, dialogue information can be managed by specifying a topic that characterizes the entire dialogue and adding a response memo on the topic to the dialogue.

(Appendix 13)
In the degree-of-relevance calculation step (S514), among the plurality of keywords associated with the first topic, the more keywords are included in the plurality of segmental speech data extracted in the speech extraction step, the smaller the weight given to the degree of relevance. 4. The program according to Supplementary Note 4, wherein the degree of matching in consideration of the weighting of the plurality of keywords associated with the first topic for each of the plurality of segmental speech data is calculated as the first degree of relevance indicating the degree of relevance with the first topic.
As a result, among the keywords associated with the topic, the weight of common keywords contained in many segmental speech data can be reduced. By increasing the importance of keywords appearing in specific segmental audio data, it is possible to more accurately calculate the degree of relevance between the segmental audio data and the topic.

(Appendix 14)
The degree-of-relevance calculation step (S514) includes, among the plurality of keywords associated with the first topic, a plurality of pieces of segmental speech data up to a predetermined number in chronological order from the target segmental speech data for which the first degree of relevance is to be calculated. The weight given to the degree of relevance becomes smaller as the number of keywords included in the first topic increases. 14. The program according to appendix 13, wherein the program is calculated as the first degree of relevance indicating the degree of relevance.
As a result, among the keywords associated with the topic, considering only a plurality of past segmental audio data near the target segmental audio data, the degree of relevance between the segmental audio data and the topic can be determined more accurately with a smaller amount of calculation. can be calculated to Also, the degree of relevance to a topic can be calculated in real time.

(Appendix 15)
The section specifying step (S525) includes a step of calculating a moving average based on the first relevance calculated for each of the plurality of section audio data arranged in chronological order; A program according to appendix 4, comprising the step of identifying the above segmental audio data as the first segment group.
As a result, even when topics with high relevance for each utterance segment change in a short period of time, by smoothing the relevance of the topic, it is possible to collectively identify a segment group referring to the topic. It becomes easier for the user to confirm what topic the speaker spoke about in the dialogue.

(Appendix 16)
The section identifying step (S525) identifies, as a first section group, a plurality of continuous section sound data having a calculated first degree of association equal to or higher than a predetermined value among the plurality of section sound data arranged in chronological order. The program according to appendix 4, which is a step of
As a result, it is possible to collectively identify the segment audio data continuously having a high degree of relevance to a specific topic as a segment group referring to the topic. It becomes easier for the user to confirm what topic the speaker spoke about in the dialogue.

(Appendix 17)
An information processing apparatus comprising a processor and a storage unit, wherein the processor executes the program according to any one of Appendixes 1 to 16.
As a result, the user can confirm at a glance what topic the speaker has communicated about in the dialogue.

(Appendix 18)
17. An information processing system including an information processing device comprising a processor and a storage unit, wherein the processor executes the program according to any one of appendices 1 to 16.
As a result, the user can confirm at a glance what topic the speaker has communicated about in the dialogue.

(Appendix 19)
17. An information processing method executed by a computer comprising a processor and a storage unit, the information processing method comprising causing the processor to execute the program according to any one of Appendices 1 to 16.
As a result, the user can confirm at a glance what topic the speaker has communicated about in the dialogue.

(Appendix 20)
An information processing terminal comprising a processor and a display device, wherein the processor displays information presented by a presentation step executed in the information processing device capable of executing the program according to any one of appendices 5 to 8 on the display device. Information processing terminal capable of display.
As a result, the user can confirm at a glance what topic the speaker has communicated about in the dialogue.

1 system, 10 server, 101 storage unit, 104 control unit, 106 input device, 108 output device, 20 first user terminal, 201 storage unit, 204 control unit, 206 input device, 208 output device, 30 second user terminal, 301 storage unit, 304 control unit, 306 input device, 308 output device, 50 CRM system, 501 storage unit, 504 control unit, 506 input device, 508 output device, 60 voice server (PBX), 601 storage unit, 604 control unit , 606 input device, 608 output device

Claims (18)

A program, comprising a processor and a storage unit, for causing a computer to process information relating to interaction between a first user and a second user,
The program causes the processor to:
a receiving step of receiving audio data relating to the dialogue;
a voice extracting step of extracting a plurality of segmental voice data for each utterance segment from the voice data received in the receiving step;
a section identification step of identifying one or a plurality of section sound data related to a first topic related to a predetermined topic among the plurality of section sound data;
Text information contained in the one or more section audio data based on the one or more section audio data identified in the section identifying step and the first topic among the plurality of section audio data, a summarizing step to generate a summarized summary text;
a keyword receiving step of receiving one or more keywords from the first user;
a topic storing step of storing the one or more keywords received in the keyword receiving step in association with the first topic related to a predetermined topic;
program to run. The summarizing step extracts the text information contained in the one or more segmental speech data by extracting portions highly relevant to the first topic from the text information contained in the one or more segmental speech data. is the step of generating a summary text that summarizes the
A program according to claim 1. The summarizing step generates the summary text by applying text information included in the one or more segmental speech data and the plurality of keywords associated with the first topic as input data to a learning model. is the step to
A program according to claim 1. The program causes the processor to:
a degree-of-relevance calculation step of calculating a first degree of relevance indicating a degree of relevance to the first topic for each of the plurality of segmental audio data;
and
The section identification step identifies a first section group including one or a plurality of section sound data whose first relevance calculated in the relevance calculation step is equal to or greater than a predetermined value, among the plurality of section sound data. is a step to
The summarizing step is included in the one or more segmental audio data based on the one or more segmental audio data included in the first segment group identified in the segment identifying step and the first topic. generating a summary text that summarizes the textual information;
A program according to claim 1. The program causes the processor to:
a presenting step of presenting the summarized text generated in the summarizing step in association with the one or more segmental speech data;
2. The program according to claim 1, causing the execution of The program causes the processor to:
a presenting step of presenting the summary text generated in the summarizing step in association with the first section group identified in the section identifying step;
5. The program according to claim 4, causing the execution of The program causes the processor to:
a presentation step of presenting the first segment group identified in the segment identification step in association with the first topic;
5. The program according to claim 4, causing the execution of In the presenting step, the first section group identified in the section identifying step in the speech graph showing the time-series transition of the utterance situation of the speaker, which is obtained by analyzing the speech data received in the receiving step. is presented on the same chronological axis as the speech graph, and the first topic is presented in association with the first segment group,
8. A program according to claim 7. The program causes the processor to:
a voice storage step of storing the voice data received in the receiving step;
a keyword presenting step of presenting one or more new keywords to be newly associated with the first topic to the first user based on the voice data stored in the voice storing step;
and
The keyword accepting step is a step of accepting one or more keywords selected by the first user from among the plurality of new keywords presented to the first user in the keyword presenting step.
A program according to claim 1 . The voice extracting step is a step of extracting a plurality of segment voice data for each utterance segment from the voice data received in the receiving step before the dialogue ends,
The degree-of-relevance calculating step is a step of calculating, before the dialogue ends, the first degree of relevance indicating the degree of relevance to the first topic for each segmental audio data included in the plurality of segmental audio data. be,
5. The program according to claim 4. The relevance calculating step is a step of calculating, for each of the plurality of segmental audio data, the relevance of each of a plurality of topics associated with a plurality of keywords,
The program causes the processor to:
a memo identifying step of identifying a response memo to the dialogue based on the degrees of relevance for each of the plurality of topics calculated in the relevance calculating step;
a storing step of storing the response memo identified in the memo identifying step in association with the dialogue;
5. The program according to claim 4, causing the execution of The relevance calculation step includes:
among the plurality of keywords associated with the first topic, the more keywords are included in the plurality of segmental speech data extracted in the speech extraction step, the smaller the weight given to the degree of association;
Calculating a matching degree considering the weighting of the plurality of keywords associated with the first topic for each of the plurality of segmental audio data as the first degree of relevance indicating the degree of relevance with the first topic;
5. The program according to claim 4. The relevance calculation step includes:
Of the plurality of keywords associated with the first topic, the more keywords are included in the plurality of segmental audio data up to a predetermined number in chronological order from the target segmental audio data for which the first relevance is to be calculated, the more relevant the keyword is. so that the weight given to degrees is small,
Calculating the degree of matching with the plurality of keywords associated with the first topic for each of the plurality of segmental audio data in consideration of the weighting as the first degree of association indicating the degree of association with the first topic ,
13. A program according to claim 12 . The section identification step includes:
calculating a moving average based on the first relevance calculated for each of the plurality of segment audio data arranged in time series;
identifying, as the first segment group, the segment audio data for which the calculated moving average is equal to or greater than a predetermined value;
including,
5. The program according to claim 4. The section identification step includes:
A step of specifying, as the first segment group, a plurality of continuous segmental audio data having a calculated first degree of association equal to or greater than a predetermined value among the plurality of segmental audio data arranged in time series. ,
5. The program according to claim 4. An information processing device comprising a processor and a storage unit,
The processor executes the program according to any one of claims 1 to 15 ,
Information processing equipment. An information processing system including an information processing device comprising a processor and a storage unit,
The processor executes the program according to any one of claims 1 to 15 ,
Information processing system. An information processing method executed by a computer comprising a processor and a storage unit,
causing the processor to execute the program according to any one of claims 1 to 15 ;
Information processing methods.

Images