JP2023076430A - Program, information processing system, and information processing method - Google Patents

Abstract

To provide a program configured so that a user can accurately identify a response on a call between the user and a customer in a short time by confirming summary information, an information processing system, and an information processing method.SOLUTION: Summary processing includes: a binding step S105 for binding text information before/after a change of speaker information as a text information group when multiple pieces of text information are arranged in order of speaking time; an importance calculating step for calculating importance levels for the multiple pieces of text information and one or more text information groups; and a summary extraction step S107 for extracting the text information or the text information group on the basis of the importance levels.SELECTED DRAWING: Figure 14

Description

There is an application for the application of Article 30, Paragraph 2 of the Patent Act. Service release date November 26, 2020 June 1, 2021 2. Website address https://miitel. revcomm. co. jp/ 3. Publisher: RevComm Inc. (b) Announcement on the website 1. Publication date on website a. November 26, 2020 b. June 1, 2021 c. October 4, 2021 2. Website address a. https://support. mitel. jp b. https://support. mitel. jp c. https://www. revcomm. co. jp 3. Publisher: RevComm Inc.

The present disclosure relates to a program, an information processing system, and an information processing method.

Conventionally, in a computer system that provides call services, there is known a technique for creating a summary document that characterizes the contents of a call.
Patent Literature 1 discloses a technique for extracting important sentences from the content of dialogue and generating a summary. Japanese Patent Laid-Open No. 2002-200000 discloses a technique for summarizing a dialogue text by using a dialogue structure to correct the dialogue text so that it is easier to read. Patent Literature 3 discloses a technique for improving the efficiency of a supervisor's monitoring work and improving the quality of an operator's response to a customer in a call center.

JP 2013-120514 JP 2017-111190 JP 2020-150408

However, in a call between a user and a customer, summary information may be generated in which either the speech or the response is omitted. It was sometimes difficult to confirm whether

Therefore, the present disclosure has been made to solve the above problems, and its object is to provide a technique for generating summary information consisting of utterances and responses between a user and a customer.

A program for causing a computer to manage data related to a call conducted between a user and a customer, the program comprising a processor and a storage unit, wherein the program causes the processor to receive voice data related to the call; a speech extraction step of extracting a plurality of segmental speech data for each utterance segment from speech data; a text extraction step of performing text recognition on each of the plurality of segmental speech data to extract a plurality of pieces of text information; A program for executing a summary extraction step of extracting, as summary information, one or a plurality of pieces of text information having different speaker information and preceding and following utterance times from the plurality of pieces of text information extracted in the step.

It is possible to generate summary information that focuses on call handling between the user and the customer.
By confirming the summary information, the user can accurately grasp the call handling between the user and the customer in a short period of time.

1 is a diagram showing an overall configuration of an information processing system 1; FIG. 3 is a block diagram showing the functional configuration of the server 10; FIG. 2 is a block diagram showing a functional configuration of a user terminal 20; FIG. 3 is a block diagram showing the functional configuration of a CRM system 30; FIG. 3 is a block diagram showing the functional configuration of a customer terminal 50; FIG. FIG. 10 is a diagram showing the data structure of a user table 1012; FIG. FIG. 10 is a diagram showing the data structure of an organization table 1013; FIG. It is a figure which shows the data structure of the call table 1014. FIG. FIG. 10 is a diagram showing the data structure of a speech recognition table 1015; FIG. FIG. 10 shows the data structure of a summary table 1016; FIG. 10 is a diagram showing the data structure of a response memo table 1017. FIG. FIG. 11 shows the data structure of a customer table 3012; FIG. FIG. 10 is a diagram showing the data structure of a response history table 3013; It is a flowchart which shows the operation|movement of a summary process (1st Example). It is a flowchart which shows the operation|movement of a summary process (2nd Example). It is a flowchart which shows the operation|movement of a response memo addition process. It is a flowchart which shows the operation|movement of a response memo proposal process. 4 is a flowchart showing the operation of score calculation processing; It is a flowchart which shows the operation|movement of CRM storage processing. It is a flowchart which shows the operation|movement of a call display process. FIG. 10 is a diagram showing an overview of binding processing in summary processing (first embodiment); It is the figure which showed the outline|summary of the additional extraction process in a summary process (2nd Example). FIG. 10 is a diagram showing an overview of summary display processing; It is the figure which showed the screen example of the CRM service in call display processing. 2 is a block diagram showing the basic hardware configuration of computer 90. FIG.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all the drawings for explaining the embodiments, common constituent elements are denoted by the same reference numerals, and repeated explanations are omitted. It should be noted that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Also, not all the components shown in the embodiments are essential components of the present disclosure. Each figure is a schematic diagram and is not necessarily strictly illustrated.

<Overview of information processing system 1>
FIG. 1 is a diagram showing the overall configuration of an information processing system 1. As shown in FIG. An information processing system 1 according to the present disclosure is an information processing system that provides a call service according to the present disclosure. The information processing system 1 is an information processing system for providing services related to calls between users and customers, and for storing and managing data related to calls.

<Basic Configuration of Information Processing System 1>
FIG. 1 shows an information processing system 1 according to the present disclosure. The information processing system 1 is connected via a network N to a server 10, a plurality of user terminals 20A, 20B, 20C, a CRM system 30, a voice server (PBX) 40, and a voice server (PBX) 40. It comprises customer terminals 50A, 50B, and 50C connected via a telephone network T. FIG.
FIG. 2 is a block diagram showing the functional configuration of the server 10. As shown in FIG. FIG. 3 is a block diagram showing the functional configuration of the user terminal 20. As shown in FIG. FIG. 4 is a block diagram showing the functional configuration of the CRM system 30. As shown in FIG. FIG. 5 is a block diagram showing the functional configuration of the customer terminal 50. As shown in FIG.

The server 10 is an information processing device that provides a service of storing and managing data (call data) related to calls made between users and customers.

The user terminal 20 is an information processing device operated by a user who uses the service. The user terminal 20 may be, for example, a stationary PC (Personal Computer), a laptop PC, or a mobile terminal such as a smart phone or a tablet. Moreover, it may be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch type terminal.

The CRM system 30 is an information processing device managed and operated by a company (CRM company) that provides CRM (Customer Relationship Management) services. CRM services include SalesForce, HubSpot, Zoho CRM, Kintone, and the like.

The voice server (PBX) 40 is an information processing device that functions as a switching system that enables communication between the user terminal 20 and the customer terminal 50 by connecting the network N and the telephone network T to each other.

The customer terminal 50 is an information processing device operated by the customer when talking to the user. The customer terminal 50 may be, for example, a mobile terminal such as a smart phone or a tablet, a stationary PC (Personal Computer), or a laptop PC. Moreover, it may be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch type terminal.

Each information processing device is composed of a computer having an arithmetic device and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. Descriptions of the server 10, the user terminal 20, the CRM system 30, the voice server (PBX) 40, and the customer terminal 50 that overlap with the basic hardware configuration of the computer and the basic functional configuration of the computer, which will be described later, will be omitted.

The configuration and operation of each device will be described below.

<Functional Configuration of Server 10>
A functional configuration realized by the hardware configuration of the server 10 is shown in FIG. The server 10 has a storage unit 101 and a control unit 104 .

<Configuration of Storage Unit of Server 10>
Storage unit 101 of server 10 includes application program 1011 , user table 1012 , organization table 1013 , call table 1014 , speech recognition table 1015 , summary table 1016 and response memo table 1017 .
FIG. 6 is a diagram showing the data structure of the user table 1012. As shown in FIG. FIG. 7 is a diagram showing the data structure of the organization table 1013. As shown in FIG. FIG. 8 is a diagram showing the data structure of the call table 1014. As shown in FIG. FIG. 9 is a diagram showing the data structure of the speech recognition table 1015. As shown in FIG. FIG. 10 is a diagram showing the data structure of the summary table 1016. As shown in FIG. FIG. 11 shows the data structure of the response memo table 1017. As shown in FIG.

The user table 1012 is a table that stores and manages information on member users (hereinafter referred to as users) who use the service. By registering to use the service, the user's information is stored in a new record in the user table 1012 . This enables the user to use the service according to the present disclosure. The user table 1012 is a table having user ID as a primary key and columns of user ID, CRM ID, organization ID, user name, cooperation mode, user attribute, and evaluation index.

User ID is an item that stores user identification information for identifying a user.
CRMID is an item that stores identification information for identifying a user in the CRM system 30 . The user can receive CRM services by logging into the CRM system 30 with the CRM ID. That is, the user ID in the server 10 and the CRMID in the CRM system 30 are linked.
The organization ID is an item that stores the organization ID of the organization to which the user belongs.
The user name is an item that stores the name of the user.
The cooperation mode is an item for storing setting items (cooperation settings) when storing data related to a call between a user and a customer in an external CRM system.
In the present disclosure, the cooperation mode is stored for each user, but may be stored for each organization or department in the organization table. In this case, the cooperation mode applied to each user is applied to each user by referring to the cooperation mode for each organization and department stored in the organization table. At this time, a uniform cooperation mode can be applied to users belonging to an organization or department.
The user attribute is an item that stores information related to user attributes such as age, gender, hometown, dialect, occupation (sales, customer support, etc.) of the user.
The evaluation index is an item that stores a quantitative evaluation index for the call handling skill of the user. Specifically, the evaluation index is each index of analysis data (Talk:Listen ratio, number of times of silence, number of times of suffering, number of rallies, fundamental frequency, intonation strength, It is a numerical value calculated by applying a predetermined algorithm to speech speed, number of fillers, talk script matching degree, etc.).
For example, in a field such as inside sales, the evaluation index quantitatively represents the customer service skill of each user, and a user with a higher evaluation index is expected to have a higher sales performance.

The organization table 1013 is a table that defines information about the organization to which the user belongs. Organizations include arbitrary organizations and groups such as companies, corporations, corporate groups, circles, and various organizations. Organizations may also be defined by more detailed sub-groups such as company departments (Sales Department, General Affairs Department, Customer Support Department). The organization table 1013 is a table having columns of organization ID, organization name, and organization attribute with organization ID as a primary key.

The organization ID is an item that stores organization identification information for identifying an organization.
The organization name is an item that stores the name of the organization. The name of the organization includes arbitrary organization names and group names such as company names, corporate names, corporate group names, circle names, and various organization names.
The organization attribute is an item that stores information on the organization attributes such as the organization type (company, corporate group, other organization, etc.) and industry (real estate, finance, etc.).

Call table 1014 is a table that stores and manages call data related to calls made between users and customers. The call table 1014 is a table having call ID as a primary key and columns of call ID, user ID, customer ID, call category, incoming/outgoing call type, voice data, presence/absence of voice recognition, presence/absence of summary, and analysis data.

The call ID is an item that stores call data identification information for identifying call data.
The user ID is an item that stores the user's user ID (user identification information) in a call between the user and the customer.
The customer ID is an item that stores the customer's customer ID (customer identification information) in a call between the user and the customer.
The call category is an item that stores the type (category) of calls made between the user and the customer. Call data is classified by call category. In the call category, values such as telephone operator, telemarketing, customer support, and technical support are stored according to the purpose of the call between the user and the customer.
The incoming/outgoing call type is an item that stores information for distinguishing whether a call made between a user and a customer is originated by the user (outbound) or received by the user (inbound). .
Voice data is an item that stores voice data of a call made between a user and a customer. As the audio data format, various audio data formats such as mp4 and wav can be used. Also, it is possible to store reference information (path) to an audio data file located at another location.
The voice data may be data in a format in which the user's voice and the customer's voice are individually identifiable as identifiers. In this case, the control unit 104 of the server 10 can perform independent analysis processing on the user's voice and the customer's voice.
In the present disclosure, video data including audio information may be used instead of audio data. Also, audio data in the present disclosure is a concept including audio data included in moving image data.
The presence/absence of voice recognition is an item that stores information for determining whether voice recognition processing has been performed on voice data of a call between a user and a customer. When voice recognition processing is performed on voice data, information indicating that voice recognition processing is being performed is stored. If speech recognition processing has not been performed on the speech data, information such as blank, null, and other information indicating that speech recognition processing has not been performed is stored.
Summarization presence/absence is an item for storing information for determining whether or not summarization processing has been performed on voice data of a call made between a user and a customer. When summarization processing is being performed, information indicating that summarization processing is being performed is stored. If the summarization process has not been performed, blank, null, or other information indicating that the summarization process has not been performed is stored.
The analysis data is an item that stores analysis data obtained by analyzing voice data of voice calls made between the user and the customer. The analysis data includes the Talk:Listen ratio, the number of times of silence, the number of times of covering, the number of times of rallying, the fundamental frequency, the intensity of intonation, the speed of speech, the number of fillers, the talk script matching degree, and the like.

The speech recognition table 1015 is a table that stores speech recognition information including speech time, speaker, and text obtained by performing speech recognition processing on speech data of a call between a user and a customer. The voice recognition table 1015 is a table having columns of call ID, text, utterance time, and speaker.

The call ID is an item that stores the call ID (call data identification information) of the call data from which the voice recognition information is generated.
The text is an item that stores text information of the text recognition result for each section (speech section) in which the voice of the voice data of the call conducted between the user and the customer exists. Specifically, the contents of sentences uttered by the speaker for each utterance section of the voice data are stored as text data.
The utterance time is an item that stores the start time in the voice data of the utterance period (interval voice data). Note that the utterance time may be an arbitrary time related to each utterance period, such as the start time of each utterance period, the end time of each utterance period, or the time between any utterance periods.
As for the speaker, information for identifying the speaker of the interval voice data is stored. Specifically, it is information for identifying a user or a customer. Note that user identification information for identifying a speaker such as a user ID and customer ID, customer identification information, and the like may be stored.

The summary table 1016 is a table that stores summary information consisting of utterance time, speaker, and text obtained by performing summary processing on voice recognition information of a call made between a user and a customer. Summary information is textual information relating to and characterizing a call between a user and a customer. By checking the summary information, the user can grasp the content of the call made between the user and the customer in a short time. Summary table 1016 is a table having columns for call ID, text, time of speech, and speaker.

The call ID is an item that stores the call ID (call data identification information) of the call data that is the source of the summary information.
Text is an item that stores the text of speech recognition information extracted as summary information.
The utterance time is an item that stores the utterance time of the speech recognition information extracted as summary information.
Speaker is an item that stores the speaker of the speech recognition information extracted as summary information.

The response memo table 1017 is a table that stores and manages information related to response memos that are associated with call data related to calls made between users and customers. A user can organize and manage a large number of call data by setting (attaching) a response memo to the call data. Further, the server 10 can change the processing conditions by using the response memo added to the call data when performing various control processing. The response memo table 1017 is a table having memo ID as a primary key and columns of memo ID, call ID, grantor ID, memo content, and memo date and time.

Memo ID is an item for storing response memo identification information for identifying a response memo.
The call ID is an item that stores the call ID (call data identification information) of the call data to which the response memo is attached.
The giver ID is an item for storing the user ID of the user who gave the response memo to the call data.
The content of the memo is an item that stores the content of the memo attached to the call data. The contents of the memo are usually character string (text) information.
The memo date and time is an item for storing the date and time when the user added a response memo to the call data.

<Configuration of Control Unit of Server 10>
The control unit 104 of the server 10 includes a user registration control unit 1041, a setting unit 1042, a recognition unit 1043, an analysis unit 1044, an importance calculation unit 1045, a summary unit 1046, a learning unit 1047, a response memo proposal unit 1048, and a response memo addition unit. 1049 , a score calculation unit 1050 , a CRM storage control unit 1051 and a display control unit 1052 . Control unit 104 implements each functional unit by executing application program 1011 stored in storage unit 101 .

The user registration control unit 1041 performs processing for storing information of users who wish to use the service according to the present disclosure in the user table 1012 .
Information stored in the user table 1012, such as user IDs, user names, and user attributes, can be obtained by opening a web page or the like operated by a service provider from any information processing terminal, filling in a predetermined input form with the user ID, user First name and user attributes are entered and transmitted to the server 10 . The user registration control unit 1041 of the server 10 stores the received user ID, user name, and user attributes in a new record of the user table 1012, and user registration is completed. As a result, the users stored in the user table 1012 can use the service.
Prior to registration of user information in the user table 1012 by the user registration control unit 1041, the service provider may perform a predetermined examination to limit whether or not the user can use the service.
The user ID may be any character string or number that can identify the user, any character string or number desired by the user, or any character string or number automatically set by the user registration control unit 1041 of the server 10. can be
The user registration control unit 1041 may store information such as the organization ID, organization name, and organization attributes of the organization to which the user belongs in the organization table 1013 in association with the user at the time of user registration. The information on the organization to which the user belongs may be input by the user himself or may be registered by the administrator of the organization to which the user belongs, the operator of the service according to the present disclosure, or the like.

The setting unit 1042 executes cooperation mode setting processing. Details will be described later.
The recognition unit 1043 executes voice recognition processing. Details will be described later.
The analysis unit 1044 executes voice analysis processing. Details will be described later.
The importance calculation unit 1045 executes importance calculation processing. Details will be described later.
Summarizing unit 1046 performs a summarizing process. Details will be described later.
The learning unit 1047 executes learning processing. Details will be described later.
The response memo proposal unit 1048 executes a response memo proposal process. Details will be described later.
The response memo adding unit 1049 executes a response memo adding process. Details will be described later.
The score calculation unit 1050 executes score calculation processing. Details will be described later.
The CRM storage control unit 1051 executes CRM storage processing. Details will be described later.
The display control unit 1052 executes call display processing. Details will be described later.

<Functional Configuration of User Terminal 20>
A functional configuration realized by the hardware configuration of the user terminal 20 is shown in FIG. The user terminal 20 includes a storage unit 201 , a control unit 204 , an input device 206 connected to the user terminal 20 and an output device 208 . Input device 206 includes camera 2061 , microphone 2062 , position information sensor 2063 , motion sensor 2064 , keyboard 2065 and mouse 2066 . The output device 208 includes a display 2081 and speakers 2082 .

<Configuration of Storage Unit of User Terminal 20>
The storage unit 201 of the user terminal 20 stores a user ID 2011 for identifying a user who uses the user terminal 20, an application program 2012, and a CRM ID 2013. FIG.
The user ID is the user's account ID for the server 10 . The user transmits the user ID 2011 from the user terminal 20 to the server 10 . The server 10 identifies the user based on the user ID 2011 and provides the user with the service according to the present disclosure. The user ID includes information such as a session ID temporarily assigned by the server 10 to identify the user using the user terminal 20 .
CRMID is the user's account ID for the CRM system 30 . The user transmits the CRMID 2013 from the user terminal 20 to the CRM system 30 . The CRM system 30 identifies the user based on the CRMID 2013 and provides the CRM service to the user. The CRMID 2013 includes information such as a session ID temporarily assigned by the CRM system 30 to identify the user using the user terminal 20 .
The application program 2012 may be stored in the storage unit 201 in advance, or may be downloaded from a web server or the like operated by the service provider via the communication IF. Application program 2012 includes an interpreted programming language such as JavaScript (registered trademark) that runs on a web browser application stored in user terminal 20 .

<Configuration of Control Unit of User Terminal 20>
The control unit 204 of the user terminal 20 has an input control unit 2041 and an output control unit 2042 . By executing the application program 2012 stored in the storage unit 201 , the control unit 204 implements functional units of an input control unit 2041 and an output control unit 2042 .
The input control unit 2041 of the user terminal 20 acquires information output from input devices such as a camera 2061, a microphone 2062, a position information sensor 2063, a motion sensor 2064, a keyboard 2065, and a mouse 2066 connected to the user terminal 20, and various Execute the process. The input control unit 2041 of the user terminal 20 executes a process of transmitting information acquired from the input device 206 to the server 10 together with the user ID 2011 . Similarly, the input control unit 2041 of the user terminal 20 performs a process of transmitting information acquired from the input device 206 to the CRM system 30 together with the CRMID 2013 .
The output control unit 2042 of the user terminal 20 receives the user's operation on the input device 206 and information from the server 10 and the CRM system 30, and controls the display content of the display 2081 connected to the user terminal 20 and the audio output content of the speaker 2082. Execute control processing.

<Functional Configuration of CRM System 30>
A functional configuration realized by the hardware configuration of the CRM system 30 is shown in FIG. The CRM system 30 has a storage unit 301 and a control unit 304 .
The user has also concluded a separate contract with a CRM business, and by using the CRM ID 2013 set for each user to access (login to) the website operated by the CRM business through a web browser, etc., the CRM service can be accessed. can be provided.

<Configuration of Storage Unit of CRM System 30>
The storage unit 301 of the CRM system 30 has a customer table 3012 and a reception history table 3013 .
FIG. 12 is a diagram showing the data structure of the customer table 3012. As shown in FIG. FIG. 13 shows the data structure of the response history table 3013. As shown in FIG.

The customer table 3012 is a table for storing and managing customer information. The customer table 3012 is a table having customer ID as a primary key and columns of customer ID, user ID, name, telephone number, customer attribute, customer organization name, and customer organization attribute.

The customer ID is an item that stores customer identification information for identifying the customer.
The user ID is an item that stores the user ID (user identification information) of the user associated with the customer. The user can display a list of customers associated with his/her own user ID, and can make calls (calls) to the customers.
In the present disclosure, the customer is associated with the user, but may be associated with the organization (organization ID of the organization table 1013). In that case, a user belonging to an organization can display a list of customers associated with his/her own organization ID, or can send a message to the customer.
The name is an item for storing the customer's name.
The phone number is an item that stores the customer's phone number.
A user accesses a website provided by the CRM system, selects a customer to call, and performs a predetermined operation such as "Call" to make a call to the customer's telephone number from the user terminal 20. be able to.
The customer attribute is an item that stores information related to customer attributes such as customer age, gender, hometown, dialect, occupation (sales, customer support, etc.).
The customer organization name is an item that stores the name of the organization to which the customer belongs. The name of the organization includes arbitrary organization names and group names such as company names, corporate names, corporate group names, circle names, and various organization names.
The customer organization attribute is an item that stores information related to organization attributes such as the customer's organization type (company, corporate group, other organization, etc.) and type of industry (real estate, finance, etc.).
The customer attribute, the customer organization name, and the customer organization attribute may be stored by inputting them by the user, or may be input by the customer when the customer accesses a predetermined website.

The response history table 3013 is a table for storing and managing records (response history information) of customer responses (response history). When the customer correspondence is sales activities, records of past sales activities (date and time, contents of sales activities, etc.) are stored. The response history table 3013 has a response history ID as a primary key, and has columns of response history ID, call ID, URL, customer ID, user ID (caller), dialing date and time, call start date and time, call end date and time, and comment. is.

In the present disclosure, each time a call is made between a user and a customer, a new record is created and reception history information is stored.
The response history ID is an item for storing response history identification information for identifying the response history.
The call ID is an item that stores the call ID (call data identification information) of call data related to the response history. Thereby, the reception history in the CRM system is associated with the call data of the server 10 .
The URL is URL (Uniform Resource Locator) information including a character string that uniquely identifies the call ID. The URL may contain the call ID directly, or may contain a character string that can decode the call ID, or a specific character string that can acquire the call ID by referring to a table (not shown). . By operating the browser of the user terminal 20 and opening the URL, the user can access the server 10 and check the analysis result screen that visualizes the analysis data of the call data related to the response history.
The customer ID is an item that stores the customer's customer ID (customer identification information) related to the response history.
The user ID is an item that stores the user ID (user identification information) of the user regarding the response history.
The date and time dialed is an item for storing the date and time when the user made a call to the customer in relation to the reception history.
The call start date and time is an item that stores the start date and time of the call made between the user and the customer in relation to the reception history.
The call end date and time is an item that stores the end date and time of the call between the user and the customer in relation to the response history.
A comment is an item that stores text information such as memos and comments regarding the response history. The user can edit the matters noticed about the response history, the matters to be transferred, etc. as comments. Also, the control unit 104 of the server 10 can freely edit the comment by sending a predetermined request to the API (Application Programming Interface) of the CRM system 30 .

<Configuration of Control Unit of CRM System 30>
The control section 304 of the CRM system 30 has a user registration control section 3041 . Control unit 304 implements each functional unit by executing application program 3011 stored in storage unit 301 .

The CRM system 30 provides functions called API (Application Programming Interface), SDK (Software Development Kit), and code snippets (hereinafter referred to as "beacons"), and the user can use the server 10 and By setting account information and the like for the CRM system 30, the control unit 104 of the server 10 and the control unit 304 of the CRM system 30 can communicate with each other to realize arbitrary information processing.

<Outline of voice server (PBX) 40>
The voice server (PBX) 40 makes a call to the customer terminal 50 when the user makes a call to the customer.
The voice server (PBX) 40 sends a message (hereinafter referred to as an "incoming call notification message") to the user terminal 20 to indicate that the customer has made a call to the user. Also, the voice server (PBX) 40 can send an incoming call notification message to the beacon, SDK, API, etc. provided by the server 10 .

<Functional Configuration of Customer Terminal 50>
FIG. 5 shows a functional configuration realized by the hardware configuration of the customer terminal 50. As shown in FIG. The customer terminal 50 includes a storage unit 501 , a control unit 504 , a touch panel 506 , a touch sensitive device 5061 , a display 5062 , a microphone 5081 , a speaker 5082 , a position information sensor 5083 , a camera 5084 and a motion sensor 5085 .

<Configuration of Storage Unit of Customer Terminal 50>
The storage unit 501 of the customer terminal 50 stores telephone numbers 5011 and application programs 5012 of customers who use the customer terminal 50 .
The application program 5012 may be stored in the storage unit 501 in advance, or may be downloaded from a web server or the like operated by the service provider via the communication IF. The application program 5012 includes an interpreted programming language such as JavaScript (registered trademark) executed on a web browser application stored in the customer terminal 50 .

<Configuration of Control Unit of Customer Terminal 50>
The control section 504 of the customer terminal 50 has an input control section 5041 and an output control section 5042 . By executing the application program 5012 stored in the storage unit 501 , the control unit 504 realizes the functional units of the input control unit 5041 and the output control unit 5042 .
The input control unit 5041 of the customer terminal 50 outputs from input devices such as the user's operation content to the touch sensitive device 5061 of the touch panel 506, voice input to the microphone 5081, position information sensor 5083, camera 5084, motion sensor 5085, etc. Acquire information and execute various processes.
The output control unit 5042 of the customer terminal 50 receives information from the user's operation on the input device and the server 10, and executes control processing of display contents of the display 5062, audio output contents of the speaker 5082, and the like.

<Operation of information processing system 1>
Each process of the information processing system 1 will be described below.
FIG. 14 is a flow chart showing the operation of summary processing (first embodiment).
FIG. 15 is a flow chart showing the operation of summary processing (second embodiment).
FIG. 16 is a flow chart showing the operation of the response memo adding process.
FIG. 17 is a flow chart showing the operation of the response memo proposal process.
FIG. 18 is a flow chart showing the operation of the score calculation process.
FIG. 19 is a flow chart showing the operation of CRM storage processing.
FIG. 20 is a flow chart showing the operation of call display processing.
FIG. 21 is a diagram showing an overview of binding processing in summary processing (first embodiment).
FIG. 22 is a diagram showing an outline of additional extraction processing in summarization processing (second embodiment).
FIG. 23 is a diagram showing an overview of summary display processing.
FIG. 24 is a diagram showing a screen example of the CRM service in call display processing.

<Definition of terms>
In describing each process of the information processing system 1, terms are defined as follows.
The call data is data related to calls made between the user and the customer, and includes data stored in each item of the call table 1014 and data stored in each item of the voice recognition table 1015 associated with the call ID. , data stored in each item of the summary table 1016 associated with the call ID. This data includes data stored in each item of the response memo table 1017 and the response history table 3013 associated with the call ID.
The call attribute is data related to the attribute of a call made between the user and the customer, and includes user attribute, organization name or organization attribute of the organization to which the user belongs, customer attribute, organization name or organization attribute of the organization to which the customer belongs. , call category, caller type, etc. In other words, call data is characterized by attribute values such as the user attribute of the user who makes the call, the customer attribute of the customer who makes the call, the call category of the call, and the type of caller/receiver.

<Outgoing process>
The calling process is a process of making a call (calling) from the user to the customer.

<Outline of call processing>
The calling process is a series of processes in which the user selects a customer who wishes to make a call from among a plurality of customers displayed on the screen of the user terminal 20 and performs a calling operation to make a call to the customer.

<Details of outgoing processing>
A call processing of the information processing system 1 when a user makes a call to a customer will be described.

When a user calls a customer, the information processing system 1 performs the following processes.

By operating the user terminal 20 , the user activates the web browser and accesses the CRM service website provided by the CRM system 30 . The user can display a list of his/her own customers on the display 2081 of the user terminal 20 by opening the customer management screen provided by the CRM service.
Specifically, the user terminal 20 transmits a CRM ID 2013 and a request to display a list of customers to the CRM system 30 . When the CRM system 30 receives the request, it searches the customer table 3012 and transmits to the user terminal 20 information about the user's customer, such as customer ID, name, telephone number, customer attributes, customer organization name, and customer organization attributes. The user terminal 20 displays the received customer information on the display 2081 of the user terminal 20 .

The user presses and selects a customer who wishes to make a call from the customers listed on the display 2081 of the user terminal 20 . With the customer selected, pressing the "call" button or phone number button displayed on the display 2081 of the user terminal 20 sends a request including the phone number to the CRM system 30 . The CRM system 30 that has received the request transmits the request including the telephone number to the server 10 . The server 10 that has received the request transmits a call origination request to the voice server (PBX) 40 . Upon receiving the call request, the voice server (PBX) 40 makes a call (call) to the customer terminal 50 based on the received telephone number.

Along with this, the user terminal 20 controls the speaker 2082 and the like to ring to indicate that the voice server (PBX) 40 is making a call (calling). Also, the display 2081 of the user terminal 20 displays information indicating that the voice server (PBX) 40 is making a call to the customer. For example, the display 2081 of the user terminal 20 may display the characters "calling".

When the customer picks up the receiver (not shown) of the customer terminal 50 or presses the "Receive" button displayed on the touch panel 506 of the customer terminal 50 when receiving a call, the customer terminal 50 becomes ready for communication. Along with this, the voice server (PBX) 40 transmits information indicating that the customer terminal 50 has made a response (hereinafter referred to as "response event") to the user terminal 20 via the server 10, the CRM system 30, and the like. Send.
As a result, the user and the customer are ready to talk using the user terminal 20 and the customer terminal 50, respectively, and can talk between the user and the customer. Specifically, the user's voice collected by the microphone 2062 of the user terminal 20 is output from the speaker 5082 of the customer terminal 50 . Similarly, the customer's voice collected from the microphone 5081 of the customer terminal 50 is output from the speaker 2082 of the user terminal 20 .

The display 2081 of the user terminal 20 receives the response event and displays information indicating that a call is being made when the call is ready. For example, the display 2081 of the user terminal 20 may display the characters "answering".

When the call is enabled, call storage processing and CRM storage processing, which will be described later, are executed.

<Incoming processing>
Incoming call processing is processing in which a user receives a call (receives a call) from a customer.

<Overview of Incoming Call Processing>
The incoming call process is a series of processes in which the user receives an incoming call when the user has started an application on the user terminal 20 and the customer has made a call to the user.

<Details of incoming call processing>
Incoming call processing of the information processing system 1 when the user receives an incoming call (receives a call) from a customer will be described.

When a user receives an incoming call from a customer, the information processing system 1 performs the following processes.

By operating the user terminal 20 , the user activates the web browser and accesses the CRM service website provided by the CRM system 30 . At this time, it is assumed that the user logs in to the CRM system 30 with his own account on the web browser and waits. It is sufficient for the user to be logged in to the CRM system 30, and the user may be performing other work related to the CRM service.

The customer operates the customer terminal 50 , inputs a predetermined telephone number assigned to the voice server (PBX) 40 , and makes a call to the voice server (PBX) 40 . The voice server (PBX) 40 receives the outgoing call from the customer terminal 50 as an incoming call event.

A voice server (PBX) 40 sends an incoming call event to the server 10 . Specifically, the voice server (PBX) 40 transmits an incoming call request including the customer's telephone number 5011 to the server 10 . The server 10 transmits an incoming call request to the user terminal 20 via the CRM system 30 .
Along with this, the user terminal 20 controls the speaker 2082 and the like to ring to indicate that the voice server (PBX) 40 is receiving an incoming call. The display 2081 of the user terminal 20 displays information indicating that the voice server (PBX) 40 has received an incoming call from the customer. For example, the display 2081 of the user terminal 20 may display the characters "incoming call".

The user terminal 20 receives a response operation by the user. The response operation is, for example, by lifting the handset (not shown) of the user terminal 20, or by operating the mouse 2066 to press the button labeled "answer the call" on the display 2081 of the user terminal 20. Realized.
Upon accepting the response operation, the user terminal 20 transmits a response request to the voice server (PBX) 40 via the CRM system 30 and the server 10 . The voice server (PBX) 40 receives the transmitted response request and establishes voice communication. As a result, the user terminal 20 becomes ready for communication with the customer terminal 50 .
The display 2081 of the user terminal 20 displays information indicating that a call is being made. For example, the display 2081 of the user terminal 20 may display the characters "busy".

When the call is enabled, call storage processing and CRM storage processing, which will be described later, are executed.

<Call memory processing>
A call storage process is the process of storing data relating to calls made between a user and a customer.

<Overview of call memory processing>
The call storage process is a series of processes for storing data related to calls in the call table 1014 when a call is started between a user and a customer.

<Details of call memory processing>
When a call is started between the user and the customer, the voice server (PBX) 40 records voice data regarding the call between the user and the customer and transmits the data to the server 10 . When the voice data is received, the control unit 104 of the server 10 creates a new record in the call table 1014 and stores the data regarding the call made between the user and the customer. Specifically, the control unit 104 of the server 10 stores the user ID, customer ID, call category, incoming/outgoing type, and contents of voice data in the call table 1014 .

The control unit 104 of the server 10 acquires the user ID 2011 of the user from the user terminal 20 in the outgoing call process or incoming call process, and stores it in the user ID item of the new record.
The control unit 104 of the server 10 makes an inquiry to the CRM system 30 based on the telephone number in the outgoing call processing or incoming call processing. The CRM system 30 acquires the customer ID by searching the customer table 3012 by telephone number and transmits it to the server 10 . The control unit 104 of the server 10 stores the acquired customer ID in the customer ID item of the new record.
The control unit 104 of the server 10 stores the value of the call category set in advance for each user or customer in the call category item of the new record. Note that the call category may be stored by the user selecting or inputting a value for each call.
The control unit 104 of the server 10 identifies whether the call being made is originated by the user or originated by the customer, and puts outbound (originating from the user) and inbound in the incoming and outgoing call type item of the new record. (originating from the customer).
The control unit 104 of the server 10 stores the voice data received from the voice server (PBX) 40 in the voice data item of the new record. It should be noted that the voice data may be stored as a voice data file in another location, and the reference information (path) to the voice data file may be stored after the end of the call. Also, the control unit 104 of the server 10 may be configured to store data after the end of the call.

<Voice recognition processing>
The voice recognition process is a process of converting voice data of a call between a user and a customer into text information and storing the data by text recognition.

<Overview of speech recognition processing>
In the speech recognition process, the speech data stored in the call table 1014 is divided into sections (speech sections) in which speech exists, and section speech data is extracted. This is a series of processes in which speech recognition information is generated by performing text recognition on each section speech data and stored in the speech recognition table 1015 .

<Details of voice recognition processing>
The recognition unit 1043 of the server 10 searches the call table 1014 for records for which the call memory processing has been performed but the voice recognition processing has not been performed. Specifically, the recognition unit 1043 of the server 10 selects a record in which information such as blank, null, or other information indicating that the voice recognition process is not performed is stored in the voice recognition presence/absence item from the call table 1014. search for. Note that the recognition unit 1043 of the server 10 may determine a record in which voice data is stored in the call table 1014 but no call ID exists in the voice recognition table 1015 as a record in which voice recognition processing has not been performed. .

The recognition unit 1043 of the server 10 acquires (accepts) the call ID and voice data of the record for which voice recognition processing has not been performed. The recognition unit 1043 of the server 10 detects segments (utterance segments) in which voice exists from the acquired (accepted) voice data, and extracts voice data for each of the utterance segments as segment voice data. The segment voice data is associated with the speaker and the utterance time for each utterance segment.
The recognition unit 1043 of the server 10 performs text recognition on the extracted segmental voice data, thereby converting the segmental voice data into characters (text). Note that the specific method of text recognition is not particularly limited. For example, it may be converted by machine learning or deep learning using signal processing technology, AI (artificial intelligence).

The recognition unit 1043 of the server 10 associates a series of data in which the text for each speech period is associated with the start time of each speech period and the speaker (user or customer) with the call ID to be processed, and generates a speech recognition table. Store in 1015 . The recognition unit 1043 of the server 10 stores information indicating that the voice recognition process has been completed in the voice recognition presence/absence item of the call table 1014 .

In the speech recognition table 1015, the text for each utterance section of the voice data is associated with the utterance time and the speaker and stored as continuous time-series data. By checking the data stored in the voice recognition table 1015, the user can check the content of the call as text information without listening to the content of the voice data.

Note that the utterance time may be an arbitrary time related to each utterance period, such as the start time of each utterance period, the end time of each utterance period, or the time between any utterance periods.

<Modified example of speech recognition processing>
In addition, during the speech recognition process, information such as fillers included in the text that is meaningless in terms of understanding the call between the user and the customer is removed from the text in advance, and the speech recognition information is converted to speech recognition. It may be configured to be stored in the table 1015 .

<Speech analysis processing>
The voice analysis process is a process of analyzing voice data of a call made between a user and a customer, generating analysis data, and storing the data.

<Outline of speech analysis processing>
The voice analysis process is a series of processes in which analysis data is generated by executing voice analysis process on voice data stored in the call table 1014 and stored in the call table 1014 .

<Details of voice analysis processing>
The analysis unit 1044 of the server 10 searches the call table 1014 for records for which the call memory processing has been performed but the voice analysis processing has not been performed. Specifically, the analysis unit 1044 of the server 10 searches the call table 1014 for a record in which voice data is stored but analysis data is not stored. The fact that the analysis data is not stored means that the item of the analysis data is blank, null, or other information indicating that it is not stored.
In addition, information indicating that the record is to be subjected to the voice analysis process may be stored in a column (not shown), and the record in which the information is stored may be subjected to the voice analysis process.

The server 10 acquires the call ID and voice data of records for which voice analysis processing has not been performed. The analysis unit 1044 of the server 10 analyzes the acquired voice data, Talk:Listen ratio, Silence count, Cover count, Rally count, Fundamental frequency, Inflection strength, Speech speed, Speech speed ratio, Filler count, Talk script match. Calculate degrees, etc. Note that the specific method of analysis is not particularly limited. For example, analysis may be performed by machine learning or deep learning using signal processing technology, AI (artificial intelligence).

The Talk:Listen ratio is the ratio between the user's speaking time and the callee's speaking time.
The number of silences is the number of times silence occurred in the call. The number of times of confusion is the number of times that confusion occurs in a call.
The number of rallies is the number of times the user and the customer interacted with each other in a call (the number of times the conversation was switched).
The fundamental frequency is defined for each user or customer, and is information corresponding to the fundamental frequency of the voice of the user or customer, that is, the pitch of the voice.
The intensity of intonation is defined for each user or customer, and is information obtained by quantitatively evaluating the magnitude of intonation of the user or customer.
Speech speed is defined for each user or customer, and is the speaking speed of the user or customer. The speed of speech is indicated by, for example, the number of characters (or the number of words) uttered in one second.
The speech speed ratio is information regarding the ratio of speech speeds of the user and the customer. Specifically, it is indicated as a numerical value obtained by dividing the customer's speech rate from the user's speech rate. For example, the higher the value, the faster the user speaks than the customer.
The number of fillers is the number of detected fillers (for example, hesitation such as ah, ah, etc.) in the sentence of the speech recognition data. The number of fillers may be defined for each user, customer.
The talk script matching degree is information regarding the degree of matching between the talk script set for each user or the organization to which the user belongs and the user's utterance content detected from the speech recognition data.

The analysis unit 1044 of the server 10 stores the analysis data in the analysis data item of the record to be processed in the call table 1014 .

<Summary processing (first embodiment)>
Summarization processing (first embodiment) is processing for generating and storing summary information from speech recognition information. Details of the summarization process (first embodiment) will be described below with reference to the flowchart of FIG.

<Overview of summary processing (first embodiment)>
Summarization processing (first embodiment) acquires the speech recognition information stored in the speech recognition table 1015, and arranges the text included in the speech recognition information in the order of utterance time. Execute binding processing to bind text as a text group, calculate the importance of the text after binding processing and the text group, extract the text to be used for the summary document based on the importance, generate the summary document, and server 10 and the CRM system 30.

<Details of summary processing (first embodiment)>
The summarizing unit 1046 of the server 10 automatically executes the summarizing process (first embodiment) periodically.
In step S101, the summarizing unit 1046 of the server 10 periodically starts summarizing processing (first embodiment). Note that the summarizing unit 1046 of the server 10 may detect the end of the call between the user and the customer and start the summarizing process (first embodiment).

In step S102, the summarizing unit 1046 of the server 10 refers to the call table 1014 and searches for a record in which information indicating that the voice recognition process has been completed is stored in the voice recognition presence/absence item. When the summarizing unit 1046 of the server 10 finds a record storing information indicating that the speech recognition processing has been completed, it acquires the record and proceeds to step S103 (Yes in step S102).
After the end of the call between the user and the customer, until the voice recognition process is completed, information indicating that the voice recognition process has been completed is displayed in the voice recognition presence/absence field of the corresponding record in the call table 1014. Since it is not stored, the summary unit of the server 10 waits in step S102 (No in step S102).

In step S103, the summarization unit 1046 of the server 10 stores information such as blanks, nulls, and other information indicating that the summarization process (first embodiment) has not been performed in the summary presence/absence item of the record acquired in step S102. Search for records that have Note that the summarization unit 1046 of the server 10 determines that a record in which voice data is stored in the call table 1014 but whose call ID does not exist in the summary table 1016 is a record in which the summarization process (first embodiment) has not been performed. You can
When the summarizing unit 1046 of the server 10 finds a record that has not undergone the summarizing process (first embodiment), it acquires the record and proceeds to step S104 (Yes in step S102).

In step S104, the summarizing unit 1046 of the server 10 searches the speech recognition table 1015 based on the call ID of the record that has not undergone the summarizing process (first embodiment), and finds the record consisting of text, utterance time, and speaker record. Get speech recognition information.

In step S105, the summarizing unit 1046 of the server 10 performs binding processing on the acquired speech recognition information. Specifically, when the acquired speech recognition information is arranged in order of utterance time, a text group is generated by collecting a plurality of texts before and after the speaker information changes. A text group is a data structure such as an array, and is information including a plurality of texts by different speakers. The set of texts may also include other speech recognition information such as time of utterance, speaker, and so on.

FIG. 21 is a diagram showing an overview of the binding process in the summarization process (first embodiment), in which texts for each speaker are arranged downward in order of utterance time. Identification numbers such as U1, U2, . . . , C1, C2 . The summarizing unit 1046 of the server 10 generates a text group by collecting the texts of C4 and U5 whose speaker information changes from the customer to the user, for example.

In the present disclosure, a text group is generated by combining one text from each of the user and the customer, but two or more texts before and after the change in speaker information may be combined to generate a text group. . The importance may be calculated in advance for the texts before and after the speaker information changes (for example, C4, U5, etc.), and a text group may be generated by collecting the number of texts according to the value of the importance. For example, a text group may be generated by collecting more texts as the degree of importance is higher.
Even in conversations involving three or more parties, a text group may be generated by summarizing a plurality of texts before and after the change in speaker information.

In step S106, the importance calculation unit 1045 of the server 10 performs importance calculation processing on the text and the text group. Any important sentence extraction algorithm may be applied to the calculation method of importance.
In the present disclosure, a method of calculating importance by an algorithm called LexRank will be described as an example. In LexRank, a plurality of input sentences are targeted, each input sentence is divided into words by morphological analysis, the similarity for each sentence (for example, cosine similarity) is calculated, and the similarity between sentences is calculated. It is an algorithm that calculates the graph structure and calculates the importance of each sentence based on the graph structure.
Specifically, the importance calculation unit 1045 of the server 10 treats each of the text and the text group as one input sentence, and applies LexRank to calculate the importance of each text and text group.

<Importance degree calculation processing (first embodiment)>
In the importance calculation process (first embodiment), the importance calculation unit 1045 of the server 10 treats a text obtained by combining a plurality of texts included in a text group as one sentence and applies LexRank. For example, C4 "Is Mr. Taguchi here?" and U5 "I'm Taguchi." Calculate.

<Importance degree calculation processing (second embodiment)>
In the importance calculation process (second embodiment), the importance calculation unit 1045 of the server 10 applies LexRank to each of the plurality of texts included in the text group as independent sentences. The importance calculation unit 1045 of the server 10 executes a process of setting the sum of the importances calculated for the independent sentences as the importance of the text group.
For example, the importance calculator 1045 of the server 10 applies LexRank to each sentence of C4 and U5 included in the text group, and calculates the importance of each of C4 and U5. After that, the importance calculation unit 1045 of the server 10 executes a process of setting the sum of the importance of C4 and the importance of U5 as the importance of the text group.
Note that statistical values (average, median, mode, maximum, minimum, etc.) obtained by statistically processing the importance calculated for independent sentences may be used as the importance of the text group. . Specifically, the average or maximum value of importance calculated for independent sentences is suitable as the importance of the text group.

In step S107, the summarizing unit 1046 of the server 10 extracts texts or text groups for which the degree of importance of a predetermined value or more has been calculated. Summarization unit 1046 of server 10 stores the extracted texts and texts included in the text group in summary table 1016 together with the utterance time and speaker of each text in speech recognition table 1015 . As a result, summary information is stored in summary table 1016 . The summarizing unit 1046 of the server 10 stores information indicating that the summarizing process (first embodiment) has been completed in the summarization presence/absence item of the call table 1014 .

The summarizing unit 1046 of the server 10 combines the extracted texts and texts included in the text group into one sentence (summary text) together with the utterance time and speaker of each text in the speech recognition table 1015, the call ID and the summary. Generate a request containing the text and send it to the CRM system 30 .
When the request is received, the CRM system 30 searches the response history table 3013 from the call ID included in the request, and stores the received summary text in the comment field of the record having the call ID.

<Summary processing (second embodiment)>
Summarization processing (second embodiment) is a second embodiment of processing for generating and storing summary information from speech recognition information. Details of the summarization process (second embodiment) will be described below with reference to the flowchart of FIG.

<Overview of Summary Processing (Second Embodiment)>
The summarization process (second embodiment) acquires the speech recognition information stored in the speech recognition table 1015, calculates the importance of the text included in the speech recognition information, and uses it for the summary document based on the importance. Extract text. When the texts included in the speech recognition information are arranged in the order of utterance time, the text before and after the speaker changes in the extracted text is extracted as the text to be used for the summary document, and the summary document is generated. It is a series of processes stored in the CRM system 30 .

<Details of summary processing (second embodiment)>
The processing from step S201 to step S204 is the same as the processing from step S101 to step S104 of the summarization processing (first embodiment), so description thereof is omitted.

In step S205, the importance calculation unit 1045 of the server 10 performs importance calculation processing on the acquired speech recognition information. Any important sentence extraction algorithm may be applied to the method of calculating the importance, but in the present disclosure, an example of the method of calculating the importance by an algorithm called LexRank is used as in the summarization process (second embodiment). described as.

<Importance degree calculation processing (third embodiment)>
The importance calculation unit 1045 of the server 10 treats the text included in the acquired speech recognition information as one sentence and applies LexRank. FIG. 22 is a diagram showing an outline of additional extraction processing in summarization processing (second embodiment), in which texts for each speaker are arranged downward in order of utterance time. In the case of FIG. 22, the importance is calculated for all the texts U1 to U8 and C1 to C6.

In step S206, the summarizing unit 1046 of the server 10 extracts texts for which the degree of importance is calculated to be equal to or greater than a predetermined value. The summarizing unit 1046 of the server 10 additionally extracts texts that are different in speaker from the extracted text and whose utterance times are before and after.
For example, when the text U5 is extracted, the text C4 that is spoken by a different speaker and whose utterance time is before or after is additionally extracted.

In addition, in the present disclosure, one text is additionally extracted from different speakers and whose utterance time is preceded or followed, but two or more texts may be additionally extracted. A number of additional texts may be extracted according to the importance value of the extracted texts. For example, the greater the importance, the more texts of different speakers may be additionally extracted at the utterance time.
Also in a conversation involving three or more parties, a plurality of texts before and after the change in speaker information may be additionally extracted.

Since the process of step S207 is the same as the process of step S107 of the summarization process (first embodiment), description thereof is omitted.

<Modified example of summary processing>
In the summarization process (first embodiment) and the summarization process (second embodiment), the importance calculation unit 1045 of the server 10 calculates the importance of the text by calculating the fillers included in the text between the user and the customer. Importance may be calculated by excluding non-meaningful information in grasping the calls that have been made.
Similarly, during the summarization process (first embodiment) and the summarization process (second embodiment), it is meaningless to grasp the call made between the user and the customer, such as fillers included in the text in advance. Such information may be excluded from the text, and the summary information may be stored in the summary table 1016. FIG.

<Summary display processing>
Summary display processing is processing for displaying summary information to the user as a summary document. The details of the summary display process will be described below using the screen example of FIG.

<Overview of summary display processing>
The summary display process acquires the summary information stored in the summary table 1016, stores the text included in the summary information in balloons of different colors for each speaker, arranges them in order of utterance time, and displays them as a summary document on the user terminal 20. This is a series of processing for displaying on the display 2081 of .

<Details of summary display processing>
A user logs in to the server 10 and performs a predetermined operation to send a request to the server 10 for displaying a list of call histories that he or his colleagues in the organization to which he belongs have made with customers in the past. Upon receiving the request, the server 10 refers to the call table 1014 and transmits a record of past call history to the user terminal 20 . The user terminal 20 displays a list of the received past call history records on the display 2081 of the user terminal 20 .

The user operates the user terminal 20 to select a call history record for which summary information is to be confirmed, and presses a summary display button or the like to send a request to the server 10 for summary display, including the call ID, of the selected call history. Send.
Upon receiving the request, the server 10 searches the summary table 1016 based on the call ID and transmits summary information regarding the call ID to the user terminal 20 .

User terminal 20 displays the received summary information on display 2081 as a summary document. A summary document displayed on the user terminal 20 will be described with reference to FIG. The display 2081 of the user terminal 20 displays the summary display screen 70. FIG. On the summary display screen, the content (text) of the user's utterance is displayed together with an utterance time 701 in a balloon 702 , and the customer's utterance content (text) is displayed together with an utterance time 711 in a balloon 712 . A speech balloon 702 displaying the user's utterance contents is displayed so as to be identifiable from a speech balloon 712 displaying the customer's utterance contents whose direction of the speech balloon is directed to the left and which is directed to the right.
The display control unit 1052 of the user terminal 20 displays the text in which the speaker is the user from the upper part of the summary display screen 70 in order of the utterance time based on the text, utterance time, and speaker information included in the acquired summary information. A speech bubble 702 displaying the user's utterance content displays a list of texts in which the speaker is the customer in a distinguishable manner in a speech balloon 712 displaying the customer's utterance content.

The color of balloon 702 displaying the user's utterance content is different from the color of balloon 712 displaying the customer's utterance content. This allows the user to visually confirm the speaker of the utterance content when confirming the summary document. In addition, the color of the speech balloons 702 and 712 displaying the utterance content is determined based on the importance of the text included in the utterance content calculated in the summarization process (first embodiment) and the summarization process (second embodiment). , at least one of hue, saturation, and density may be changed for display. For example, balloons 702 and 712 containing text of higher importance may be colored darker. As a result, the user can visually confirm the importance of each utterance content when confirming the summary document, and can understand the text content in a short period of time.

<Voice Recognition Display Processing>
The voice recognition display process is a process of displaying voice recognition information as a voice recognition document to the user.

<Overview of speech recognition display processing>
The speech recognition display process acquires the speech recognition information stored in the speech recognition table 1015, stores the text included in the speech recognition information in balloons of different colors for each speaker, arranges them in order of utterance time, and performs speech recognition. This is a series of processes for displaying on the display 2081 of the user terminal 20 as a document.

<Details of speech recognition display processing>
The speech recognition display process uses the speech recognition information instead of the summary information in the summary display process, that is, the speech recognition table 1015 is used instead of the summary table 1016, and the processing contents are the same, so the description is omitted. In the voice recognition display process, a list of voice recognition information is displayed in balloon form on a voice recognition display screen substantially the same as that shown in FIG.
In the speech recognition display process, the color of the speech balloon displaying the content of the user's utterance is different from the color of the speech balloon displaying the content of the customer's utterance. This allows the user to visually confirm the speaker of the utterance content when confirming the speech recognition document. In addition, the color of the speech balloon displaying the utterance content is determined based on the degree of importance of the text included in the utterance content calculated in the summary processing (first embodiment) and the summary processing (second embodiment). At least one of saturation and density may be changed for display. For example, the color of a balloon containing text of higher importance may be darker. As a result, when the user confirms the speech recognition document, the user can visually confirm the importance of each utterance content, and can understand the sentence content in a short time. That is, even when the user confirms all the voice recognition information of the call made between the user and the customer, the user can visually and intuitively grasp only the important utterance contents.

<Response memo addition processing>
The response memo adding process is a process of automatically adding a response memo to a call between the user and the customer. The details of the response memo adding process will be described below with reference to the flowchart of FIG.

<Overview of processing to add a response memo>
In the answering memo adding process, voice data relating to a call between a user and a customer is obtained, a learning model is applied to the voice data to infer answering memo candidates, and the inferred answering memo candidates are sent to the relevant party. This is a series of processes that are stored in association with call data related to the call.

<Details of response memo addition processing>
In step S301, the response memo adding unit 1049 of the server 10 detects the end of the call between the user and the customer, and starts the response memo adding process.

In step S302, the response memo adding unit 1049 of the server 10 acquires the call attribute of the call to be processed.
Specifically, the response memo adding unit 1049 of the server 10 searches the call table 1014 based on the call ID of the call to be processed, and acquires the call category and incoming/outgoing type.
The response memo adding unit 1049 of the server 10 searches the user table 1012 based on the user ID of the call to be processed and acquires the user attribute. The response memo adding unit 1049 of the server 10 searches the organization table 1013 from the organization ID stored in the user table 1012 based on the user ID of the call, and acquires the organization name and organization attributes of the organization to which the user belongs.
The response memo adding unit 1049 of the server 10 makes an inquiry to the CRM system 30 based on the customer ID of the call to be processed, and acquires the customer attribute, the customer organization name, and the customer organization attribute from the customer table 3012 of the CRM system 30. .
The response memo adding unit 1049 of the server 10 does not need to acquire all call attributes, and may acquire at least one call attribute among a plurality of call attributes as necessary.

In step S303, the response memo adding unit 1049 of the server 10 selects a learning model based on the acquired call attributes. A learning model may be prepared for each call attribute, or may be prepared for each combination of a plurality of call attributes, for example, each combination of user attributes and customer attributes.
The learning model includes arbitrary machine learning, deep learning models, etc., and is learned from a data set created according to call attributes. Details of the learning process will be described later.
In this disclosure, a deep learning model will be described as an example of a learning model. The deep learning model is any deep learning model that takes arbitrary time series data as input data, such as RNN (Recurrent Neural Network), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), etc. It doesn't matter if there is. A learning model includes, for example, any deep learning model including Attention, Transformer, and the like.

In step S304, the response memo adding unit 1049 of the server 10 acquires the voice data of the call to be processed, applies the voice data as input data of the selected learning model, and outputs a plurality of response memos as output data. Candidates (response memo group) are output (deduced) together with probability distributions.
Specifically, probability distributions such as "0.6", "0.3", and "0.1" are output for correspondence notes "AAA", "BBB", "CCC", and the like, respectively. The probability distribution may or may not be normalized by a softmax function or the like.

In step S305, the response memo adding unit 1049 of the server 10 associates the response memo candidate with the highest probability among the output response memo candidates with the call ID of the call to be processed, and adds the memo contents of the response memo table 1017 to the memo content of the response memo table 1017. stored in the item The current date and time may be stored in the item of date and time of grant. Information indicating that a system other than the user, such as the server 10, has automatically given the ID may be stored in the item of the giver ID.
Note that the response memo adding unit 1049 of the server 10 may store, among the output response memo candidates, a plurality of response memo candidates whose probability is equal to or greater than a predetermined value in the item of memo contents of the response memo table 1017 .

<Response memo addition processing (modification)>
In step S303, the response memo adding unit 1049 of the server 10 may select a plurality of learning models based on a plurality of different call attributes. For example, the response memo adding unit 1049 of the server 10 may select a first learning model prepared for each user attribute and a second learning model prepared for each customer attribute. At this time, the response memo adding unit 1049 of the server 10 may select a plurality of learning models based on a plurality of arbitrary call attributes.

In step S304, the response memo adding unit 1049 of the server 10 acquires the voice data of the call to be processed, and applies the voice data as input data for a plurality of selected learning models. Then, a plurality of answering memo candidates are output (deduced) as output data together with the probability distribution.
The reception memo adding unit 1049 of the server 10 may calculate the probability distribution for each reception memo candidate by applying an arbitrary operation to the probability distribution of a plurality of reception memo candidates. For example, the sum or product of the probability distributions of a plurality of answering memo candidates may be used as the probability distribution for each answering memo candidate to be output.
Specifically, the inference results of the probability distribution for the first learning model are "0.6", "0.3", "0. 1”, and when the inference results of the probability distribution for the second learning model are “0.3”, “0.4”, and “0.1”, their products “0.18”, “0 .12” and “0.01” may be used as the probability distributions of the response memos “AAA”, “BBB” and “CCC”. The probability distribution may or may not be normalized by a softmax function or the like.

<Response memo proposal processing>
The answering memo proposing process is a process of proposing answering memo candidates to the user in a call between the user and the customer. Details of the response memo adding process will be described below with reference to the flowchart of FIG.

<Overview of response memo proposal processing>
The processing for proposing a response memo acquires voice data relating to a call between a user and a customer, applies a learning model to the voice data, infers a candidate for a response memo, and presents the inferred candidate for a response memo to the user. This is a series of processes in which an answering memo candidate selected by the user is stored in association with call data related to the call.

<Details of response memo proposal processing>
Since the processing from step S401 to step S404 is the same as the processing from step S301 to step S304 of the reception memo addition processing, description thereof is omitted.

In step S<b>405 , the answering memo proposal unit 1048 of the server 10 transmits the output answering memo candidate and probability distribution to the user terminal 20 . The display 2081 of the user terminal 20 displays a list of the received answering memo candidates so that the user can select them.
At this time, the user terminal 20 may display the probability of each candidate of the answering memo as the priority of the candidate of the answering memo, and display the candidate of the answering memo with a higher priority at a position on the display 2081 of the user terminal 20 that is easier for the user to select. good. For example, it is conceivable to display the answering memo candidate with the highest priority at the top of the list, or to display a plurality of answering memo candidates by carousel or page turning, to display them on the first page.
That is, the user terminal 20 displays the answering memo candidate with a high probability at a position on the display 2081 of the user terminal 20 where the user can easily select it. This allows the user to more accurately and easily select a more likely response memo from a plurality of response memos.

In step S<b>406 , the user selects one or more reception memo candidates from the reception memo candidates displayed on the display 2081 of the user terminal 20 . By pressing a send button displayed on the display 2081 of the user terminal 20 , the user sends the selected answering memo candidate and the user ID 2011 to the server 10 .

In step S407, the response memo adding unit 1049 of the server 10 associates the received response memo candidate and user ID 2011 with the call ID of the call to be processed, and stores them in the items of memo content and grantor ID of the response memo table 1017. do. The current date and time may be stored in the item of date and time of grant.

<Learning processing>
The learning process is a process of constructing a learning model to be used in a response memo addition process, a response memo proposal process, and the like.

<Overview of learning process>
In the learning process, for each call attribute related to the call, the voice data related to the call between the user and the customer is used as input data, and the memo content linked to the voice data is used as output data (output vector) of the learning model. This is a process for learning parameters.

<Details of learning process>
The learning unit 1047 of the server 10 searches the call table 1014 and acquires the contents of the memo linked to the voice data by referring to the response memo table 1017 via the voice data and the call ID. The learning unit 1047 of the server 10 divides the voice data for each call attribute, and creates data sets such as training data, test data, and verification data for each call attribute.
The learning unit 1047 of the server 10 allows the learning model prepared for each call attribute to learn parameters of the learning model by using a data set corresponding to each call attribute.

The learning unit 1047 of the server 10 may exclude voice data associated with a predetermined response memo. Specifically, when creating a data set, the learning unit 1047 of the server 10 determines whether the contents of the memo are "answering machine", "customer (person in charge) absent", "customer reception block", etc. Exclude voice data relating to a call to which a memo indicating that the call is not actually established. As a result, a more accurate learning model can be created by excluding from the data set data that is not desirable for creating a learning model for inferring the content of the memo.

As an example, the learning unit 1047 of the server 10 may create a data set for each piece of information related to any one of the job type, the type of organization to which the user belongs, and the name of the organization to which the user makes a call.
As an example, the learning unit 1047 of the server 10 may create a data set for each piece of information related to one of the customer attributes of the job type of the customer making the call, the industry of the organization to which the customer belongs, and the name of the organization to which the customer belongs. ,
As an example, the learning unit 1047 of the server 10 may create a data set for each information regarding the call category of calls made, such as telephone operator, telemarketing, customer support, technical support, and the like.
As an example, the learning unit 1047 of the server 10 may create a data set for each piece of information relating to either outbound sent from the user to the customer or inbound received from the customer by the user.

When the learning unit 1047 of the server 10 learns the learning model for each user attribute of the user who makes the call, the learning process is performed using as input data the process of extracting only the user's voice (excluding the customer's voice). may be performed. In this case, only the user's voice is similarly extracted (the customer's voice is excluded) for voice data applied to the learning model in the inference processing in steps S304 and S404 of the response memo addition processing and the response memo proposal processing. Perform processing using the processed data as input data.
A learning model according to user attributes can obtain a learning model that can more accurately infer a response memo by making the learning model learn based only on the contents of the user's utterances.

When the learning unit 1047 of the server 10 learns a learning model for each customer attribute of a customer who makes a call, the learning process is performed using as input data a process of extracting only the customer's voice (excluding the user's voice). may be performed. In this case, only the customer's voice is similarly extracted (user's voice is excluded) for the speech data applied to the learning model in the inference processing in steps S304 and S404 of the response memo addition processing and the response memo proposal processing. Perform processing using the processed data as input data.
A learning model according to customer attributes can obtain a learning model that can more accurately infer a response memo by learning only based on the contents of the customer's utterances.

<Modified Example of Learning Processing>
In the present disclosure, voice data is used as the input data for the learning model, but data converted by performing arbitrary information processing on the voice data may be used as the input data. For example, instead of voice data, text data obtained by performing text recognition (text transcription) on the voice data may be used as input data. Data converted by performing arbitrary information processing on text data may be used as input data. In the present disclosure, applying speech data to a learning model also includes applying a learning model to data converted by performing arbitrary information processing on certain speech data.
In this case, when inferring an answering memo candidate in a learning model such as an answering memo addition process or an answering memo proposal process, the same information processing may be performed on voice data as input data, and the learning model may be applied. For example, text obtained by performing text recognition (text transcription) on voice data instead of voice data when inferring a response memo candidate in a learning model such as a response memo addition process, a response memo proposal process, etc. Data should be used as input data.

<Score calculation process>
The score calculation process is a process of calculating a call score for each user. The details of the score calculation process will be described below with reference to the flowchart of FIG.

<Overview of score calculation process>
The score calculation process is a process of calculating a call score for each user by performing analysis processing and statistical processing on call data for each user. Thereby, for example, the customer service skill of each user can be evaluated using a quantitative index.

<Details of score calculation processing>
In step S<b>501 , the user operates the user terminal 20 to transmit a user list request to the server 10 . Based on the received request, the score calculation unit 1050 of the server 10 obtains from the user table 1012 a list of users belonging to the same organization (having the same organization ID) as the user who sent the request, and transmits the list to the user terminal 20. .
The display 2081 of the user terminal 20 displays the obtained list of users so that the user can select one. The user selects a user whose score is to be calculated from the user list displayed on the display 2081 of the user terminal 20 . The user sends the user ID of the selected user to the server 10 by pressing the send button displayed on the display 2081 of the user terminal 20 .

In step S502, the score calculation unit 1050 of the server 10 searches the call table based on the received user ID, and acquires analysis data for each call of the target user.

At this time, the score calculation unit 1050 of the server 10 may exclude analysis data associated with a predetermined response memo. Specifically, when acquiring the analysis data for each call of the target user, the score calculation unit 1050 of the server 10 acquires the memo content of the response memo table 1017 associated with the call ID of the call. The score calculation unit 1050 of the server 10 determines that the contents of the memo are "answering machine", "customer (person in charge) absent", "customer reception block", etc., and that the communication with the customer has not been established. Excludes analysis data related to calls with memo content shown. Thereby, when calculating the call score of the target user, by excluding the calls in which the call is not established substantially, it is possible to calculate the call score with higher accuracy.

The score calculation unit 1050 of the server 10 calculates a call score by applying a predetermined algorithm to the analysis data for each call. Specifically, the score calculation unit 1050 of the server 10 calculates a predetermined weighted sum based on the degree of divergence of various index values (number of times of silence, number of times of being covered, number of times of rallies, etc.) from the reference index value, etc. included in the analysis data. Calculate the call score by taking At this time, arbitrary processing such as normalization may be applied to the index values.

In step S504, the score calculation unit 1050 of the server 10 performs statistical processing on the call score calculated based on the analysis data for each call of the target user (average, median, mode, maximum value, minimum value, etc.) as the user evaluation index for the target user. Specifically, the average call score calculated based on the analysis data for each call of the target user is suitable as the user evaluation index.

The score calculation unit 1050 of the server 10 stores the calculated user evaluation index in the evaluation index item of the record of the target user in the user table 1012 . The score calculation unit 1050 of the server 10 transmits the calculated user evaluation index to the user terminal 20 of the user who executed the score calculation process. The display 2081 of the user terminal 20 displays the received user evaluation index of the target user to the user.

<Link mode setting processing>
The cooperation mode setting process is a process for setting the storage process of the call information to the CRM system 30 in connection with the call storage process.

<Overview of link mode setting processing>
A user or an administrator of an organization or department to which the user belongs opens a predetermined website provided by the server 10 and displays an edit screen for setting the cooperation mode. The user can select a desired cooperation mode on the cooperation mode setting screen and execute a save operation or the like to set storage processing of call information to the CRM system 30 in the server 10 .
It should be noted that the cooperation mode setting may have a predetermined prescribed value in advance without being set by the user. The user may set this when using the voice call service according to the present disclosure for the first time.

<Details of link mode setting processing>
A user or an administrator of an organization or department to which the user belongs operates his or her own user terminal 20 and opens a website related to an edit screen for cooperation mode settings provided by the server 10 using a web browser or the like. The user or the like operates the user terminal 20 and selects either the first mode or the second mode as the desired cooperation mode on the cooperation mode setting screen. A user or the like operates the user terminal 20 to transmit his/her own user ID 2011 and the selected cooperation mode to the server 10 . The setting unit 1042 of the server 10 searches the user table 1012 with the received user ID 2011, and stores the received cooperation mode in the cooperation mode item of the record of the user.
Note that the cooperation mode may be stored for each organization to which the user belongs instead of for each user. That is, it may be stored in the item of cooperation mode provided in the organization table 1013, and the cooperation mode of each user may refer to the item of cooperation mode of the organization table 1013 linked by the organization ID.

<CRM storage processing>
The CRM storage process is a process for storing, in the CRM system 30, information regarding calls made between users and customers using the voice call service according to the present disclosure. The details of the CRM storage process will be described below with reference to the flowchart of FIG.

<Overview of CRM storage processing>
The CRM storage process acquires the cooperation mode set for each user or organization when a call is started between the user and the customer, and sends data related to the call to the CRM system 30 according to the setting value of the cooperation mode. It is a series of processing to be stored. As a result, the data related to the call is stored in the CRM system 30 in association with the customer information of the call target.

<Details of CRM storage processing>
In step S601, after the originating process or the receiving process (receiving/transmitting process) is performed, a call is started between the user and the customer. Thereby, the server 10 detects that a call has been started between the user and the customer.

In step S602, the CRM storage control unit 1051 of the server 10 searches the user table 1012 based on the user ID 2011 received from the user terminal 20, and acquires the cooperation mode of the user making the call. If the cooperation mode is stored in the organization table 1013 or the like for each organization, the organization table 1013 or the like is searched based on the organization ID of the user, and the cooperation mode linked to the user is determined by the user's organization. Set to cooperation mode.

In step S603, the CRM storage control unit 1051 of the server 10 determines whether the acquired cooperation mode is the first mode or the second mode. If the acquired cooperation mode is the first mode, the CRM storage control unit 1051 of the server 10 proceeds to step S604. If the acquired cooperation mode is the second mode, the CRM storage control unit 1051 of the server 10 skips step S604 and proceeds to step S605.

In step S604, the CRM storage control unit 1051 of the server 10 transmits to the CRM system 30 a request to store the first call data and the call data including the call ID in the call table 1014 of the call in association with the customer ID. The first call data is data related to a call that can be obtained from the start of the call until the end of the call.
Specifically, the CRM system 30 associates the call ID with the user ID of the user who is making the call, the customer ID of the customer, the date and time of the call (only in the case of call), and the date and time of the start of the call as the first call data. are stored in the fields of user ID, customer ID, dial date and time, and call start date and time of the response history table 3013, respectively. Note that the first call data may include at least one of the user ID, the customer ID of the customer, the dial date and time, and the call start date and time.

In step S605, the CRM storage control unit 1051 of the server 10 detects termination of the call between the user and the customer.

In step S606, the CRM storage control unit 1051 of the server 10 determines whether or not the voice analysis processing regarding the call has been completed. Specifically, the CRM storage control unit 1051 of the server 10 refers to the call table 1014, and the item of analysis data in the record of the call is blank, null, or other information indicating that the voice analysis process has not been completed. Determine whether or not it is stored.
If the speech analysis processing has not been completed, the CRM storage control unit 1051 of the server 10 waits in step S606. If the voice analysis process has been completed, the CRM storage control unit 1051 of the server 10 proceeds to step S607.

In step S607, the CRM storage control unit 1051 of the server 10 determines whether the acquired cooperation mode is the first mode or the second mode. If the acquired cooperation mode is the first mode, the CRM storage control unit 1051 of the server 10 proceeds to step S609. If the acquired cooperation mode is the second mode, the CRM storage control unit 1051 of the server 10 proceeds to step S608.

In step S608, the CRM storage control unit 1051 of the server 10 transmits to the CRM system 30 a request to store the second call data and the call data including the call ID in the call table 1014 of the call in association with the customer ID. The second call data is data related to a call that can be obtained after the call ends.
The CRM system 30 associates the call ID with the user ID of the user who is making the call, the customer ID of the customer, the date and time of the call (only in the case of call), the start date and time of the call, and the end date and time of the call as second call data. are stored in the fields of user ID, customer ID, dial date and time, call start date and time, and call end date and time in the response history table 3013, respectively. The CRM system 30 stores the speech recognition information about the call and the summary information about the call in the comment field of the response history table 3013 in association with the call ID. Specifically, like the speech recognition result and summary result of the comment 807 in FIG. The CRM system 30 stores the URL generated based on the call ID in the URL field of the response history table 3013 in association with the call ID. Note that the response history table 3013 may be configured to store only part of the second call data.

In step S609, the CRM storage control unit 1051 of the server 10 transmits to the CRM system 30 a request to store the third call data and the call data including the call ID in the call table 1014 of the call in association with the customer ID. The third call data is the data related to the call that can be obtained after the end of the call, excluding the data included in the first call data.
The CRM system 30 stores the end date and time of the call as second call data in the item of call end date and time of the response history table 3013 in association with the call ID. The CRM system 30 adds the speech recognition information about the call and the summary information about the call to the comment item of the response history table 3013 in association with the call ID. Specifically, like the voice recognition result and summary result of the comment 807 in FIG.
At this time, if the cooperation mode is the first mode, the CRM storage control unit 1051 of the server 10 has already generated a record in the response history table 3013 in step S604. Add so as not to overwrite the content. Specifically, when the cooperation mode is the first mode, a record of the call is newly created in the reception history table 3013, and another user may add the comment to the comment. Even in such a case, the CRM storage control unit 1051 of the server 10 adds the comment so as not to overwrite the comment of the record.
The CRM system 30 stores the URL generated based on the call ID in the URL field of the response history table 3013 in association with the call ID. Note that the response history table 3013 may be configured to store only part of the third call data.

<Modification>
The server 10 can be configured to store customer information itself by acquiring customer information from the customer table 3012 of the CRM system 30 and storing it in a database (not shown). In this case, the CRM storage control unit 1051 of the server 10 sends the call data to the customer to be called by sending a request including the customer ID, the name of the customer, the name of the customer organization, etc. to the CRM system 30 instead of the call ID. may be stored in association with . In this case as well, the data related to the call is stored in the CRM system 30 in association with the customer information of the call target.

In the present disclosure, the server 10 also provides services related to calls between users and customers, but the call services may be provided by an external service (not shown). At this time, the CRM storage control unit 1051 of the server 10 detects that a call between the user and the customer has started by receiving a request for starting a call between the user and the customer provided by the external service. may start the CRM storage process. Similarly, the termination of the call between the user and the customer may be detected upon receipt of a request for termination of the call between the user and the customer provided by the external service. In addition, based on voice data between the user and the customer, for example, the end of the call between the user and the customer may be detected when the voice data disappears.

<Call display processing>
Call display processing is processing for displaying call data stored in the CRM system 30 to the user. Details of the call display processing will be described below with reference to the flowchart of FIG. FIG. 24 is a diagram showing an example of a screen output by the CRM system 30 in call display processing.

<Overview of call display processing>
After finishing the call with the customer, the user operates the user terminal 20 to display the customer reception history stored in the CRM system 30 . When the user operates the user terminal 20, selects the customer service history for which the analysis data is to be displayed from the service history with the customer, and requests display of the analysis data, the CRM system 30 acquires the analysis data from the server 10, The customer information and analysis data are displayed on the display 2081 of the user terminal 20 .

<Details of call display processing>
In step S701, the user operates the user terminal 20 to open a predetermined website provided by the CRM system 30, and displays a reception history display screen.
Specifically, the user terminal 20 transmits a request for displaying a list of customer information to the CRM system 30 . CRM system 30 searches customer table 3012 and sends the record to user terminal 20 . The display 2081 of the user terminal 20 displays a list of the received customer information so that the user can select it.
The user selects a customer whose response history is to be displayed from the customer information displayed in a list on the display 2081 of the user terminal 20, and presses the "Send" button. to send. When the customer ID is received, the CRM system 30 searches the customer service history table 3013 and transmits to the user terminal 20 a customer service history record for the selected customer. The display 2081 of the user terminal 20 displays a list of received reception histories so that the user can select them.

In step S702, the user selects a record whose response history is to be displayed from the list of response histories displayed on the display 2081 of the user terminal 20, and presses the "Send" button. Send the response history ID of the response history. The CRM system 30 generates a display screen of the selected reception history information and transmits it to the user terminal 20 . The display 2081 of the user terminal 20 displays the received reception history information display screen. The display screen of the response history information includes the URL, customer information, user information, dialing date and time, call start date and time, call end date and time, voice recognition information stored in the comment by CRM storage processing, and voice recognition information stored in the response history table 3013. Summary information and so on are displayed.
For example, after the call between the user and the customer ends without performing the processes of steps S701 and S702, the user terminal 20 sets the selected response history as the response history related to the call, and sets the response history ID as You may transmit to the CRM system 30.

In step S703, the user operates the user terminal 20 and presses the "analysis result" button 801 displayed on the display screen of the response history information. The user terminal 20 transmits a request to the CRM system 30 to display the analysis result including the reception history ID related to the call. The CRM system 30 searches the response history table 3013 based on the received response history ID to identify the call ID. CRM system 30 sends a request to server 10 for analytical data, including the identified call ID. The server 10 searches the call table 1014 based on the received call ID and confirms the presence or absence of analysis data. If the analysis data does not exist, it waits in step S703.
If analysis data exists, the process proceeds to step S704.

In step S<b>704 , the server 10 searches the call table 1014 based on the received call ID, and transmits analysis data to the CRM system 30 . Based on the received analysis data, the CRM system 30 generates an analysis result screen 808 that visualizes the analysis data and transmits it to the user terminal 20 . Analysis result screen 808 includes voice analysis result 802, response evaluation 804, voice evaluation 805, and speech speed 806 shown in FIG. The analysis result screen 808 may include a comment 807 that is a speech recognition result in text format and a summary result stored in the comment item of the response history table 3013 . The user can reproduce the call voice by pressing the play/stop button 803 .
Customer information such as a customer table 3012 is stored in the CRM system 30, and the user selectively switches between the customer information stored in the CRM system 30 and an analysis result screen that visualizes the analysis data. By displaying them side by side, it is possible to grasp the customer's response history in more detail in a short time. In other words, by operating the CRM system 30, the user can check the visualization results of the analysis data in addition to managing the customer information.

<Modification>
In the present disclosure, the voice server (PBX) 40 and the customer terminal 50 have been described as being connected via the telephone network T, but this is not a particular limitation. In other words, the voice server (PBX) 40 and the customer terminal 50 may be connected by any means of communication including, for example, the Internet.

In this case, in the calling process, a call may be made to the customer terminal 50 based on arbitrary customer identification information for identifying the customer, URL, etc., instead of the customer's telephone number. Similarly, in the incoming call process, the incoming call request may contain information such as arbitrary customer identification information and URL for identifying the customer, and the user terminal 20 receiving the incoming call is specified based on the information such as customer identification information and URL. However, the configuration may be such that an incoming call (power reception) is performed.

<Computer hardware configuration>
FIG. 25 is a block diagram showing the basic hardware configuration of the computer 90. As shown in FIG. The computer 90 includes at least a processor 901, a main storage device 902, an auxiliary storage device 903, and a communication IF 991 (Interface). These are electrically connected to each other by a communication bus 921 .

The processor 901 is hardware for executing an instruction set described in a program. The processor 901 is composed of an arithmetic unit, registers, peripheral circuits, and the like.

The main storage device 902 is for temporarily storing programs and data processed by the programs. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

Auxiliary storage device 903 is a storage device for storing data and programs. For example, flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, semiconductor memory, and the like.

The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard.
The network is composed of various mobile communication systems constructed by the Internet, LAN, wireless base stations, and the like. For example, networks include 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), wireless networks (for example, Wi-Fi (registered trademark)) that can be connected to the Internet through predetermined access points, and the like. When connecting wirelessly, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), and the like. In the case of wired connection, the network includes direct connection using a USB (Universal Serial Bus) cable or the like.

It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration to a plurality of computers 90 and connecting them to each other via a network. Thus, the computer 90 is a concept that includes not only the computer 90 housed in a single housing or case, but also a virtualized computer system.

<Basic Functional Configuration of Computer 90>
A functional configuration of the computer realized by the basic hardware configuration of the computer 90 (FIG. 25) will be described. The computer includes at least functional units of a control section, a storage section, and a communication section.

Note that the functional units included in the computer 90 can also be implemented by distributing all or part of each functional unit to a plurality of computers 90 interconnected via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

The control unit is implemented by the processor 901 reading out various programs stored in the auxiliary storage device 903, developing them in the main storage device 902, and executing processing according to the programs. The control unit can implement functional units that perform various information processing according to the type of program. Thereby, the computer is implemented as an information processing device that performs information processing.

A storage unit is realized by the main storage device 902 and the auxiliary storage device 903 . The storage unit stores data, various programs, and various databases. Also, the processor 901 can secure a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 according to a program. In addition, the control unit can cause the processor 901 to execute addition, update, and deletion processing of data stored in the storage unit according to various programs.

A database refers to a relational database, and is for managing tabular tables structurally defined by rows and columns, and data sets called masters in association with each other. In a database, a table is called a table, a master is called a column, and a row is called a record. In a relational database, relationships between tables and masters can be set and associated.
Normally, each table and each master has a primary key column for uniquely identifying a record, but setting a primary key to a column is not essential. The control unit can cause the processor 901 to add, delete, and update records in specific tables and masters stored in the storage unit according to various programs.

A communication unit is implemented by the communication IF 991 . The communication unit implements a function of communicating with another computer 90 via a network. The communication section can receive information transmitted from another computer 90 and input it to the control section. The control unit can cause the processor 901 to execute information processing on the received information according to various programs. Also, the communication section can transmit information output from the control section to another computer 90 .

<Appendix>
The items described in the above embodiments will be added below.

(Appendix 1)
A program for causing a computer to manage data related to a call conducted between a user and a customer, the program comprising a processor and a storage unit, wherein the program causes the processor to receive voice data related to the call; a voice extraction step (S104, S204) for extracting a plurality of segmental voice data for each utterance segment from the voice data; a step (S104, S204); and a summary extraction step (S107, S107, S107, S107, S107, S204) of extracting, from the plurality of pieces of text information extracted in the text extraction step, one or a plurality of pieces of text information having different speaker information and preceding or succeeding in speech time. S207) and a program that executes.
As a result, it is possible to generate summarized information focusing on the call response between the user and the customer.
By confirming the summary information, the user can accurately grasp the call handling between the user and the customer in a short period of time.

(Appendix 2)
The program stores, in the processor, a text information group of text information before and after a change in speaker information when the plurality of text information extracted in the text extraction step is arranged in order of utterance time. and a calculating step (S106) of calculating the importance of a plurality of pieces of text information and one or more groups of text information bound in the binding step, and summarizing The program according to appendix 1, wherein the extracting step is a step of extracting the text information or the text information group as summary information based on the degree of importance.
As a result, in a computer that manages data related to a call made between a user and a customer, a group of text information of utterances and responses that are bound based on the call made between the user and the customer. It is possible to generate summary information that focuses on the call response between the customer and the customer.
By confirming the summary information, the user can accurately grasp the call handling between the user and the customer in a short period of time.

(Appendix 3)
In the calculating step, for each of the one or more text information groups bound in the binding step, the importance is calculated as one sentence formed by combining the text information contained in the text information group (S106). A program according to Appendix 2.
This makes it possible to calculate the importance of each text information group of bound utterances and responses. Summary information is generated that focuses on the call handling between the user and the customer.
By confirming the summary information, the user can accurately grasp the call handling between the user and the customer in a short period of time.

(Appendix 4)
In the calculation step, the importance is calculated for each of the plurality of text information contained in the text information group bound in the binding step, and the values obtained by performing statistical processing on the calculated plurality of importance are added to the text. The program according to appendix 2, which is a step (S106) of calculating the importance of the information group.
This makes it possible to calculate the importance of each text information group of bound utterances and responses. Summary information is generated that focuses on the call handling between the user and the customer.
By confirming the summary information, the user can accurately grasp the call handling between the user and the customer in a short period of time.

(Appendix 5)
4. The program according to appendix 4, wherein the calculating step is a step (S106) of setting the highest degree of importance among the plurality of calculated degrees of importance as the degree of importance of the text information group.
As a result, the importance of the text information group is calculated based on the importance of the text information with the highest importance among the text information included in the bound utterance and response text information group. Summary information can be generated that includes textual information with a high degree of importance.
By checking the summary information, the user can check without overlooking the text information of high importance in the call handling between the user and the customer.

(Appendix 6)
4. The program according to appendix 4, wherein the calculating step is a step (S106) of setting the sum or average of the calculated plurality of degrees of importance as the degree of importance of the text information group.
As a result, the importance of the text information group is calculated based on the sum or average of the importance of the text information included in the bound utterance and response text information group. It is possible to generate summary information that includes more text information with a high degree of importance.
By confirming the summary information, the user can confirm without overlooking the text information including a large amount of text information with high importance among the call responses between the user and the customer.

(Appendix 7)
The program causes the processor to perform a calculation step (S205) of calculating importance for a plurality of pieces of text information extracted in the text extraction step, and the summary extraction step is based on the importance, and the text extraction step a step of extracting one or more pieces of text information from the plurality of pieces of text information extracted in step of extracting text information that differs from the extracted one or more pieces of text information and speaker information and that precedes or follows in utterance time as summary information. The program according to Supplementary Note 1, comprising a step (S206).
As a result, in the computer that manages the data related to the call, we focused on the call response between the user and the customer, which is the text information group of the utterance and response that was bound based on the call between the user and the customer. Summary information can be generated.
By confirming the summary information, the user can accurately grasp the call handling between the user and the customer in a short period of time.

(Appendix 8)
The program executes, on the processor, a display step (S207) of displaying, on the processor, a summary document in which the text information included in the summary information is arranged in order of utterance time, extracted in the summary extraction step. or the program described.
Accordingly, by confirming the summary document, the user can accurately grasp the call handling performed between the user and the customer in a short time.

(Appendix 9)
The summary document is a summary document displayed in balloons for each piece of text information, and the balloons are displayed in different colors for each speaker of the text information. 9. The program according to appendix 8, wherein at least one of color brightness, hue, saturation, and density is changed for display.
Thereby, the user can visually grasp the importance of the text information contained in the balloon by confirming the lightness, hue, saturation, and density of the color of the balloon containing the text information.

(Appendix 10)
10. The program according to appendix 8 or 9, wherein the program causes the processor to perform a storage step (S107, S207) of storing the summary document in an external CRM system.
This allows the user to systematically manage summaries of calls made between the user and the customer together with the customer information in the CRM system.

(Appendix 11)
An information processing system comprising a processor and a storage unit for managing data related to a call conducted between a user and a customer, wherein the processor receives voice data related to the call; , a speech extraction step (S104, S204) for extracting a plurality of segmental speech data for each utterance segment, and a text extraction step (S104 , S204), and a summary extraction step (S107, S207) for extracting, as summary information, one or a plurality of pieces of text information having different speaker information and preceding or following utterance times from the plurality of pieces of text information extracted in the text extraction step (S107, S207). , an information processing system that runs
As a result, it is possible to generate summary information focusing on the call response between the user and the customer.
By confirming the summary information, the user can accurately grasp the call handling between the user and the customer in a short period of time.

(Appendix 12)
An information processing method, comprising a processor and a storage unit, for causing a computer to manage data related to a call conducted between a user and a customer, the method comprising: a receiving step of receiving voice data related to the call into the processor; A voice extraction step (S104, S204) for extracting a plurality of segmental voice data for each utterance segment from the voice data, and a text extraction step for performing text recognition on each of the plurality of segmental voice data and extracting a plurality of pieces of text information. (S104, S204), and a summary extraction step (S107, S207) for extracting, as summary information, one or more pieces of text information having different speaker information and preceding or succeeding in speech time from the plurality of text information extracted in the text extraction step (S107, S207). ) and the information processing method that performs the
As a result, it is possible to generate summarized information focusing on the call response between the user and the customer.
By confirming the summary information, the user can accurately grasp the call handling between the user and the customer in a short period of time.

1 information processing system, 10 server, 101 storage unit, 103 control unit, 20A, 20B, 20C user terminal, 201 storage unit, 204 control unit, 30 CRM system, 301 storage unit, 304 control unit, 50A, 50B, 50C customer terminal, 501 storage unit, 504 control unit

Claims (12)

A program, comprising a processor and a storage unit, for causing a computer to manage data related to a call made between a user and a customer,
The program causes the processor to:
a receiving step of receiving voice data related to the call;
a voice extracting step of extracting a plurality of segmental voice data for each utterance segment from the voice data;
a text extraction step of performing text recognition on each of the plurality of segmental audio data to extract a plurality of pieces of text information;
a summary extraction step of extracting, as summary information, one or a plurality of pieces of text information having different speaker information and preceding or following an utterance time from the plurality of pieces of text information extracted in the text extraction step;
A program that runs The program causes the processor to:
For the plurality of text information extracted in the text extraction step, when the plurality of text information are arranged in the order of utterance time, the text information before and after the speaker information changes are bound as a text information group. a binding step;
a calculation step of calculating the importance of the plurality of text information and the one or more text information groups bound in the binding step;
and
The summary extraction step is a step of extracting the text information or the text information group as the summary information based on the importance.
A program according to claim 1. In the calculating step, for each of the one or more text information groups bound in the binding step, the importance is calculated as one sentence formed by combining the text information included in the text information group. be,
3. A program according to claim 2. The calculating step calculates the importance of each of a plurality of pieces of text information included in the text information group bound in the binding step, and performs statistical processing on the calculated plurality of pieces of importance. is calculated as the importance of the text information group,
3. A program according to claim 2. The calculating step is a step of setting the highest degree of importance among the plurality of calculated degrees of importance as the degree of importance of the text information group,
5. The program according to claim 4. The calculating step is a step of setting the sum or average of the calculated plurality of degrees of importance as the degree of importance of the text information group,
5. The program according to claim 4. The program causes the processor to:
a calculation step of calculating the degree of importance of the plurality of pieces of text information extracted in the text extraction step;
and
The abstract extraction step includes:
extracting one or more pieces of text information from the plurality of pieces of text information extracted in the text extraction step, based on the degree of importance;
a step of extracting, as the summary information, the text information that is different from the extracted one or the plurality of text information and the speaker information, and that has an utterance time before and after the text information;
including,
A program according to claim 1. The program causes the processor to:
a display step of displaying a summary document in which the text information included in the summary information extracted in the summary extraction step is arranged in order of utterance time;
8. The program according to any one of claims 1 to 7, which executes the summary document is a summary document displayed in a balloon for each piece of text information;
the balloon is displayed in a different color for each speaker of the text information;
The color of the balloon is displayed by changing at least one of color brightness, hue, saturation, and density based on the text information or the importance of the text information.
9. A program according to claim 8. The program causes the processor to:
a storage step of storing the summary document in an external CRM system;
to run
10. A program according to claim 8 or 9. An information processing system, comprising a processor and a storage unit, for managing data associated with a call conducted between a user and a customer, comprising:
to the processor;
a receiving step of receiving voice data related to the call;
a voice extracting step of extracting a plurality of segmental voice data for each utterance segment from the voice data;
a text extraction step of performing text recognition on each of the plurality of segmental audio data to extract a plurality of pieces of text information;
a summary extraction step of extracting, as summary information, one or a plurality of pieces of text information having different speaker information and preceding or following an utterance time from the plurality of pieces of text information extracted in the text extraction step;
An information processing system that executes An information processing method, comprising a processor and a storage unit, for causing a computer to manage data related to a call conducted between a user and a customer, comprising:
to the processor;
a receiving step of receiving voice data related to the call;
a voice extracting step of extracting a plurality of segmental voice data for each utterance segment from the voice data;
a text extraction step of performing text recognition on each of the plurality of segmental speech data to extract a plurality of pieces of text information;
a summary extraction step of extracting, as summary information, one or a plurality of pieces of text information having different speaker information and preceding or following an utterance time from the plurality of pieces of text information extracted in the text extraction step;
Information processing method that performs

Images