WO2023119992A1 - 情報処理装置、および情報処理方法、並びにプログラム - Google Patents

情報処理装置、および情報処理方法、並びにプログラム Download PDF

Info

Publication number
WO2023119992A1
WO2023119992A1 PCT/JP2022/042968 JP2022042968W WO2023119992A1 WO 2023119992 A1 WO2023119992 A1 WO 2023119992A1 JP 2022042968 W JP2022042968 W JP 2022042968W WO 2023119992 A1 WO2023119992 A1 WO 2023119992A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
user state
meeting
estimator
information processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/042968
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
裕麻 平井
大祐 稲石
りんな 金尾
タンガドウレイ プレム アジャント フランシス
弦樹 岡田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to EP22910710.7A priority Critical patent/EP4455972A4/en
Priority to JP2023569183A priority patent/JPWO2023119992A1/ja
Priority to US18/720,089 priority patent/US20250054291A1/en
Publication of WO2023119992A1 publication Critical patent/WO2023119992A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/01—Customer relationship services
    • G06Q30/015—Providing customer assistance, e.g. assisting a customer within a business location or via helpdesk
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/87—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0201—Market modelling; Market analysis; Collecting market data
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776—Validation; Performance evaluation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Definitions

  • the present disclosure relates to an information processing device, an information processing method, and a program. More specifically, the process of generating a user state estimator (learning model) that estimates the emotions of participants in remote meetings via a communication network, such as remote business negotiations, based on images, voices, etc., and the generated user state estimation
  • the present invention relates to an information processing device, an information processing method, and a program that execute user state estimation processing using a device (learning model).
  • the communication terminal of the sales side user who intends to sell products and the communication terminal of the customer who is the customer are connected via a communication network such as the Internet, and images and voices are transmitted between the terminals. Send and receive to conduct business negotiations.
  • Patent Literature 1 International Publication No. WO2019/082687 is a conventional technique disclosing human emotion estimation processing.
  • This patent document 1 discloses a configuration for estimating a person's emotion by analyzing a person's electroencephalogram and calculating a score based on the analysis result.
  • the present disclosure has been made in view of the above problems, for example, and is a user state estimator (learning model) that estimates the emotions and states of users participating in a remote meeting via a communication network based on images, voices, etc. is efficiently generated, and furthermore, the generated user state estimator (learning model) is used to perform highly accurate user state estimation processing, an information processing method, and a program. and
  • the generated user state estimator (learning model) is used to perform highly accurate user state estimation processing, an information processing method, and a program.
  • a first aspect of the present disclosure includes: a user state estimator generation unit that generates a user state estimator by executing machine learning processing using user state information input to a user terminal by a user who participates in a meeting via a communication network;
  • the user state estimator generator An information processing apparatus that generates a user state estimator that estimates a user state based on at least one of image data and voice data of users participating in a meeting.
  • a second aspect of the present disclosure is a user state estimator for estimating a user state based on at least one of image data and voice data of a user participating in a meeting via a communication network;
  • the information processing apparatus includes a user state estimation result output unit that outputs identification information indicating the user state estimated by the user state estimator to user terminals of users participating in the meeting.
  • a third aspect of the present disclosure is An information processing method executed in an information processing device, A user state estimator generator, executing a user state estimator generation process for generating a user state estimator by executing machine learning processing using user state information input to a user terminal by a user who participates in a meeting via a communication network;
  • the user state estimator generator An information processing method for generating a user state estimator for estimating a user state based on at least one of image data and voice data of users participating in a meeting.
  • a fourth aspect of the present disclosure is An information processing method executed in an information processing device,
  • the user state estimator is executing a user state estimation process for estimating a user state based on at least one of image data and voice data of a user participating in a meeting via a communication network;
  • the user state estimation result output unit The information processing method performs a user state estimation result output process of outputting identification information indicating the user state estimated by the user state estimator to user terminals of users participating in the meeting.
  • a fifth aspect of the present disclosure is A program for executing information processing in an information processing device,
  • the user state estimator generator executing a user state estimator generation process for generating a user state estimator by executing machine learning processing using user state information input to a user terminal by a user who participates in a meeting via a communication network;
  • the user state estimator is a program that is a user state estimator that estimates a user state based on at least one of image data and voice data of users participating in a meeting.
  • a sixth aspect of the present disclosure is A program for executing information processing in an information processing device, to the user state estimator, executing user state estimation processing for estimating a user state based on at least one of image data and voice data of a user participating in a meeting via a communication network;
  • the program executes user state estimation result output processing for outputting identification information indicating the user state estimated by the user state estimator to user terminals of users participating in the meeting.
  • the program of the present disclosure is, for example, a program that can be provided in a computer-readable format to an information processing device or computer system capable of executing various program codes via a storage medium or communication medium.
  • processing according to the program is realized on the information processing device or computer system.
  • a system is a logical collective configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same housing.
  • a configuration in which a user state estimator for estimating a user's interest level, understanding level, etc. is generated and used by learning processing using user state information input by a meeting participant user.
  • a user who participates in a meeting via a communication network executes machine learning processing using user state information input to the user terminal, and based on at least one of the image or voice of the meeting participant user
  • a user state estimator is generated that estimates the user state, eg, the user's interest level, comprehension level, and fatigue level.
  • the generated user state estimator is used to estimate the user state based on the images and voices of the users participating in the meeting, and the identification information and icon indicating the estimated user state are output to the user terminal.
  • a configuration is realized in which a user state estimator for estimating a user's degree of interest, understanding, etc. is generated and used by learning processing using user state information input by a user participating in a meeting. Note that the effects described in this specification are merely examples and are not limited, and additional effects may be provided.
  • FIG. 1 is a diagram illustrating the configuration of an information processing system of the present disclosure and an outline of processing to be executed;
  • FIG. FIG. 10 is a diagram illustrating an overview of user state estimator (learning model) generation processing;
  • FIG. 10 is a diagram illustrating a usage example of a user state estimator (learning model);
  • FIG. 4 is a diagram illustrating a configuration example in which processing of an information processing device is executed by a sales-side user terminal;
  • FIG. 7 is a diagram showing a flowchart for explaining a sequence of user state estimator (learning model) generation processing executed by the information processing apparatus;
  • FIG. 11 is a diagram illustrating a specific example of meeting condition (MTG tag) input processing;
  • FIG. 10 is a diagram illustrating an example of processing for inputting a user state score (evaluation value) indicating a user state such as one's emotions during a meeting;
  • FIG. 4 is a diagram for explaining an example of data showing a part of data (meeting log) used for machine learning processing in an information processing device;
  • FIG. 4 is a diagram illustrating an example of time-series data that can be generated by an information processing apparatus based on user state scores (evaluation values) acquired from customer-side user terminals; It is a figure explaining the collection structural example of the data utilized for the machine-learning process which an information processing apparatus performs.
  • FIG. 10 is a diagram illustrating an example of processing for inputting a user state score (evaluation value) indicating a user state such as one's emotions during a meeting;
  • FIG. 10 is a diagram illustrating a setting example of user state scores
  • FIG. 10 is a diagram illustrating a configuration example in which images, voices, and user state scores are acquired only from a customer-side user terminal, and these are applied to learning processing; It is a figure explaining the structure and processing of the machine-learning process which an information processing apparatus performs.
  • FIG. 10 is a diagram illustrating an example of learning processing for generating an interest level estimator (interest level estimation learning model) by an interest level estimator generation unit (interest level estimation learning model generation unit);
  • FIG. 10 is a diagram illustrating an example of learning processing for generating an interest level estimator (interest level estimation learning model) by an interest level estimator generation unit (interest level estimation learning model generation unit);
  • FIG. 10 is a diagram illustrating an example of learning processing for generating an interest level estimator (interest level estimation learning model) by an interest level estimator generation unit (interest level estimation learning model generation unit);
  • FIG. 10 is a diagram illustrating an example of learning processing for generating an interest level estimator (interest level estimation learning model) by an interest level estimator generation unit (interest level estimation
  • FIG. 10 is a diagram illustrating an example of learning processing for generating an interest level estimator (interest level estimation learning model) by an interest level estimator generation unit (interest level estimation learning model generation unit);
  • FIG. 10 is a diagram illustrating a processing example of outputting an estimated value of interest/concern/favorability score of a user on the customer side using an interest level estimator (interest level estimation learning model);
  • FIG. 10 is a diagram for explaining the configuration and processing of an interest level estimator generation unit (interest level estimation learning model generation unit) that generates a plurality of meeting condition corresponding interest level estimators (interest level estimation learning models);
  • FIG. 10 is a diagram illustrating an example of combination types of meeting conditions;
  • FIG. 10 is a diagram illustrating an example of evaluation processing of a meeting-condition corresponding interest level estimator (learning model);
  • FIG. 3 is a diagram illustrating a configuration example of data stored in each database;
  • FIG. 10 is a diagram illustrating a specific example of estimator (learning model) evaluation processing executed by an estimator (learning model) evaluation unit;
  • FIG. 4 is a diagram illustrating a specific example of meeting conditions used as test data;
  • FIG. 4 is a diagram illustrating a specific example of meeting conditions used as test data;
  • FIG. 4 is a diagram illustrating a specific example of meeting conditions used as test data;
  • FIG. 10 is a diagram illustrating an example of evaluation processing of a meeting-condition corresponding interest level estimator (learning model);
  • FIG. 3 is a diagram illustrating a configuration example of data stored in each database;
  • FIG. 10 is a diagram illustrating a specific example of estimator (learning model) evaluation processing executed by an estimator (learning model) evaluation unit;
  • FIG. 4 is
  • FIG. 10 is a diagram illustrating details of user state estimation processing using a user state estimator (user state estimation learning model); It is a figure explaining the specific example of the interest degree estimation process of the customer side user using the interest degree estimator (interest degree estimation learning model) which the information processing apparatus produced
  • FIG. 10 is a diagram illustrating an example of a correspondence relationship between estimated score values and display icons;
  • FIG. 10 is a diagram illustrating an example of a correspondence relationship between estimated score values and display icons;
  • FIG. 10 is a diagram illustrating an example of a correspondence relationship between estimated score values and display icons;
  • It is a figure explaining the structural example of an information processing apparatus. It is a figure explaining the example of a structure of an information processing apparatus and a user terminal. It is a figure explaining the hardware configuration example of an information processing apparatus and a user terminal.
  • the information processing apparatus of the present disclosure executes the following two processes.
  • Processing 2) A process of estimating user emotion using the generated user state estimator (learning model).
  • FIG. 1 is a diagram showing an example of a remote meeting via a communication network.
  • FIG. 1 shows a customer side user 11 who is a customer who wishes to purchase a product, and a sales side user 12 who is a product provider.
  • a customer-side user terminal 21 such as a smartphone and a sales-side user terminal 22 such as a PC are connected via a communication network, and voices and images are mutually transmitted and received between these communication terminals to carry out business negotiations. .
  • a customer side user 11 who is a customer is a person who wishes to purchase an apartment
  • a sales side user 12 is an apartment seller.
  • the sales-side user 12 listens to the customer-side user 11's wishes and explains by selecting an apartment that meets the customer-side user's 11 wishes.
  • the sales side user 12 can observe the customer side user 11 through the screen, but the information obtained from the image is less than the information obtained when actually meeting the customer side user 11 .
  • the sales side user 12 determines whether the customer side user 11 understands the explanation of the sales side user 12, whether the customer side user 11 is interested in the explanation, whether the customer side user 11 is angry, and so on. It becomes difficult to accurately grasp emotions.
  • the information processing apparatus of the present disclosure generates a user state estimator (learning model) for solving this problem, and uses the generated user state estimator (learning model) to evaluate the user's emotion, understanding level, etc. Estimate user state.
  • the information processing apparatus 100 which is a cloud-side device, performs processing for generating a user state estimator (learning model), and uses the generated user state estimator (learning model) to estimate the user state. do.
  • (process 1) described above that is, (Processing 1) Processing for generating a user state estimator (learning model) for estimating user states such as emotions of users participating in a remote meeting via a communication network based on images, voices, and the like.
  • (processing 1) user state estimator (learning model) generation processing An outline of this “(processing 1) user state estimator (learning model) generation processing” will be described.
  • FIG. 2 shows a customer side user 11 who is a customer who wishes to purchase a product, and a sales side user 12 who is a product provider.
  • a customer-side user terminal 21 such as a smartphone and a sales-side user terminal 22 such as a PC are connected via a communication network, and voices and images are mutually transmitted and received between these communication terminals to carry out business negotiations. .
  • the customer side user 11 is not an actual customer, but a person who performs the customer's role. For example, an employee of the same company as the sales side user 12 (employee of a condominium sales company) or a part-time worker plays the role of the customer.
  • a sales side user 12 conducts a simulated business negotiation with a customer side user 11 acting as a customer via a network.
  • a customer side user 11 playing the role of a customer inputs user states such as his/her feelings to a customer side user terminal 21 at any time during execution of a simulated business negotiation.
  • Input items are, for example, the following three user states as shown in FIG. (User state 1) Interest, concern, good feeling (User state 2) Understanding, consent, satisfaction (User state 3) Fatigue, stress
  • Interest, Concern, and Favorability are user states indicating whether or not the customer side user 11 has an interest in, interest in, or a favorable impression of the sales side user 12's explanation and conversation.
  • a customer-side user 11 playing the role of a customer judges his/her own interest, concern, and favorability level during a meeting (business negotiation) with a sales-side user 12, and inputs a score (evaluation value) based on the judgment at any time. .
  • the score is on a scale of 1 to 5, and the score (evaluation value) is higher as the interest, concern, and favorability are higher.
  • the customer side user 11 playing the role of the customer during the period of the meeting (business negotiation) with the sales side user 12, at any timing when he feels that the level of "interest, concern, good impression” has changed, he always gives a score (evaluation value). input.
  • (User State 2) Understanding, Consent, and Satisfaction are user states as to whether the customer side user 11 understands, consents, and is satisfied with the explanation of the sales side user 12 .
  • a customer-side user 11 acting as a customer inputs a score (evaluation value) at any time during a meeting (negotiation) with a sales-side user 12 .
  • Fatigue and stress are user states indicating whether or not the customer side user 11 felt fatigue or stress due to the sales side user 12's explanation or conversation.
  • a customer-side user 11 acting as a customer inputs a score (evaluation value) at any time during a meeting (negotiation) with a sales-side user 12 .
  • the customer side user 11 who plays the role of the customer inputs the score (evaluation value) at any time during the meeting (business negotiation) period with the sales side user 12 at any time when the level of "fatigue and stress” changes. .
  • the information processing device 100 further inputs images and voices of each user during the execution of the meeting via the customer-side user terminal 21 and the sales-side user terminal 22 .
  • the information processing apparatus 100 inputs the following data during the execution period of the meeting.
  • the information processing apparatus 100 inputs the above data (A) to (C) during the execution period of the meeting, executes learning processing using this input data, and generates a user state estimator (learning model).
  • the details of this user state estimator (learning model) generation process will be described later.
  • the user state estimator (learning model) generated by the information processing apparatus 100 is used in meetings such as business negotiations with actual customers. That is, it is used in the above-described (process 2), that is, "(process 2) process of estimating user emotion using the generated user state estimator (learning model).”
  • FIG. 3 also shows a customer side user 11 who is a customer who wishes to purchase a product, and a sales side user 12 who is a product provider.
  • a customer-side user terminal 21 such as a smartphone and a sales-side user terminal 22 such as a PC are connected via a communication network. can proceed.
  • the customer side user 11 is an actual customer, and actual negotiations are held instead of simulated negotiations.
  • the information processing device 100 inputs the following data via the communication network during the meeting period between the customer side user 11 and the sales side user 21 .
  • the information processing apparatus 100 inputs these data (A) and (B) to the user state estimator (learning model) generated in the previous learning process.
  • a user state estimator estimates a user state based on input image and voice data. That is, the user state is estimated based on at least one of image data and voice data of users participating in a meeting via a communication network.
  • the user states estimated by the user state estimator are three user states of the customer side user 11, that is, (User state 1) Interest, Concern, Likeability (User state 2) Understanding, Acceptance, Satisfaction (User state 3) Fatigue, Stress These are estimated scores of these user states.
  • the information processing apparatus 100 transmits to the sales side user terminal 22 an identification icon corresponding to the scores of the above (user states 1 to 3) indicating the user state of the customer side user 11 estimated by the user state estimator (learning model). displayed.
  • the sales-side user 12 can grasp the state of the customer side, and changes the method and content of the explanation according to the grasped result. It is possible to take the most appropriate measures such as
  • the information processing device on the cloud is a device that executes a user state estimator (learning model) generating process and a user state estimating process that uses the generated user state estimator (learning model).
  • a user state estimator learning model
  • a user state estimating process that uses the generated user state estimator (learning model).
  • An example of 100 has been described.
  • These processes are not limited to devices on the cloud, and may be configured to be executed using, for example, the sales side user terminal 22 .
  • the sales-side user terminal 22 executes user state estimator (learning model) generation processing and user state estimation processing using the generated user state estimator (learning model). That is, the processing of the information processing apparatus 100 described with reference to FIGS. 1 to 3 may be executed by the sales-side user terminal 22.
  • FIG. 4 the sales-side user terminal 22 executes user state estimator (learning model) generation processing and user state estimation processing using the generated user state estimator (learning model). That is, the processing of the information processing apparatus 100 described with reference to FIGS. 1 to 3 may be executed by the sales-side user terminal 22.
  • the information processing apparatus 100 executes processing for generating a user state estimator (learning model).
  • the information processing apparatus 100 stores the following data during the execution period of the meeting, that is, (A) Image and voice of the customer side user 11 from the customer side user terminal 21; (B) Image and voice of the sales user 12 from the sales user terminal 22; (C) From the customer-side user terminal 21, input the above data (A) to (C) of the score (evaluation value) data string of the above (user status 1 to 3), and perform the learning process using these input data. Run to generate a user state estimator (learning model).
  • a sequence of user state estimator (learning model) generation processing executed by the information processing apparatus 100 will be described with reference to FIG.
  • FIG. 5 is a flowchart for explaining the sequence of user state estimator (learning model) generation processing executed by the information processing apparatus 100 .
  • the customer side user 11 is not an actual customer, but a person who performs the role of the customer. process using .
  • a sales side user 12 holds a meeting such as a simulated business negotiation with a customer side user 11 acting as a customer via a network.
  • processing of each step of the flowchart shown in FIG. 5 will be described in order.
  • Step S101 First, in step S101, a meeting condition (MTG tag) is input.
  • FIG. 6 shows an example of an initial setting screen displayed on the display units of the customer-side user terminal 12 and the sales-side user terminal 22 when executing the process of generating the user state estimator (learning model).
  • the customer side user 11 playing the role of the customer and the sales side user 12 enter meeting conditions (MTG tags) in the Comment column of the initial setting screen shown in FIG. Enter each meeting condition (MTG tag) below.
  • TMG tags meeting conditions
  • FIG. 6 shows an example of an initial setting screen displayed on the display units of the customer-side user terminal 12 and the sales-side user terminal 22 when executing the process of generating the user state estimator (learning model).
  • (Tag a) Meeting (MTG) Genre includes meeting genres such as “business talks”, “external meetings”, “internal meetings”, “briefings”, “interviews”, and “classes”. Enter data that indicates the type of
  • tags may be set in three categories, large, medium, and small, or in two categories, large and small. These tags are preferably set according to predetermined tag setting rules.
  • tags c and d For (tag c) fatigue (customer) and (tag d) busyness (customer), the fatigue level and busyness level of the customer side user 11 at the start of the meeting are input.
  • These tags c and d may also be classified into three categories, large, medium, and small, or two categories, large and small. Preferably, these tags are set according to predetermined tag setting rules.
  • the tags a to b are entered by the sales side user 12, and the tags c to d are entered by the customer side user 11 acting as the customer.
  • the sales user 12 may listen to the status of the customer user 11 who plays the role of the customer, and the sales user 12 may enter all the tags a to d.
  • meeting conditions MMG tags
  • the configuration may be such that only one of the conditions is used, or the configuration is such that meeting conditions other than the above conditions are set.
  • the input meeting conditions are transmitted from the customer-side user terminal 12 or the sales-side user terminal 22 to the information processing device 100 and stored in the storage section of the information processing device 100 .
  • Step S102 Next, in step S102 of the flow shown in FIG. 5, the meeting is started.
  • the customer side user 11 playing the role of the customer and the sales side user 12 start a meeting such as a mock business negotiation via the network.
  • Step S103 After the meeting starts, the customer-side user 11 successively inputs a user state score (evaluation value) indicating the user state such as his or her emotions in step S103.
  • a user state score evaluation value
  • FIG. 7 shows an example of a display image of the customer-side user terminal 21 during execution of the meeting.
  • the face image of the sales side user 21 who is the meeting partner is displayed on the customer side user terminal 21 .
  • the user terminal 21 on the customer side further displays a UI for inputting the user condition score (evaluation value) as shown in the figure.
  • the customer side user 11 playing the role of the customer can input his or her emotional state during the execution of the meeting through this UI at any time.
  • the input items are the following three user states described above with reference to FIG. (User state 1) Interest, concern, good feeling (User state 2) Understanding, consent, satisfaction (User state 3) Fatigue, stress
  • (User State 1) Interest, Concern, and Favorability are user states indicating whether or not the customer side user 11 has an interest in, interest in, or a favorable impression of the sales side user 12's explanation and conversation.
  • (User State 2) Understanding, Consent, and Satisfaction are user states as to whether the customer side user 11 understands, consents, and is satisfied with the explanation of the sales side user 12 .
  • the customer-side user 11 acting as a customer inputs the scores (evaluation values) of these three types of user states at any time during a meeting (business negotiation) with the sales-side user 12.
  • the scores (evaluation values) are on a scale of 1 to 5.
  • the customer-side user 11 acting as a customer executes the following score (evaluation value) input processing.
  • the score (evaluation value) of interest, concern, and favorable impression is higher (closer to 5) as the customer-side user 11's interest, interest, and favorable impression level to the explanation and conversation of the sales-side user 12 is higher.
  • a score (evaluation value) that is lower (closer to 1) is entered as the level of interest, interest, and favorability is lower.
  • the score (evaluation value) of understanding, consent, and satisfaction is higher (closer to 5) as the level of understanding, consent, and satisfaction of the customer side user 11 with respect to the explanation and conversation of the sales side user 12 is higher.
  • a score (evaluation value) that is lower (closer to 1) is entered as the level of understanding, acceptance, and satisfaction is lower.
  • Fatigue and stress score (evaluation value): Input a higher score (evaluation value) (closer to 5) as the fatigue and stress level of the customer side user 11 to the explanation and conversation of the sales side user 12 is higher. and input a lower (closer to 1) score (evaluation value) as the fatigue and stress levels are lower.
  • the information processing device 100 further inputs images and voices of each user during the execution of the meeting via the customer-side user terminal 21 and the sales-side user terminal 22 .
  • Step S104 Next, in step S104 of the flow shown in FIG. 5, the meeting ends.
  • Step S105 When the meeting ends, in step S105, the information processing apparatus 100 executes machine learning processing using the image and voice of each user during the meeting and the user state score (evaluation value) input during the meeting. .
  • a user state estimator (learning model) is generated as a result of this machine learning process. For example, a plurality of user state estimators (learning models) corresponding to meeting conditions are generated.
  • Step S106 the information processing apparatus generates a user state estimator (learning model) that performs more accurate estimation processing from the plurality of user state estimators (learning models) corresponding to the meeting conditions generated in step S105. Execute the process of selecting .
  • step S106 For example, a process of selecting a user state estimator (learning model) capable of performing highly accurate estimation processes corresponding to various meeting conditions is executed. Details of the processing of step S106 will also be described later.
  • the processing described below is machine learning processing for generating a user state estimator (learning model) executed by the information processing apparatus 100 in (step S105) of the flowchart described with reference to FIG. Details of the machine learning process for generating the user state estimator (learning model) executed by the information processing apparatus 100 in (step S105) will be described with reference to FIG. 9 and the subsequent figures.
  • the information processing apparatus 100 inputs the following data from these two communication terminals.
  • the scores (evaluation values) of (user states 1 to 3) in (C) above are scores (evaluation values) for each of the following three user states.
  • the information processing device 100 further inputs images and voices of each user during the execution of the meeting via the customer-side user terminal 21 and the sales-side user terminal 22 .
  • FIG. 9 shows data input by the information processing apparatus 100 from the customer side user terminal 21 and the sales side user terminal 22, and is part of the data (meeting log) used for machine learning processing in the information processing apparatus 100. It is an example of data showing
  • the recording area [label] at the beginning of the log data (Meeting log) shown in FIG. 9 is a recording area for the score (evaluation value) of the user status input by the customer-side user 11 playing the role of the customer during the meeting.
  • the next recording area is a recording area for the meeting conditions set before the start of the meeting described above with reference to FIG.
  • the information processing device 100 inputs log data (Meeting Log) composed of these data from at least one of the customer side user terminal 21 and the sales side user terminal 22, and performs machine learning using the input data. Execute processing to generate a user state estimator (learning model).
  • the log data shown in FIG. 9 is part of the log data acquired during the meeting period.
  • [Label] data acquired as log data during one meeting time-series data of user status scores (evaluation values) sequentially input by the customer side user 11 during the meeting is generated. can do.
  • FIG. 10 shows an example of time-series data that the information processing apparatus 100 can generate based on the score (evaluation value) of the user state acquired from the customer-side user terminal 21 .
  • FIG. 10 shows time-series data for each of the following user states.
  • (User state 1) Interest, interest, good feeling (User state 2) Understanding, consent, satisfaction (User state 3) Fatigue, stress
  • the state of the customer-side user terminal 21 during the meeting period can be determined. Detailed analysis of changes becomes possible.
  • the information processing apparatus 100 executes machine learning processing using the log data shown in FIG. 9 to generate a user state estimator (learning model). The details of the machine learning process executed by the information processing apparatus 100 will be described with reference to FIG. 11 and subsequent figures.
  • FIG. 11 is a diagram showing an example of a configuration for collecting data used for machine learning processing executed by the information processing apparatus 100.
  • the information processing device 100 acquires the following data during the meeting period from the customer-side user terminal 21 .
  • Image data Audio data Score corresponding to user condition 1)
  • Score corresponding to user condition 3) Fatigue, stress score
  • the image data and voice data are the image data including the face image of the customer user 11 acquired by the camera and microphone of the customer user terminal 21 and the voice data of the customer user 11 .
  • the information processing device 100 stores these data in a storage unit (database) within the information processing device 100 .
  • a storage unit database
  • image data and audio data are stored in an image/audio database 101 .
  • the interest, interest, and favorability scores are stored in the interest, interest, and favorability score database 121
  • the understanding, consent, and satisfaction scores are stored in the understanding, consent, and satisfaction score database 122
  • the fatigue and stress scores are stored in the fatigue and stress scores.
  • the information processing device 100 also acquires the following data during the meeting period from the sales side user terminal 22 as well.
  • Image Data Audio Data These image data and audio data are image data including the facial image of the sales user 12 acquired by the camera and microphone of the sales user terminal 22 and speech data of the sales user 12 . These data are also recorded in the image/audio database 101 of the information processing apparatus 100 .
  • the image and sound data recorded in the image/sound database 101 include recording time information such as a time stamp indicating the acquisition time of the image and sound as attribute information, and the image and sound acquired from the user terminal 21 on the customer side.
  • recording time information such as a time stamp indicating the acquisition time of the image and sound as attribute information
  • User identification information or the like for identifying whether the data is the data of the customer side user 11 or the data of the sales side user 12 acquired from the sales side user terminal 22 is recorded.
  • the learning data collection example of the information processing apparatus 100 shown in FIG. 11 is a configuration example in which image data and voice data during the meeting period are also acquired from the sales side user terminal 22.
  • FIG. image data and voice data from the user terminal 22 on the sales side, without obtaining image data and voice data from the user terminal 22 on the customer side. Configurations are also possible.
  • FIG. 13 is a diagram illustrating the configuration and processing of machine learning processing executed by the information processing apparatus 100.
  • the information processing apparatus 100 includes an interest level estimator generation unit (interest level estimation learning model generation unit) 131, an understanding level estimator generation unit (understanding level estimation learning model generation unit) 132, a fatigue level estimation It has a device generation unit (fatigue level estimation learning model generation unit) 133 .
  • the interest level estimator generation unit (interest level estimation learning model generation unit) 131 executes machine learning processing using the data stored in the interest/interest/likeability score database 121 and the data stored in the image/audio database 101 to determine the interest level.
  • a degree estimator (interest degree estimating learning model) 141 is generated.
  • the interest level estimator generation unit (interest level estimation learning model generation unit) 131 executes machine learning processing using the following data to generate the interest level estimator (interest level estimation learning model) 141. .
  • the comprehension estimator generation unit (comprehension estimation learning model generation unit) 132 executes machine learning processing using the data stored in the comprehension/satisfaction/satisfaction score database 122 and the data stored in the image/audio database 101. , generates an understanding level estimator (understanding level estimation learning model) 142 .
  • the understanding level estimator generation unit (understanding level estimation learning model generation unit) 132 executes machine learning processing using the following data to generate the understanding level estimator (understanding level estimation learning model) 142 .
  • the fatigue level estimator generation unit (fatigue level estimation learning model generation unit) 133 executes machine learning processing using the stored data of the fatigue/stress score database 123 and the stored data of the image/audio database 101, A degree estimator (fatigue degree estimation learning model) 143 is generated.
  • the fatigue level estimator generation unit (fatigue level estimation learning model generation unit) 133 executes machine learning processing using the following data to generate the fatigue level estimator (fatigue level estimation learning model) 143 .
  • the learning processing units of the information processing apparatus 100 that is, the interest level estimator generation unit (interest level estimation learning model generation unit) 131, the understanding level estimator generation unit (understanding level estimation learning model generation unit) 132, and the fatigue level estimator
  • the generation unit (fatigue level estimation learning model generation unit) 133 and these learning processing units execute deep learning processing as machine learning processing, for example.
  • the interest level estimator generation unit (interest level estimation learning model generation unit) 131 uses the data stored in the interest/interest/likeability score database 121 and the data stored in the image/audio database 101 as training data for “supervised learning processing. ”.
  • a degree estimator (interest degree estimating learning model) 141 is generated.
  • the comprehension estimator generation unit (comprehension estimation learning model generation unit) 132 uses the data stored in the comprehension/satisfaction/satisfaction score database 122 and the data stored in the image/audio database 101 as teacher data for “supervised learning processing. ”.
  • the comprehension/satisfaction/satisfaction score of the customer-side user is estimated based on at least one of image and voice data of the customer-side user and image and voice data of the sales-side user.
  • a degree estimator (understanding degree estimation learning model) 142 is generated.
  • the fatigue level estimator generation unit (fatigue level estimation learning model generation unit) 133 performs “supervised learning processing” using the data stored in the fatigue/stress score database 123 and the data stored in the image/audio database 101 as teacher data. Execute.
  • Fatigue level estimation for estimating the fatigue/stress score of the user on the customer side based on at least one of image and voice data of the user on the customer side and image and voice data on the sales side by this learning process
  • a device (fatigue level estimation learning model) 143 is generated.
  • FIG. 14 is a diagram illustrating an example of learning processing for generating the interest level estimator (interest level estimation learning model) 141 by the interest level estimator generation unit (interest level estimation learning model generation unit) 131 .
  • FIG. 14 shows the following data used as learning data.
  • Interest level estimator generation unit (interest level estimation learning model generation unit) 131 first extracts interest, interest, and favorable impression score database 121 shown in FIG. Get one score (evaluation value). Furthermore, the time stamp of the acquired interest/interest/favorability score (evaluation value) is confirmed, and the image and voice data of the customer side user 11 up to a predetermined time (for example, 30 seconds) before the time matching this time stamp are displayed. Acquired from the image/sound database 101 .
  • the interest level estimator generation unit (interest level estimation learning model generation unit) 131 acquires the feature amount of the image and voice data of the customer side user 11 acquired from the image/sound database 101, and converts the feature amount into the figure.
  • a learning process is performed using a data set in which the image and voice data of the customer side user 11 for 30 seconds immediately before the score input is associated with the interest, concern, and favorability score (evaluation value) input by the customer side user 11 as teacher data.
  • the interest/interest/favorability score (evaluation value) input by the customer-side user 11 is used as annotation data (answer metadata) for the image/audio data.
  • the customer-side user 11's interest, interest, and favorability score (evaluation value) can be generated.
  • the image and voice of the customer side user 11 immediately before inputting the interest/interest/favorability score (evaluation value) 1 shown in FIG. , and voice data such as "Eh ⁇ " and "Hmm ⁇ " are recorded.
  • the interest level estimator generating unit (interest level estimating learning model generating unit) 131 determines the interest, interest, and favorability of the customer side user 11 when such data is recorded in the image and voice of the customer side user 11 . It can be learned that the score (evaluation value) tends to be low.
  • FIG. 15 shows examples of different learning data.
  • Interest level estimator generating unit interest level estimating learning model generating unit 131 calculates interest, interest, and favorable impression score ( It can be learned that the value of evaluation value) tends to be high.
  • FIG. 16 is an example of learning processing using image and voice data of the sales side user 12 .
  • FIG. 16(a) records the troubled face of the sales side user 12 and the voice data such as "About that matter” and "I'll look into it.”
  • the interest level estimator generation unit (interest level estimation learning model generation unit) 131 calculates the customer side user 11's interest/interest/favorability score ( It can be learned that the value of evaluation value) tends to be low.
  • the interest level estimator generation unit (interest level estimation learning model generation unit) 131 generates the interest/concern/likeability score (evaluation value) input by the customer side user 11 during the meeting period, the customer side user 11, Alternatively, the sales side user 12 inputs a large number of learning data composed of at least one of image and voice data, and executes learning processing.
  • the interest level estimator generation unit (interest level estimation learning model generation unit) 131 generates at least one of image and voice data of the user on the customer side, or image and voice data of the user on the sales side, and voice data as a learning processing result. based on, an interest level estimator (interest level estimation learning model) 141 for estimating the interest, interest, and favorability score of the user on the customer side is generated.
  • the interest level estimator generator (interest level estimation learning model generator) 131 generates at least one of image and voice data of the user on the customer side and image and voice data of the user on the sales side.
  • An interest level estimator (interest level estimating learning model) 141 is generated which inputs image and voice data and outputs, as an output, estimated values of customer-side users' interest, interest, and favorability scores.
  • an estimator that estimates the user state based on at least one of image data and voice data of one or more users participating in a meeting via a communication network is generated.
  • the example described with reference to FIGS. 14 to 17 includes learning processing by an interest level estimator generation unit (interest level estimation learning model generation unit) 131 and an interest level estimator generated as a learning result (interest level estimation learning model). model) 141.
  • the other comprehension level estimator generation unit (comprehension level estimation learning model generation unit) 132 and fatigue level estimator generation unit (fatigue level estimation learning model generation unit) 133 also execute similar learning processing.
  • the comprehension estimator generation unit (comprehension estimation learning model generation unit) 132 inputs and outputs at least one of the image and voice data of the user on the customer side and the image and voice data of the user on the sales side, and outputs the data.
  • an understanding level estimator (understanding level estimating learning model) 142 that outputs an estimated value of the understanding/satisfaction/satisfaction score of the user on the customer side is generated.
  • the fatigue level estimator generation unit (fatigue level estimation learning model generation unit) 133 inputs at least one of the image and voice data of the user on the customer side and the image and voice data of the user on the sales side, and voice data.
  • a fatigue level estimator (fatigue level estimation learning model) 143 that outputs an estimated fatigue/stress score of the user on the customer side as an output.
  • the processing described below includes machine learning processing for generating a user state estimator (learning model) executed by the information processing apparatus 100 in (step S105) of the flowchart described with reference to FIG. 5, and (step S106) This corresponds to selection processing of the user state estimator (learning model) executed in .
  • An interest level estimator generation unit (interest level estimation learning model generation unit) 131 generates one interest level estimator (interest level estimation learning model generation unit) 141, and an understanding level estimator generation unit (understanding level estimation learning model generation unit).
  • 132 generates one understanding level estimator (understanding level estimation learning model) 142, and
  • a fatigue level estimator generation unit (fatigue level estimation learning model generation unit) 133 generates one fatigue level estimator (fatigue level estimation learning model ) 143 is an example of processing.
  • the processing example described below is a processing example in which the estimator generators (learning model generators) 131 to 133 are generated as a plurality of user state estimators (learning models) according to meeting conditions.
  • the meeting condition is the meeting condition (MTG tag) previously described with reference to FIG. That is, meeting conditions (MTG tags) set before the start of the meeting are the following conditions.
  • each estimator generator (learning model generator) 131 to 133 generates a plurality of user state estimators (learning models) according to meeting conditions. do.
  • an example of processing for generating an interest level estimator (interest level estimation learning model) corresponding to meeting conditions by the interest level estimator generation unit (interest level estimation learning model generation unit) 131 will be described.
  • FIG. 18 is a diagram illustrating a configuration and a processing example of an interest level estimator generation unit (interest level estimation learning model generation unit) 131 that generates a plurality of meeting condition corresponding interest level estimators (interest level estimation learning models). .
  • the interest level estimator generation unit (interest level estimation learning model generation unit) 131 has a data selection unit 150 and a meeting condition corresponding interest level estimator (interest level estimation learning model) generation unit 160 .
  • the data selection unit 150 acquires only data that matches predetermined meeting conditions from the image/sound database 101 and the interest/interest/likeability score database 121 .
  • These databases are the databases described above with reference to FIG. 11, and store image and audio data acquired in past meetings and interest/interest/likeability scores (evaluation values). .
  • Meeting (MTG) Genre Business negotiation
  • the image and audio data acquired in the meeting for which this meeting condition is set, and the interest/interest/favorability score (evaluation value) data are stored in the image/audio database 101 and the interest/interest/favorability score. Acquired from the database 121 .
  • Meeting (MTG) Scale Medium Image and audio data and interest/interest/favorability score (evaluation value) data acquired in a meeting for which this meeting condition is set are stored in the image/audio database 101 and interest/interest/favorability scores. Acquired from the database 121 .
  • Meeting (MTG) scale medium Execute machine learning processing using the image and audio data obtained in the meeting with this meeting condition, and the interest / interest / favorability score (evaluation value) data
  • the image and audio data acquired in the meeting set with these meeting conditions Then, the interest/interest/favorability score (evaluation value) data is acquired from the image/audio database 101 and the interest/interest/favorability score database 121 .
  • the information processing apparatus 100 executes the same processing for all the types of combination of meeting conditions, and " Interest degree estimator corresponding to meeting conditions (learning model)” is generated.
  • tags that can be set as meeting conditions.
  • (Tag c) Fatigue (customer) large, medium, small
  • (Tag d) Busyness (customer ) large, medium, small
  • the information processing device 100 performs the same processing for all of these meeting condition combination types, and generates meeting condition corresponding interest level estimators (learning models) corresponding to all of the meeting condition combination types.
  • the information processing apparatus 100 When the information processing apparatus 100 completes the process of generating interest level estimators (learning models) corresponding to meeting conditions for all combinations of meeting conditions, the information processing apparatus 100 next generates these many meeting condition corresponding interest level estimators. Execute evaluation processing of the device (learning model). A specific example of this estimator evaluation process will be described with reference to FIG.
  • the information processing apparatus 100 has an interest level estimator (interest level estimation learning model) storage unit 170 and an estimator (learning model) evaluation unit 180 .
  • Interest level estimator (interest level estimation learning model) storage unit 170 stores interest level estimator (interest level estimation learning model) 141 generated by the learning process described above with reference to FIG.
  • the interest level estimator corresponding to meeting conditions (interest level estimation learning model) described above is stored.
  • the interest level estimator (interest level estimation learning model) 141 is an interest level estimator (interest level estimation learning model) generated using almost all log data generated using data not limited by meeting conditions. model).
  • the estimator (learning model) evaluation unit 180 executes evaluation processing for each interest level estimator (interest level estimation learning model) stored in the interest level estimator (interest level estimation learning model) storage unit 170 .
  • Data stored in the image/sound database 101, the interest, interest, and favorable impression score database 121, the comprehension, consent, and satisfaction score database 122, and the fatigue and stress score database 123 are used for this evaluation processing. As described above with reference to FIG. This is the score (evaluation value) of each user state.
  • the estimator (learning model) evaluation unit 180 shown in FIG. 20 executes evaluation processing for each interest level estimator (interest level estimation learning model) stored in the interest level estimator (interest level estimation learning model) storage unit 170. do.
  • a specific example of the estimator (learning model) evaluation process executed by the estimator (learning model) evaluation unit 180 will be described with reference to FIG. 22 .
  • the evaluation process executed by the estimator (learning model) evaluation unit 180 is the following evaluation process of each estimator.
  • An evaluation process example of an interest level estimator (interest level estimation learning model) generated by the interest level estimator generation unit (interest level estimation learning model generation unit) 131 will be described below as a representative example with reference to FIG. .
  • FIG. 22 is a diagram explaining the details of the interest level estimator (interest level estimation learning model) evaluation process executed by the estimator (learning model) evaluation unit 180 shown in FIG.
  • the estimator (learning model) evaluation unit 180 first selects the degree of interest to be evaluated from the interest estimator (interest degree estimation learning model) stored in the interest degree estimator (interest degree estimation learning model) storage unit 170. Select one estimator (interest estimation learning model).
  • An estimator (learning model) evaluation unit 180 inputs image and audio data, which are test data acquired from the image/audio database 101, to one selected interest level estimator (interest level estimation learning model), and evaluates interest. - Execution of estimation processing of the interest/favorability score (evaluation value).
  • One selected interest level estimator (interest level estimation learning model) is sequentially input with test data (image and audio data) corresponding to various meeting conditions to execute score estimation processing.
  • the test data is composed of data groups for a plurality of meeting conditions such as all data, condition 1, condition 2, condition 3, and so on. These are sequentially input to the interest degree estimator (interest degree estimation learning model) to be evaluated.
  • Data matching these conditions are sequentially acquired from the image/sound database 101 and the interest/interest/favorability score database 121, and evaluation processing is executed.
  • the estimator (learning model) evaluation unit 180 inputs image and audio data that match certain meeting conditions acquired from the image/sound database 101 to the interest degree estimator (learning model) to be evaluated, and Output the estimated value of the interest/favorability score (evaluation value).
  • the estimator (learning model) evaluation unit 180 compares this estimated score with the actual score (evaluation value) stored in the interest/interest/likeability score database 121, that is, the score input by the customer side user 11 during the meeting.
  • the rate of matching by comparison is calculated. [xy%] shown in FIG. 22 is a numerical value indicating this matching rate.
  • the score (evaluation value) stored in the interest/interest/favorability score database 121 is regarded as the correct answer, and the rate at which the estimated score, which is the output of the interest level estimator (learning model), matches the correct answer is calculated.
  • image and audio data input from the image/sound database 101 to the interest level estimator (learning model) are stored for a predetermined period of time from the time stamp of the actual score (evaluation value) stored in the interest/interest/likeability score database 121. , for example, image and audio data from 30 seconds before to the time matching the time stamp.
  • the estimator (learning model) evaluation unit 180 first sets one meeting condition, for example, condition 1
  • the score (evaluation value) actually input by the customer-side user 11 in the meeting for which the matching meeting conditions have been set is acquired from the interest/interest/likeability score database 121.
  • the image and audio data from the time stamp of the interest/interest/good feeling score (evaluation value) acquired from the interest/interest/good feeling score database 121 are displayed for a predetermined period of time, for example, 30 seconds before until the time matching the time stamp.
  • the estimated score of the interest degree estimator 002 (learning model 002 ) to be evaluated matches the actual user input score acquired from the interest/interest/likeability score database 121 .
  • An estimator (learning model) capable of highly accurate estimation is determined as the matching rate is high.
  • the test data is divided into a plurality of meeting condition units such as all data, condition 1, condition 2, condition 3, and so on. All the data are data regardless of the meeting conditions, and evaluation processing is performed using all test data stored in the image/sound database 101 and the interest/interest/favorability score database 121.
  • FIG. 22 shows that all data, condition 1, condition 2, condition 3, and so on. All the data are data regardless of the meeting conditions, and evaluation processing is performed using all test data stored in the image/sound database 101 and the interest/interest/favorability score database 121.
  • FIG. 22 shows the evaluation result obtained as a result of such evaluation processing, that is, the matching rate between the estimated score of each estimator (learning model) and the actual user input score as the answer.
  • the match rate (accuracy rate) as an evaluation value obtained by inputting the test data of [Condition 2] to each estimator (learning model) is 85% for the interest estimator 002 (learning model 002). Yes, showing the highest value. From this result, it can be concluded that it is preferable to use the degree-of-interest estimator 002 (learning model 002), which has the highest accuracy rate, in a meeting in which conditions matching meeting condition 2 are set.
  • estimator evaluation process described with reference to FIG. 22 is an evaluation process example of the interest level estimator (interest level estimation learning model) generated by the interest level estimator generation unit (interest level estimation learning model generation unit) 131. is.
  • the estimator (learning model) evaluation unit 180 evaluates the comprehension level estimator (understanding level estimation learning model) generated by the understanding level estimator generation unit (understanding level estimation learning model generation unit) 132.
  • the fatigue level estimator Evaluation processing of the fatigue level estimator (fatigue level estimation learning model) generated by the generation unit (fatigue level estimation learning model generation unit) 133 also executes evaluation processing in the same manner.
  • the image/audio database 101 and the understanding/ Data stored in the consent/satisfaction score database 122 is used.
  • the image / audio database 101, the fatigue / Data stored in the stress score database 123 is used.
  • the evaluation process performed in the estimator (learning model) evaluation unit 180 selects the optimum estimator (learning model) to be used under various meeting conditions, that is, the optimum estimator (learning model) capable of executing the most accurate estimation process (learning model). model) can be determined.
  • the information processing apparatus of the present disclosure executes the following two processes.
  • Process 2) A process of estimating user emotion using the generated user state estimator (learning model).
  • FIG. 25 is a diagram showing an example of a remote meeting via a communication network.
  • FIG. 25 shows a customer side user 11 who is a customer who wishes to purchase a product, and a sales side user 12 who is a product provider.
  • a customer-side user terminal 21 such as a smartphone and a sales-side user terminal 22 such as a PC are connected via a communication network, and voices and images are mutually transmitted and received between these communication terminals to carry out business negotiations.
  • the customer side user 11 was not the actual customer but the user who played the role of the customer, but the customer side user 11 shown in FIG. 25 is the actual customer.
  • the customer side user 11 who is a customer is a person who wishes to purchase an apartment
  • the sales side user 12 is an apartment seller.
  • the sales side user 12 selects an apartment that meets the customer side user's 11 request and explains it while listening to the customer side user's 11 request.
  • An information processing device 100 connected to a network is a device that has generated an estimator (learning model) by the above-described (processing 1). That is, an estimator (learning model) for estimating each user state below is generated and held.
  • an estimator (learning model) for estimating each user state below is generated and held.
  • the example shown in FIG. 26 is a specific example of interest level estimation processing of the customer side user 11 using the interest level estimator (interest level estimation learning model) 190 generated by the information processing apparatus 100 .
  • FIG. 26( a ) shows an example of data acquired by the customer-side user terminal 21 and transmitted to the information processing device 100 .
  • the camera and microphone of the customer-side user terminal 21 acquire image data including the face of the customer-side user and voice data including the voice of the customer-side user 11, and transmit them to the information processing apparatus 100 via the network. be done.
  • FIG. 26(b) shows the customer-side user's 11 interest degree estimation process executed in the information processing apparatus 100.
  • the information processing apparatus 100 inputs the customer-side user's image data and voice data received from the customer-side user terminal 21 to an interest level estimator (interest level estimation learning model) 190 .
  • an interest level estimator interest level estimation learning model
  • the interest level estimator (interest level estimation learning model) 190 used here is determined based on the meeting conditions executed between the customer side 11 and the sales side user 12 .
  • the estimator selected in the evaluation process of the interest level estimator (interest level estimation learning model) described above with reference to FIG. 22 is used. Specifically, an interest level estimator (interest level estimation learning model) determined to be capable of estimating the score with the highest accuracy under the meeting conditions executed between the customer side 11 and the sales side user 12 is selected. .
  • the information processing device 100 transmits to the sales side user terminal 22 the estimated values of the interest/interest/favorability score obtained as the output of the interest level estimator (interest level estimation learning model) 190 .
  • FIG. 26(c) is a diagram showing a processing example of the sales side user terminal 22.
  • the sales-side user terminal 22 receives the estimated interest/interest/favorable score, which is the output value of the interest level estimator (interest level estimation learning model) 190, from the information processing apparatus 100, and sells an icon corresponding to the received score. displayed on the side user terminal 22 .
  • the example shown in the figure is an icon display example in the case where the customer side user 11's estimated interest/interest/favorability score is 5, which is the highest value. It is an icon indicating that there is A green icon containing the letter of interest is displayed as shown.
  • the sales-side user 12 can check the display of this icon to confirm that the customer-side user 11 is in a state of high interest, concern, and favorability.
  • the icons displayed on the sales-side user terminal 22 are estimated scores (evaluation values) obtained by applying the interest level estimator (interest level estimation learning model) 190 in the information processing apparatus 100, that is, the values of interest, interest, The icon becomes different depending on the value of the favorable feeling estimation score (evaluation value).
  • FIG. 27 shows three types of icon output examples for the sales side user terminal 22 .
  • FIG. 27(1) Icon output example 1 shows a case where the estimated value of the interest/interest/likeability score output by the interest level estimator (interest level estimation learning model) 190 of the information processing apparatus 100 is the maximum value (5). It is an icon output example. In this case, a green icon containing the letter of interest is displayed as shown.
  • FIG. 27(2) Icon output example 2 is an icon output when the estimated interest/interest/likeability score output by the interest level estimator (interest level estimation learning model) 190 of the information processing apparatus 100 is (3). For example. In this case, a yellow icon containing the letter of interest is displayed as shown. Alternatively, icon display is stopped.
  • FIG. 27(3) Icon output example 3 shows a case where the estimated value of the interest/interest/likeability score output by the interest level estimator (interest level estimation learning model) 190 of the information processing apparatus 100 is the lowest value (1).
  • This is an icon output example. In this case, a red icon containing the character of interest is displayed as shown.
  • the sales side user 12 can check the display of these icons to check whether the customer side user 11 is in a state of high or low interest, concern, and good feeling, and can interact according to the check result. It is possible to take appropriate measures such as changing the content.
  • the information processing apparatus 100 uses an interest level estimator (interest level estimation learning model) to estimate the customer side user's 11 interest, concern, and favorability score, and This is an example of processing for displaying an icon reflecting the score estimation result on the sales side user terminal 22 based on the estimation result.
  • interest level estimator interest level estimation learning model
  • the information processing apparatus 100 holds different user state estimators in addition to the interest level estimator (interest level estimation learning model) 190 . That is, it holds an estimator (learning model) for estimating each of the following user states.
  • an estimator for estimating each of the following user states.
  • FIG. 28 shows icons displayed on the sales side user terminal 22 when the information processing apparatus 100 estimates the understanding/satisfaction/satisfaction score of the customer side user 11 using the understanding level estimator (understanding level estimation learning model). , that is, a diagram showing display examples of various icons reflecting estimation results.
  • Icon output example 1 is an icon when the estimated value of the comprehension/conviction/satisfaction score output by the understanding level estimator (understanding level estimation learning model) of the information processing apparatus 100 is the highest value (5). This is an output example. In this case, a green icon with the word comprehension is displayed as shown.
  • FIG. 28 (2) Icon output example 2 is an icon output example when the estimated value of the understanding/satisfaction/satisfaction score output by the comprehension level estimator (understanding level estimation learning model) of the information processing apparatus 100 is (3). is. In this case, a yellow icon with the word comprehension is displayed as shown. Alternatively, icon display is stopped.
  • Icon output example 3 is an icon when the estimated value of the comprehension/conviction/satisfaction score output by the understanding level estimator (understanding level estimation learning model) of the information processing apparatus 100 is the lowest value (1). This is an output example. In this case, a red icon with the word comprehension is displayed as shown.
  • the sales-side user 12 can check the display of these icons to check whether the customer-side user 11 is in a high or low state of understanding, consent, and satisfaction, and can conduct a dialogue according to the confirmation result. It is possible to take appropriate measures such as changing the content.
  • FIG. 29 shows an icon displayed on the sales-side user terminal 22 when the information processing apparatus 100 estimates the fatigue/stress score of the customer-side user 11 using the fatigue level estimator (fatigue level estimation learning model).
  • FIG. 10 is a diagram showing display examples of various icons reflecting estimation results;
  • Icon output example 1 is an icon output example when the estimated fatigue/stress score output by the fatigue level estimator (fatigue level estimation learning model) of the information processing apparatus 100 is the highest value (5). is. In this case, a red icon containing characters of stress is displayed as shown in the figure.
  • FIG. 29 (2) Icon output example 2 is an icon output example when the fatigue/stress score estimated value output by the fatigue level estimator (fatigue level estimation learning model) of the information processing apparatus 100 is (3). . In this case, a yellow icon containing the words Relax is displayed as shown in the figure. Alternatively, icon display is stopped.
  • FIG. 29 (3) Icon output example 3 is an icon output example when the estimated value of the fatigue/stress score output by the fatigue level estimator (fatigue level estimation learning model) of the information processing apparatus 100 is the lowest value (1). is. In this case, a green icon containing the words Relax is displayed as shown in the figure.
  • the sales side user 12 can check the display of these icons to check whether the fatigue/stress of the customer side user 11 is high or low, and can change the content of the dialogue according to the check result. Appropriate measures such as changing are possible.
  • the information processing apparatus 100 of the present disclosure performs the evaluation process of the estimator (learning model) evaluation unit 180 in advance to perform the estimator (learning model) to be used under various meeting conditions. model). That is, an estimator (learning model) to be used by an estimator (learning model) capable of outputting an evaluation value with the highest accuracy is determined according to the meeting conditions.
  • the most accurate estimator (learning model) is used according to the conditions of the meeting.
  • Such an estimator (learning model) selection process makes it possible to perform a highly accurate user state estimation process.
  • FIG. 30 is a block diagram showing a configuration example of the information processing apparatus 100 of the present disclosure.
  • the information processing apparatus 100 of the present disclosure executes the following two processes.
  • Process 2) A process of estimating user emotion using the generated user state estimator (learning model).
  • the information processing apparatus 100 shown in FIG. 30 has a configuration for executing these two processes.
  • the information processing apparatus 100 includes a communication unit 201, a storage unit 202, a learning processing unit (estimator (learning model) generation unit) 203, a user state estimator (learning model) 204, an estimator (learning model ) evaluation/update unit 205 , usage estimator (learning model) selection unit 206 , and user state estimation result output unit 207 .
  • the storage unit 202 has an image/sound database 211 , an interest/interest/favorable score database 212 , an understanding/agreement/satisfaction score database 213 , and a fatigue/stress score database 214 .
  • the user state estimator (learning model) 204 includes interest level estimators 1 to n (interest level estimation learning models 1 to n) 221 and understanding level estimators 1 to n (understanding level estimation learning models 1 to n) 222. , fatigue level estimators 1 to n (fatigue level estimation learning models 1 to n) 223 .
  • the communication unit 201 executes communication with the customer side user terminal 21 and the sales side user terminal 22 . Input image and audio data from each terminal. In addition, when performing processing for generating a user state estimator (learning model), the user state score (evaluation value) input by the customer side user is input from the customer side user terminal 21 .
  • a user state estimator learning model
  • the value of the user state estimation score applied to the user state estimator (learning model) for the sales side user terminal 22 or send icon data according to the score value.
  • the image/audio database 211 of the storage unit 202 stores image and audio data transmitted from the customer-side user terminal 21 and the sales-side user terminal 22 .
  • the interest/interest/favorable impression score database 212, the understanding/satisfaction/satisfaction score database 213, and the fatigue/stress score database 214 are each input from the customer-side user terminal 21 when the user state estimator (learning model) generation process is executed. Stores the user state score (evaluation value). As described above, these have time stamps added as attribute data.
  • a learning processing unit (estimator (learning model) generating unit) 203 executes learning processing using data stored in each database of the storage unit 202 to generate a user state estimator (learning model). Specifically, the following three types of user state estimators (learning models) are generated.
  • the learning processing unit (estimator (learning model) generation unit) 203 generates a plurality of user state estimators (learning models) corresponding to meeting conditions, as described with reference to FIG.
  • the user state estimator (learning model) generated by the learning processing unit (estimator (learning model) generating unit) 203 is the following estimators shown as the user state estimator (learning model) 204 in FIG.
  • Interest level estimators 1 to n interest level estimation learning models 1 to n
  • Understanding level estimators 1 to n understanding level estimation learning models 1 to n
  • Fatigue level estimators 1 to n fatigue level estimation learning models 1 to n 223
  • the learning processing unit (estimator (learning model) generation unit) 203 performs learning processing using the data to generate an estimator that has already been generated. (learning model) is also sequentially updated.
  • the estimator (learning model) evaluation and update unit 205 executes evaluation processing and update processing of the estimator (learning model) generated by the learning processing unit (estimator (learning model) generation unit) 203 . This process is the process described above with reference to FIG.
  • An estimator (learning model) evaluation and updating unit 205 acquires test data corresponding to various meeting conditions from each database in the storage unit 202 and executes evaluation processing of each estimator (learning model).
  • the estimator (learning model) evaluation and updating unit 205 includes the following estimators in the user state estimator (learning model) 204, that is, Interest level estimators 1 to n (interest level estimation learning models 1 to n) 221, Understanding level estimators 1 to n (understanding level estimation learning models 1 to n) 222, Fatigue level estimators 1 to n (fatigue level estimation learning models 1 to n) 223 In each of these estimators, meeting condition information to be used based on evaluation results is recorded as attribute information.
  • the estimator (learning model) evaluation and updating unit 205 performs estimation at any time when updating the estimator (learning model) by accumulating new learning data or accumulating new test data. Execute evaluation processing of the device (learning model) and update processing of the evaluation result. Further, processing is performed to reflect the evaluation and update results to the attribute information of each estimator in the user state estimator (learning model) 204 .
  • a usage estimator (learning model) selection unit 206 performs processing for selecting a user state estimator (learning model) to be used when a meeting is actually held for user state estimation processing. That is, each of the following estimators in user state estimator (learning model) 204: Interest level estimators 1 to n (interest level estimation learning models 1 to n) 221, Understanding level estimators 1 to n (understanding level estimation learning models 1 to n) 222, Fatigue level estimators 1 to n (fatigue level estimation learning models 1 to n) 223 An estimator (learning model) to be used is selected from each of these estimators.
  • the estimator (learning model) with the highest accuracy is selected as the estimator (learning model) to be used, depending on the meeting conditions of the meeting to be held.
  • the user state estimation result output unit 207 outputs a user state estimation score estimated using a user state estimator (learning model) in an actual meeting, that is, Interest/Interest/Favorability Estimation Score (Evaluation Value) Understanding, Consent, Satisfaction Estimated Score (Evaluation Value) Fatigue, stress estimation score (evaluation value)
  • a user state estimator learning model
  • Interest/Interest/Favorability Estimation Score Evaluation Value
  • Consent Consent
  • Satisfaction Estimated Score Evaluation Value
  • Fatigue stress estimation score
  • the sales-side user terminal 22 outputs the icons previously described with reference to FIGS. 27 to 29 to the display section.
  • the information processing apparatus 100 shown in FIG. 31 has the same configuration as the information processing apparatus 100 described with reference to FIG. 203 , a user state estimator (learning model) 204 , an estimator (learning model) evaluation and updating unit 205 , a usage estimator (learning model) selection unit 206 , and a user state estimation result output unit 207 . Since each of these components has been described with reference to FIG. 30, description thereof will be omitted.
  • the customer-side user terminal 21 has an input section 310 , an output section 320 and a communication section 330 .
  • the input unit 310 has a voice input unit (microphone) 311 , an image input unit (camera) 312 and a user input unit (UI) 313 .
  • the output unit 320 has an audio output unit (speaker) 321 and an image output unit (display unit) 322 .
  • a voice input unit (microphone) 311 of the input unit 310 acquires voice data such as the voice of the user on the customer side. The acquired voice data is transmitted to the sales side user terminal 22 and the information processing device 100 via the communication unit 330 .
  • An image input unit (camera) 312 acquires image data such as a face image of a user on the customer side. The acquired image data is transmitted to the sales side user terminal 22 and the information processing device 100 via the communication unit 330 .
  • the user input unit (UI) 313 is an interface for inputting a user state score (evaluation value) by the user 11 on the customer side when executing a user state estimator (learning model) generation process in the information processing apparatus 100, for example.
  • a touch panel type display unit is used.
  • the input data of the user condition score (evaluation value) input by the customer-side user 11 is transmitted to the information processing device 100 and used in the learning process executed during the estimator (learning model) generation process.
  • the sales side user terminal 22 has an input section 410 , an output section 420 and a communication section 430 .
  • the input unit 410 has an audio input unit (microphone) 411 and an image input unit (camera) 412 .
  • the output unit 420 has an audio output unit (speaker) 421 and an image output unit (display unit) 422 .
  • a voice input unit (microphone) 411 of the input unit 410 acquires voice data such as the voice of the user on the sales side. The acquired voice data is transmitted to the customer-side user terminal 21 and the information processing device 100 via the communication unit 430 .
  • An image input unit (camera) 412 acquires image data such as a face image of a sales user. The obtained image data is transmitted to the customer-side user terminal 21 and the information processing device 100 via the communication unit 430 .
  • the image output unit (display unit) 422 displays icons indicating the user states described above with reference to FIGS. The display icon is determined according to the user state estimation score estimated using the user state estimator (learning model).
  • FIG. 32 is a diagram showing an example of the hardware configuration of the information processing apparatus 100 of the present disclosure, and the customer-side user terminal 21 and the sales-side user terminal 22 that are user terminals.
  • the hardware configuration shown in FIG. 32 will be described below.
  • a CPU (Central Processing Unit) 501 functions as a control section and a data processing section that execute various processes according to programs stored in a ROM (Read Only Memory) 502 or a storage section 508 . For example, the process according to the sequence described in the above embodiment is executed.
  • a RAM (Random Access Memory) 503 stores programs and data executed by the CPU 501 .
  • These CPU 501 , ROM 502 and RAM 503 are interconnected by a bus 504 .
  • the CPU 501 is connected to an input/output interface 505 via a bus 504.
  • the input/output interface 505 is connected to an input unit 506 including various switches, a keyboard, a mouse, a microphone, sensors, etc., and an output unit 507 including a display, speakers, etc. It is
  • the CPU 501 executes various types of processing in response to commands input from the input unit 506 and outputs processing results to the output unit 507, for example.
  • a storage unit 508 connected to the input/output interface 505 consists of, for example, a hard disk, and stores programs executed by the CPU 501 and various data.
  • a communication unit 509 functions as a transmitting/receiving unit for Wi-Fi communication, Bluetooth (registered trademark) (BT) communication, and other data communication via networks such as the Internet and local area networks, and communicates with external devices.
  • a drive 510 connected to the input/output interface 505 drives a removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card to record or read data.
  • a removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card to record or read data.
  • the technique disclosed in this specification can take the following configurations. (1) having a user state estimator generation unit that generates a user state estimator by executing machine learning processing using user state information input to a user terminal by a user who participates in a meeting via a communication network; The user state estimator generator, An information processing apparatus that generates a user state estimator that estimates a user state based on at least one of image data and voice data of users participating in a meeting.
  • the user state information is an interest score indicating the user's interest level in the meeting;
  • the user state estimator generator The information processing apparatus according to (1), which executes machine learning processing using interest scores to generate a user state estimator for estimating interest levels of meeting participating users.
  • the user state information is a comprehension score indicating the user's comprehension level of the meeting;
  • the user state estimator generator The information processing apparatus according to (1) or (2), which executes machine learning processing using comprehension scores to generate a user state estimator that estimates comprehension levels of users participating in the meeting.
  • the user state information is a fatigue score indicating the user's fatigue level for the meeting;
  • the user state estimator generator The information processing apparatus according to any one of (1) to (3), which generates a user state estimator for estimating the fatigue level of a user participating in a meeting by executing machine learning processing using the fatigue score.
  • the user state estimator generator The information processing apparatus according to any one of (1) to (4), which generates a plurality of meeting-condition-supported user state estimators according to meeting conditions.
  • the said meeting conditions are: (a) the genre of the meeting; (b) the size of the meeting; (c) fatigue of meeting participants; (d) the busyness of meeting participants;
  • the information processing apparatus according to (5) including at least one of the above conditions (a) to (d).
  • the information processing device an estimator evaluation unit that evaluates the user state estimation accuracy of the plurality of generated meeting condition corresponding user state estimators;
  • the estimator evaluation unit Inputting test data corresponding to meeting conditions to each of the generated user state estimators corresponding to meeting conditions to estimate the user state, and inputting the estimation result and correct data included in the test data by the user.
  • the information processing apparatus according to (5) or (6), which determines whether or not the obtained user state score matches, and calculates an accuracy rate.
  • the information processing device a usage estimator selection unit that selects an optimal meeting condition corresponding user state estimator according to meeting conditions;
  • the utilization estimator selection unit The information processing according to any one of (5) to (7), wherein the meeting-condition user state estimator having the highest accuracy rate calculated by the estimator evaluation unit is selected as the optimum meeting-condition user state estimator.
  • the user state estimator Any one of (1) to (9), which is a user state estimator that executes a user state estimation process using at least one of image data and voice data of meeting participants other than a user whose user state is to be estimated.
  • the information processing device according to .
  • the user state estimator generator Generating the user state estimator by executing machine learning processing using user state information input to a user terminal by a user participating in the meeting and image data or voice data within a predetermined period up to the input timing of the user state information
  • the information processing apparatus according to any one of (1) to (10).
  • a user state estimator for estimating a user state based on at least one of image data and voice data of a user participating in a meeting via a communication network;
  • An information processing apparatus having a user state estimation result output unit that outputs identification information indicating the user state estimated by the user state estimator to user terminals of users participating in the meeting.
  • the information processing device further includes: a utilization estimator selection unit that selects an optimal meeting-condition corresponding user state estimator that executes highly accurate user state estimation processing according to meeting conditions;
  • the information processing device is The information processing apparatus according to any one of (12) to (14), wherein the user state estimation process is performed using the meeting condition corresponding user state estimator selected by the usage estimator selection unit.
  • the meeting conditions are: (a) the genre of the meeting; (b) the size of the meeting; (c) fatigue of meeting participants; (d) the busyness of meeting participants;
  • An information processing method executed in an information processing device A user state estimator generator, executing a user state estimator generation process for generating a user state estimator by executing machine learning processing using user state information input to a user terminal by a user who participates in a meeting via a communication network;
  • the user state estimator generator An information processing method for generating a user state estimator for estimating a user state based on at least one of image data and voice data of users participating in a meeting.
  • the user state estimator is executing a user state estimation process for estimating a user state based on at least one of image data and voice data of a user participating in a meeting via a communication network;
  • the user state estimation result output unit An information processing method for executing a user state estimation result output process of outputting identification information indicating the user state estimated by the user state estimator to user terminals of users participating in the meeting.
  • a program for executing information processing in an information processing device In the user state estimator generator, executing a user state estimator generation process for generating a user state estimator by executing machine learning processing using user state information input to a user terminal by a user who participates in a meeting via a communication network;
  • the program wherein the user state estimator is a user state estimator that estimates the user state based on at least one of image data and voice data of users participating in the meeting.
  • the series of processes described in the specification can be executed by hardware, software, or a composite configuration of both.
  • a program recording the processing sequence is installed in the memory of a computer built into dedicated hardware and executed, or the program is loaded into a general-purpose computer capable of executing various processing. It can be installed and run.
  • the program can be pre-recorded on a recording medium.
  • the program can be received via a network such as a LAN (Local Area Network) or the Internet and installed in a recording medium such as an internal hard disk.
  • a system is a logical collective configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same housing.
  • a user state estimator that estimates a user's interest level, understanding level, etc. by learning processing using user state information input by a user participating in a meeting.
  • a construct to generate and use is realized. Specifically, for example, a user who participates in a meeting via a communication network executes machine learning processing using user state information input to the user terminal, and based on at least one of the image or voice of the meeting participant user
  • a user state estimator is generated that estimates the user state, eg, the user's interest level, comprehension level, and fatigue level.
  • the generated user state estimator is used to estimate the user state based on the images and voices of the users participating in the meeting, and the identification information and icon indicating the estimated user state are output to the user terminal.
  • a configuration is realized in which a user state estimator for estimating a user's degree of interest, understanding, etc. is generated and used by learning processing using user state information input by a user participating in a meeting.

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Strategic Management (AREA)
  • Development Economics (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Multimedia (AREA)
  • General Business, Economics & Management (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Data Mining & Analysis (AREA)
  • Human Resources & Organizations (AREA)
  • Game Theory and Decision Science (AREA)
  • Human Computer Interaction (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
PCT/JP2022/042968 2021-12-24 2022-11-21 情報処理装置、および情報処理方法、並びにプログラム Ceased WO2023119992A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP22910710.7A EP4455972A4 (en) 2021-12-24 2022-11-21 INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD AND PROGRAM
JP2023569183A JPWO2023119992A1 (https=) 2021-12-24 2022-11-21
US18/720,089 US20250054291A1 (en) 2021-12-24 2022-11-21 Information processing device, information processing method, and program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021211522 2021-12-24
JP2021-211522 2021-12-24

Publications (1)

Publication Number Publication Date
WO2023119992A1 true WO2023119992A1 (ja) 2023-06-29

Family

ID=86902215

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/042968 Ceased WO2023119992A1 (ja) 2021-12-24 2022-11-21 情報処理装置、および情報処理方法、並びにプログラム

Country Status (4)

Country Link
US (1) US20250054291A1 (https=)
EP (1) EP4455972A4 (https=)
JP (1) JPWO2023119992A1 (https=)
WO (1) WO2023119992A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7711992B1 (ja) 2024-05-29 2025-07-23 Ｎｅｃプラットフォームズ株式会社 コールセンタサーバ、その制御プログラム及びコールセンタシステム

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023135939A1 (ja) * 2022-01-17 2023-07-20 ソニーグループ株式会社 情報処理装置、および情報処理方法、並びにプログラム

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014069121A1 (ja) * 2012-10-31 2014-05-08 日本電気株式会社 会話分析装置及び会話分析方法
WO2019082687A1 (ja) 2017-10-27 2019-05-02 ソニー株式会社 情報処理装置および情報処理方法、プログラム、並びに情報処理システム
JP6792224B1 (ja) * 2020-07-01 2020-11-25 ニューラルポケット株式会社 情報処理システム、情報処理装置、サーバ装置、プログラム、又は方法
US20210306173A1 (en) * 2020-03-26 2021-09-30 Ringcentral, Inc. Methods and systems for managing meeting notes
WO2021210332A1 (ja) * 2020-04-14 2021-10-21 ソニーグループ株式会社 情報処理装置、情報処理システム、および情報処理方法、並びにプログラム

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11238872B2 (en) * 2017-11-22 2022-02-01 [24]7.ai, Inc. Method and apparatus for managing agent interactions with enterprise customers
US11228625B1 (en) * 2018-02-02 2022-01-18 mmhmm inc. AI director for automatic segmentation, participant behavior analysis and moderation of video conferences

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014069121A1 (ja) * 2012-10-31 2014-05-08 日本電気株式会社 会話分析装置及び会話分析方法
WO2019082687A1 (ja) 2017-10-27 2019-05-02 ソニー株式会社 情報処理装置および情報処理方法、プログラム、並びに情報処理システム
US20210306173A1 (en) * 2020-03-26 2021-09-30 Ringcentral, Inc. Methods and systems for managing meeting notes
WO2021210332A1 (ja) * 2020-04-14 2021-10-21 ソニーグループ株式会社 情報処理装置、情報処理システム、および情報処理方法、並びにプログラム
JP6792224B1 (ja) * 2020-07-01 2020-11-25 ニューラルポケット株式会社 情報処理システム、情報処理装置、サーバ装置、プログラム、又は方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4455972A4

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7711992B1 (ja) 2024-05-29 2025-07-23 Ｎｅｃプラットフォームズ株式会社 コールセンタサーバ、その制御プログラム及びコールセンタシステム
JP2025180020A (ja) * 2024-05-29 2025-12-11 Ｎｅｃプラットフォームズ株式会社 コールセンタサーバ、その制御プログラム及びコールセンタシステム

Also Published As

Publication number Publication date
EP4455972A4 (en) 2025-04-09
US20250054291A1 (en) 2025-02-13
JPWO2023119992A1 (https=) 2023-06-29
EP4455972A1 (en) 2024-10-30

Similar Documents

Publication Publication Date Title
US10720175B2 (en) Conferencing system and method for controlling the conferencing system
Rafaeli et al. The impact of call center employees' customer orientation behaviors on service quality
CN106663383B (zh) 分析受试者的方法和系统
CN107818798A (zh) 客服服务质量评价方法、装置、设备及存储介质
Saenz et al. The usability analysis of chatbot technologies for internal personnel communications
WO2024142031A1 (en) A natural language processing and ai-based platform for user feedback collection and product recommendation and follow-up with personalized vocie assistant
US20230290505A1 (en) Context Aware Assessment
WO2023119992A1 (ja) 情報処理装置、および情報処理方法、並びにプログラム
CN111405224A (zh) 在线交互控制方法、装置、存储介质及电子设备
CN111510659B (zh) 在线交互方法、装置、存储介质及电子设备
CN119150814A (zh) 会议纪要生成方法、装置、终端和计算机可读存储介质
US20260089024A1 (en) Group Engagement Analysis In Video Conferencing
CN114566187A (zh) 具有会话后表示的系统、电子装置和相关方法
JP2026034773A (ja) 情報処理装置、情報処理方法、情報処理プログラム及び記録媒体
JP2007257330A (ja) 顧客対応要員を評価させるプログラム、評価方法、および顧客対応要員の評価装置
WO2023135939A1 (ja) 情報処理装置、および情報処理方法、並びにプログラム
CN114298515A (zh) 生成学生素质画像的方法、装置及存储介质
JP2021162628A (ja) 情報処理装置、情報処理方法、情報処理プログラム及び記録媒体
JP7759523B1 (ja) 会議評価のためのシステム及びプログラム
CN119624491A (zh) 沟通技巧评估方法、电子设备和存储介质
KR20260013727A (ko) Ai 기반 연애 대화 훈련 시스템 및 방법
CN112232101A (zh) 一种用户沟通状态的评价方法、装置及系统
WO2023162020A1 (ja) 回答支援システム、回答支援方法および記録媒体
CN119646153A (zh) 一种基于客户行为生成对话文本的方法及相关产品
CN118335123A (zh) 一种通过通话进行智能测评的方法、系统、设备和介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22910710

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2023569183

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 18720089

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022910710

Country of ref document: EP

Effective date: 20240724