JP7123856B2 - Presentation evaluation system, method, trained model and program, information processing device and terminal device - Google Patents

Description

The present invention relates to a presentation evaluation system, method, trained model and program, as well as an information processing apparatus and terminal device used for presentation evaluation.

2. Description of the Related Art Conventionally, an evaluation device for evaluating a presentation given by a presenter is known. For example, in Patent Document 1, in a presentation performed by displaying a screen on a personal computer as a slide, which is a presentation material, on a screen for an audience member, it is possible to measure the percentage of the audience's gaze direction. Detect and judge with a direction detection device, and based on the results of the line of sight judgment and the results of comparison between the time allocated to each agenda page of the presentation material and the time actually spent by the presenter on each page A presentation evaluator is disclosed for evaluating the. According to this presentation evaluation device, the amount of time spent on the presentation material is recorded, the recorded value is quantified to what extent the evaluation can be given, and the It is said that it is possible to obtain an objective evaluation value of how much a person is making a presentation while looking at and contacting the audience.

JP 2007-219161 A

The evaluation by the above-described conventional evaluation apparatus is a limited evaluation such as the degree of time spent on the presentation and the degree of eye contact with the audience, and the viewpoint of evaluating the presentation is limited. Originally, a presentation should be performed by integrating the presenter's voice, utterance contents, facial expressions, gestures, etc., but the above-described conventional evaluation apparatus cannot realize such a presentation evaluation. In addition, since the above-described conventional evaluation device requires special hardware (line-of-sight direction detection device) for detecting the line of sight of the presenter, there is a high hurdle when implementing it in a terminal device or the like used by the user.

A system according to one aspect of the present invention is a system for evaluating presentations. This system includes a data acquisition unit that acquires voice data and video data of a target person who is giving a presentation, extracts language feature values and prosodic feature values of the presentation from the voice data, and extracts the presentation from the video data. a feature quantity extraction unit for extracting the motion feature quantity of the subject during presentation, and analyzing the linguistic feature quantity, the prosodic feature quantity, and the motion feature quantity to quantitatively evaluate a predetermined evaluation item of the presentation. and an analysis result output unit for outputting the analysis result.

The system further comprises a data confirmation unit for confirming whether the audio data and the video data acquired by the data acquisition unit have a predetermined quality, wherein the audio data and the video data having the predetermined quality. is used to extract the linguistic feature amount, the prosody feature amount, and the action feature amount, and the voice data and video data that do not have a predetermined quality are extracted from the linguistic feature amount, the prosody feature amount, and the action feature amount. It may not be used for extraction.
Here, the quality to be confirmed is that the sound pressure of the audio data is within a predetermined range, that the magnitude of noise in the audio data is equal to or less than a threshold, and that the motion feature amount is extracted from the image in the moving image data. At least one of the following: that the body part of the subject whose coordinates are to be acquired is included; may be of the quality of
and a data acquisition advice output unit for outputting an advice message regarding acquisition of the audio data and the video data when the audio data and the video data acquired by the data acquisition unit do not have a predetermined quality. may

In the system, the inference unit is an analysis model that outputs an analysis result including the quantitative evaluation value by processing an input including the linguistic feature amount, the prosodic feature amount, and the action feature amount with a predetermined algorithm. may be used.
In the inference unit, the linguistic feature amount, the prosodic feature amount, and the action feature amount each include a plurality of types of feature amounts, and the analysis model used in the inference unit is the speech data among the plurality of types of feature amounts. Further, the analysis model may not use, as the input, a feature quantity that requires a predetermined time or longer for patterning and extracting the moving image data.
The inference unit may use a plurality of types of analysis models with mutually different algorithms for the analysis.
The inference unit may have a plurality of types of analysis models with mutually different algorithms, and may use an analysis model selected from the plurality of types of analysis models for the analysis.
The inference unit may select an analysis model to be used for the analysis from the plurality of types of analysis models based on at least one of the linguistic feature amount, the prosodic feature amount, and the action feature amount.
The inference unit may select an analysis model to be used for the analysis from the plurality of types of analysis models based on at least one of an attribute of a target person of the presentation and a type of the presentation.
The analysis model is a learning created by machine learning using supervised learning data including the linguistic feature amount, the prosodic feature amount, the action feature amount, and the correct answer data of the evaluation value acquired for a plurality of presentations. It may be a finished model.

In the system, the linguistic feature amount may include a feature amount relating to at least one of filler number, noun number, verb number, interjection, verb repetition number, and noun repetition number in the full sentence of the presentation. Further, the prosodic feature amount may include a feature amount relating to at least one of pitch, intensity, sound pressure, intonation, speech speed, speech length, silence length, and speech ratio in speech of the presentation. Further, the motion feature amount may include a feature amount related to at least one of motion amount of each part of the subject's body during the entire presentation and motion amount of each part of the subject's body during speech.

In the system, the evaluation items may include coverage of purpose of the presentation, logicality of content, how it looks and sounds, and effective production elements.

In the system, the inference unit stores a plurality of comment data as comment candidates for the presentation, and based on the evaluation value and a coefficient of determination when the evaluation value is estimated, from the plurality of comment data: One or a plurality of comment data may be selected, and comments generated using the selected comment data may be included in the analysis result.

The system includes a terminal device and an information processing device that can communicate with each other via a communication network, and the terminal device includes the data acquisition unit and data for transmitting the audio data and the moving image data to the information processing device. a transmission unit, an analysis result reception unit that receives the analysis result from the information processing device, and the analysis result output unit, and the information processing device receives the audio data and the video data from the terminal device. A data receiving unit for receiving data, the feature quantity extracting unit, the inferring unit, and an analysis result transmitting unit for transmitting the analysis result to the terminal device may be provided.

A trained model according to another aspect of the invention is a trained model for operating a computer or processor to evaluate a presentation. This trained model was quantitatively evaluated with respect to the linguistic feature amount, prosodic feature amount, and motion feature amount extracted based on the subject's voice data and video data obtained for a plurality of presentations, and the predetermined evaluation items of the presentation. It is created by machine learning using the correct evaluation value data and supervised learning data, and is extracted based on the voice data and video data of the target audience for the presentation to be evaluated. When an input including a feature amount is received, an evaluation value obtained by quantitatively evaluating a predetermined evaluation item of the presentation is output.

A method for evaluating a presentation according to still another aspect of the present invention includes acquiring voice data and video data of a subject who is giving a presentation, and determining linguistic features and prosodic features of the presentation from the voice data. and extracting the motion feature of the subject during the presentation from the moving image data; analyzing the language feature, the prosody feature, and the motion feature to perform the presentation; It includes estimating an analysis result including an evaluation value quantitatively evaluated for a predetermined evaluation item, and outputting the analysis result.

A terminal device according to still another aspect of the present invention is a terminal device capable of communicating with an information processing device via a communication network. This terminal device includes a data acquisition unit that acquires audio data and video data of a target person who is giving a presentation, a data transmission unit that transmits the audio data and the video data to the information processing device, and the audio data and video data. An analysis result receiving unit for receiving, from the information processing device, an analysis result including an evaluation value quantitatively evaluated for a predetermined evaluation item of the presentation based on the moving image data, and an analysis result output unit for outputting the analysis result. And prepare.

An information processing device according to still another aspect of the present invention is an information processing device capable of communicating with a terminal device via a communication network. This information processing device includes a data receiving unit that receives voice data and moving image data of a target person who is giving a presentation from the terminal device; a feature amount extracting unit for extracting a motion feature amount of the subject during the presentation from video data; and an analysis result transmission unit configured to transmit the analysis result to the terminal device.

A program according to still another aspect of the present invention is a program executed by a computer or processor provided in a terminal device capable of communicating with an information processing device via a communication network. This program includes a program code for acquiring audio data and video data of a target person who is giving a presentation, a program code for transmitting the audio data and the video data to the information processing device, and the audio data. and a program code for receiving from the information processing apparatus an analysis result including an evaluation value quantitatively evaluated for a predetermined evaluation item of the presentation based on the moving image data; and a program for outputting the analysis result. including code and

A program according to still another aspect of the present invention is a program executed by a computer or processor provided in an information processing device capable of communicating with a terminal device via a communication network. This program extracts a program code for receiving voice data and video data of a target person who is giving a presentation from the terminal device, extracts the linguistic feature amount and the prosodic feature amount of the presentation from the voice data, and extracts the video data. A program code for extracting the motion feature of the subject during the presentation from data; A program code for estimating an analysis result including an evaluation value obtained by quantitatively evaluating an evaluation item, and a program code for transmitting the analysis result to the terminal device.

In the system, the trained model, the method, the terminal device, the information processing device, and the program, the prosodic feature amount includes a voice feature amount of the subject giving the presentation, and the linguistic feature amount may include feature amounts of utterance content of the subject who is giving the presentation, and the motion feature amounts may include feature amounts of gestures of the subject who is giving the presentation.
In addition, in the system, the trained model, the method, the terminal device, the information processing device, and the program, the data to be acquired is measured by an infrared sensor, a heartbeat sensor, or the like of the target person who is giving the presentation. Detection data detected by various sensors may be included, and the feature amount used for the analysis may include body temperature, heart rate, etc. extracted from the detection data. Further, in the system, the trained model, the method, the terminal device, the information processing device, and the program, analysis results including past evaluation values and the latest evaluation values are to be output or transmitted. It may include the analysis results including the evaluation values at the same time, or the difference between the analysis results including the past evaluation values and the latest evaluation values.

According to the present invention, it is possible to quantitatively evaluate the presentation from multiple viewpoints based on the multimodal information of the presentation, and the hurdles for implementation are low.

FIG. 4 is an explanatory diagram showing an example of an overview of presentation evaluation in the system according to the embodiment; FIG. 4 is an explanatory diagram showing an example of a presentation evaluation procedure according to the embodiment; 1 is a block diagram showing an example of schematic configurations of a terminal device and an information processing device in a system according to an embodiment; FIG. FIG. 2 is a block diagram showing an example of a main configuration of a feature amount extraction unit of the information processing apparatus according to the embodiment; figure. 4A is an explanatory diagram showing an example of a skeleton detection image used for extraction processing of motion amounts of various parts of a presenter by a feature amount extraction unit according to the embodiment; FIG. (b) is an explanatory diagram showing an example of temporal change (trajectory) of the position of the detection point of the presenter's head. FIG. 5 is an explanatory diagram showing an example of a display screen of analysis results in the terminal device according to the embodiment;

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is an explanatory diagram showing an example of an overview of presentation evaluation in the system according to this embodiment. The presentation evaluation of this embodiment is based on multimodal feature amounts including prosodic feature amounts, language feature amounts, and action feature amounts extracted from audio and video data when the subject of evaluation is giving a presentation. Quantitative evaluation values are calculated from multiple viewpoints. The system of this embodiment may be used as a presentation evaluation system, or may be used as a presentation ability estimation system.

Presentations to be evaluated are presentations of plans, proposals, estimates, lecture contents, self-appeal, etc., using visual and auditory means, at meetings, meetings, lectures, meetings, interviews, etc. Presented or presented by a person to other attendees.

In the example of presentation evaluation shown in FIG. is input to the machine-learned model as the analysis model as the value of The machine-learned model analyzes the input multimodal feature amount according to a predetermined algorithm, and outputs a presentation evaluation value obtained by quantitatively evaluating a predetermined evaluation item of the presentation as the value of the objective variable.

The presentation evaluation value is the value of a plurality of evaluation items relating to each of the large items of purpose coverage, content logic (the logic of content), visuals and vocals (how they look and hear), and effective production elements. For example, the objective coverage evaluation item is an item for evaluating whether or not the three contents related to the objective of "who", "what", and "how do you want me to do it" are covered. The content logic evaluation item is an item for evaluating whether or not three contents related to, for example, "conclusion", its "basis", and "interest of the other party" are included. Visual and vocal evaluation items are, for example, evaluation items related to appearance and hearing, such as intonation, volume, eye contact, and gesture. The effective production elements are evaluation items related to effective production such as emphasis, repetition, concrete expression, and interactivity.

The analysis model used for presentation evaluation is, for example, for each of a plurality of presentations, the prosodic feature amount, language feature amount, and action feature amount extracted from the audio and video data of the presentation, and the machine for each presentation. By performing machine learning in advance using multiple supervised learning data that associates evaluation values as correct data given by human evaluators for each of the above evaluation items, parameters such as weights for each feature value in the model It is a machine-learned model created by determining the value of

A plurality of types of machine-learned models may be created according to the individuality of the presenter (presenter) 10 or the individuality of the user 40 who is the subject of evaluation. The machine-learned model may collectively output the plurality of evaluation items, or may be provided for each evaluation item in order to increase the accuracy of the evaluation values output for each of the plurality of evaluation items. In addition, a plurality of types of machine-learned models are created in advance so as to correspond to a plurality of types of target person groups with different attributes of the presenter 10 (for example, age group, gender, presentation experience). All of the plurality of types of machine-learned models may be used, or a plurality of types of machine-learned models may be selected and used. Also, a plurality of types of machine-learned models may be created in advance so as to correspond to each of a plurality of types of presentations, and all of the plurality of types of machine-learned models may be used. It is also possible to select and use from the machine-learned models of The machine-learned model may be selected manually by the user, or based on at least one of the attribute of the presenter, the type of presentation, the language feature, the prosodic feature, and the action feature. good too. Further, calibration may be performed for each of a plurality of types of machine-learned models, and selection may be made based on the calibration information. Here, “calibration” is processing for bringing the prediction probability calculated by the trained model closer to the original probability. Alternatively, analysis may be performed using all of a plurality of types of machine-learned models, and the machine-learned model with the highest accuracy may be selected and used for subsequent analysis.

In addition, the analysis model (machine-learned model) used to evaluate the presentation is patterned and extracted from the voice data and video data among the multiple types of features included in each of the prosodic features, language features, and action features. An analysis model that does not use, as an input, a feature amount that requires a predetermined time or more for processing or a feature amount that is difficult to pattern (for example, the number of fillers described later) may be used. Here, for the feature quantity that requires more than a predetermined time for the extraction process and the feature quantity that is difficult to pattern, for example, since the pattern is diverse, the feature quantity using the existing service does not have a sufficient level for analysis. It includes feature values that cannot be obtained, and feature values that cannot be processed in real time because manual work is required to meet the standard.

FIG. 2 is an explanatory diagram showing an example of the presentation evaluation procedure according to the present embodiment. In FIG. 2 , when using the presentation evaluation system according to the embodiment, the user operates the terminal device 20 to start a presentation evaluation application program preinstalled in the terminal device 20 . When the user performs a data acquisition start operation, the microphone and camera of the terminal device 20 are used to input the voice of the presentation given by the presenter 10, who is the target person, and to capture the video. is obtained (step S1). The data to be acquired may include detection data detected by various sensors such as an infrared sensor and a heart rate sensor, which are measured by the presenter 10 who is giving a presentation.

Next, when the presentation is over and the user performs a data acquisition end operation, the audio data and video data of the presentation are transmitted from the terminal device 20 via the mobile communication network to the information processing device (cloud service) built on the communication network. platform) 30 (step S2).

The information processing device (cloud service platform) 30 performs transcription processing for converting voice data received from the terminal device 20 into text data, extracts language features, extracts presentation prosodic features from the voice data, Since it is received from the terminal device 20, the motion feature amount of the presenter 10, who is the target person during the presentation, is extracted (step S3). Furthermore, the information processing device 30 analyzes the extracted linguistic feature amount, prosodic feature amount, and action feature amount, and analyzes the above-mentioned target coverage, content logic (content logicality), visual and vocal (appearance and hearing). method), and an analysis result including an evaluation value quantitatively evaluated for each evaluation item of the effective production element is estimated (step S4), and the analysis result is transmitted to the terminal device 20 via the mobile communication network. (Step S5). The prosodic feature amount includes the feature amount of the voice of the presenter 10 who is giving a presentation, the language feature amount includes the feature amount of the utterance content of the presenter 10 who is giving the presentation, and the action feature amount is: Gesture features of the presenter 10 who is giving a presentation may also be included. Moreover, the feature amount used for the analysis may include body temperature, heart rate, etc. extracted from the detection data of the various sensors.

When the terminal device 20 receives the analysis result of the presentation from the information processing device 30, the terminal device 20 displays the analysis result on its own screen (step S6).

FIG. 3 is a block diagram showing an example of schematic configurations of the terminal device 20 and the information processing device 30 in the system according to this embodiment. In the example of FIG. 3 and FIG. 2 described above, the information processing device 30 is a cloud platform built on a communication network, but the information processing device 30 may be a server consisting of one or more computer devices. good. In the example of FIG. 3, the user of the terminal device 20 of this system is the presenter of the presentation, but the user of the terminal device 20 may be someone other than the presenter of the presentation. .

3, the terminal device 20 includes a data acquisition unit 201, a data confirmation unit 202, a data acquisition advice display unit (data acquisition advice output unit) 203, a data transmission unit 204, an analysis result reception unit 205, and an analysis result display unit (analysis result output unit) 206. The functions of data processing and signal processing in each part of the terminal device 20 are realized by executing a predetermined application program in a computer or processor provided in the terminal device 20, for example.

The data acquisition unit 201 captures the state of the presenter 10 giving a presentation with a camera and converts it into moving image data, and also acquires the voice of the presenter 10 with a microphone into voice data. Video data and audio data may be temporarily stored in memory.

The data confirmation unit 202 confirms whether or not the quality of the moving image data and the audio data acquired by the data acquisition unit 201 poses no problem in subsequent analysis. For example, the data confirmation unit 202 confirms whether the value of the sound pressure of the audio data is within a specified range and whether the volume of the ambient noise is within a predetermined threshold for the video data. , whether all the body parts of the presenter 10 whose coordinates are to be obtained for extracting the motion feature amount are included in the image, and whether the angle of view of the moving image capturing direction with respect to the front direction of the presenter 10 is within a predetermined angle range. to confirm that Here, the predetermined angle range is an angle range (for example, an angle range of ±30 degrees) in which motion feature amounts can be extracted in subsequent processing. In addition, the data confirmation unit 202 asks the presenter 10 to read out a specific sentence, and if the speech is correctly recognized when the speech is transcribed, the speech data does not cause any problem in the subsequent analysis. quality”. Further, if the presenter 10 is asked to perform a specific action and the specific skeleton information is recognized, the data confirmation unit 202 determines that the moving image data has an "analyzable quality" that poses no problem for subsequent analysis. You can judge. For example, it may be determined that the moving image data has “analyzable quality” if the user is asked to stretch his/her elbows and raise both hands straight up, and if the raised hands are recognized.

If there is a problem with the quality of the moving image data and the audio data (in the case of NG (negative result) in the figure), the data confirmation section 202 sends the information to the data acquisition advice display section 203 . Based on the information received from the data confirmation unit 202, the data acquisition advice display unit 203 displays an advice message on the display of the terminal device 20, such as raising the voice or changing the presentation position. Note that the advisory message may be output by voice in addition to or instead of being displayed.

On the other hand, if there is no problem with the quality (in the case of OK (positive result) in the drawing), the data confirmation section 202 sends the moving image data and the audio data to the data transmission section 204 . The data transmission unit 204 transmits moving image data and audio data to the information processing device (cloud service platform) 30 via a communication network such as a mobile communication network, for example, using a wireless communication device or the like.

Note that the data transmission unit 204 may transmit information about the type of corresponding presentation and information about the attributes of the presenter to the information processing device 30 together with the moving image data and the audio data. In addition, the data transmission unit 204 may transmit a data group ID for identifying the moving image data and the audio data, or a presentation ID for identifying the presentation corresponding to the data.

The analysis result receiving unit 205 receives the analysis result including the evaluation value quantitatively evaluated for the predetermined evaluation item of the presentation from the information processing device 30 via a communication network such as a mobile communication network, for example, by using a wireless communication device or the like. do.

The analysis result display unit 206 displays the analysis result of the presentation received from the information processing device 30 on the display of the terminal device 20 . The analysis result of the presentation may be output by voice in addition to or instead of display, as illustrated in FIG. 6, for example.

The information processing device (cloud service platform) 30 includes a data reception unit 301 , a feature amount extraction unit 302 , an analysis model determination unit 303 , an inference unit 304 , an analysis model database (DB) 305 and an analysis result transmission unit 306 . The functions of data processing and signal processing in each part of the information processing device 30 are realized by executing a predetermined program in one or more computers or processors provided in the information processing device 30, for example.

The data receiving unit 301 receives moving image data and audio data from the terminal device 20 via a communication network such as a mobile communication network, for example, using a wireless communication device or the like. The data receiving unit 301 may receive from the terminal device 20 information about the type of corresponding presentation and information about the attributes of the presenter, together with the moving image data and the audio data.

The feature amount extraction unit 302 extracts various feature amounts to be input as explanatory variables to an analysis model (machine-learned model), which will be described later, from the video data and audio data received from the terminal device 20 . For example, the feature amount extraction unit 302 illustrated in FIG. 4 includes a speech analysis unit 321, a language analysis unit 322, and a motion analysis unit 323. A transcription unit 3211 of the voice analysis unit 321 performs a transcription process of converting voice data received from the terminal device 20 into text data by voice recognition.

The linguistic analysis unit 322 extracts the linguistic features of the presentation shown in Table 1 from the text data obtained by the transcription unit 3211 . Fillers in Table 1 are words or sounds used to fill gaps between words such as "uh", "uh", "ha", "uh". The verb repetition count is the maximum number of repetitions of the same verb in the presentation, and the noun repetition count is the maximum number of repetitions of the same noun in the presentation.

Also, the prosody analysis unit 3212 of the speech analysis unit 321 illustrated in FIG. 4 extracts the prosody feature amount of the presentation illustrated in Table 2 from the speech data. Pitch in Table 2 is the pitch (frequency) of the voice. Intensity is the physical strength of sound, for example sound power transmitted through a unit area [W/m ² ]. Also, the total speech length is the total value of the speech time in the entire presentation, and the total speech length (one second or longer) is the total value of the speech time of one second or longer. Also, the total silent length is the total value of silent time in the entire presentation, and the total silent length (one second or longer) is the total value of one second or longer of silent time. The speech ratio is the ratio of the total speech time to the entire presentation time, and the speech ratio (one second or more) is the ratio of the total speech time of one second or more to the entire presentation time.

Further, the motion analysis unit 323 illustrated in FIG. 4 extracts motion feature amounts illustrated in Table 3 by analyzing the moving image data. The motion amount of each part of the body in Table 3 is calculated, for example, as follows. As shown in FIG. 5(a), detection points (joint points) 101 to 119 of a skeleton detection image 100 in a moving image of a presenter 10 are detected, for example, as shown in FIG. 5(b). A time variation (trajectory) of the dimensional position coordinates (X, Z) is calculated. The average and standard deviation of the amount of movement, which is the amount of change in position coordinates in units of frames at all the detection points 101 to 119, are the average and standard deviation of the amount of movement of each part of the body in Table 3. In addition, the average and standard deviation of the amount of movement of each part of the body during speech in Table 3 are the amounts of movement at detection points 101 to 119 of the skeleton detection image 100 of the presenter calculated for the time period during which the presenter is speaking. is the mean and standard deviation of

In addition to or instead of the average and standard deviation of the amount of motion, the average and standard changes of the velocity, acceleration, or both of the detection points 101 to 119 of the skeleton detection image 100 of the presenter are used as the motion feature amount. may

Based on the prosodic feature amount, language feature amount, and action feature amount extracted by the feature amount extraction unit 302, the analysis model determination unit 303 analyzes these feature amounts and uses them for quantitative evaluation of presentation evaluation items. Determine the analytical model. For example, the analysis model determination unit 303 determines the type of the presentation to be evaluated and the type of the presenter based on the prosodic feature amount, the language feature amount, and the action feature amount. A suitable analysis model is selected and determined from a plurality of types of pre-registered analysis models, and an analysis model ID for identifying the determined analysis model is output to the feature quantity extraction unit 302 . Further, the analysis model determination unit 303 selects and determines all of the plurality of types of analysis models registered in advance, and outputs a plurality of analysis model IDs for identifying each of the determined plurality of types of analysis models to the feature amount extraction unit 302. You may

In addition, for the selection/decision of the analysis model, at least one of the information of the attributes of the presenter (for example, age group, gender, experience of presentation) received from the terminal device 20 and the information of the type of presentation. information may be used.

Based on one or more analysis model IDs received from the feature amount extraction unit 302, the inference unit 304 selects from a plurality of analysis models stored in the analysis model DB 305 to quantitatively evaluate the evaluation items of the presentation. Select an analysis model (estimation program and learned parameter values used for it) to be used.

The analysis model is a machine-learned model created by determining parameter values such as weights for each feature quantity in the model by carrying out machine learning in advance using a plurality of supervised learning data as described above. The machine-learned model used in the inference unit 304 may collectively output the plurality of evaluation items, or for each evaluation item in order to increase the accuracy of the evaluation value output for each of the plurality of evaluation items. may be provided.

The algorithm used for the machine-learned model of this embodiment is not limited to a specific algorithm. For example, as a machine-learned model algorithm that learns using supervised learning data, it is possible to use SVR (support vector regression), which is classified as "regression" that learns numerical data and predicts numerical values. can. Instead of this SVR, Linear (Ordinary) Regression, Bayesian Linear Regression, Random (Decision) Forest, Boosted decision tree, Fast forest distribution quantile), neural network, Poisson regression, support vector ordinal regression, ridge regression, lasso regression, etc. may be used.

The inference unit 304 adds the linguistic feature quantity received from the feature quantity extraction unit 302 to one or more analysis models (estimation program and machine-learned model including learned parameter values used therein) selected based on the analysis model ID , the prosodic feature amount and the action feature amount are input, and an evaluation value obtained by quantitatively evaluating a predetermined evaluation item is output. For example, the inference unit 304 outputs quantitative evaluation values in three stages (1 to 3) for each of the 14 types of evaluation items shown in Table 4.

The analysis result output by the inference unit 304 is used as a qualitative evaluation to be fed back (FB) to the presenter, such as one-word comments (feedback comments) on the entire presentation, such as the following examples (1) to (4). may include
(1) In particular, "emphasis" has been fully demonstrated.
(2) Please be conscious of "emphasis" in particular.
(3) There is no eye contact because the line of sight is in the air. Occasionally gestures come out naturally, but there is a habit of swaying left and right. A smile is always present.
(4) The voice is loud and easy to hear. There is an occasional intonation, but there is no pause. Gestures come out naturally, but sometimes gestures that don't match the content of the talk appear.

The one-word comment (feedback comment) can be generated, for example, as shown in Table 5, based on the evaluation value of the evaluation item and the coefficient of determination estimated by the aforementioned analysis model such as SVM. Here, the coefficient of determination is a value indicating the accuracy of estimation of an evaluation value by an analysis model such as SVM, and takes a value from -1 to +1. For example, when the absolute value of this coefficient of determination is less than 0.2, the accuracy of estimation of the evaluation value is low, and when the absolute value of the determination coefficient is 0.2 or more and 1 or less, the accuracy of estimation of the evaluation value is sufficient. can be judged to be high.

Table 5 is an example of generating a one-word comment (feedback comment) based on the evaluation value and coefficient of determination of "emphasis" of the above-described effective effect element. For example, if the evaluation value of the evaluation item “emphasis” in Table 5 is 1 and the absolute value of the coefficient of determination is 0.2 or more and the estimation accuracy is sufficiently high, “especially “emphasis” I have been able to fully demonstrate it. ] is generated. In addition, if the evaluation value of the evaluation item "emphasis" is 0 and the absolute value of the coefficient of determination is 0.2 or more, and it is judged that the estimation accuracy is sufficiently high, "especially "emphasis" should be considered. please ] is generated. If it is determined that the absolute value of the coefficient of determination is smaller than 0.2 and the estimation accuracy is low, no brief comment is generated.

Here, the inference unit 304 selects “especially 'evaluation item name' is fully demonstrated” as a short comment candidate. and "Please be aware of the 'evaluation item name' in particular. ] is stored, and evaluation item names such as “emphasis” and “repetition” are put in the part inside “ ” to generate a one-word comment.

The analysis model database (DB) 305 assigns different analysis model IDs to each other and stores a plurality of types of analysis models as described above. The analytical model is a machine-learned model consisting of an estimation program and learned parameter values used therein. For example, multiple types of machine-learned models each have a common input and output format, and quantitative evaluation items for presentations created in advance for multiple combinations of multiple types of presentations and multiple types of presenters. It is an analytical model suitable for evaluation. A plurality of types of machine-learned models may be created according to the personality of the presenter 10 or the personality of the user 40 . In addition, as described above, the machine-learned model may output multiple evaluation items at once. may be set to

The analysis result transmission unit 306 transmits the analysis result including the evaluation value output from the inference unit 304 to the terminal device 20 via a communication network such as a mobile communication network. The analysis result transmission unit 306 may transmit the aforementioned data group ID or presentation ID corresponding to the analysis result together with the analysis result.

FIG. 6 is an explanatory diagram showing an example of the analysis result display screen 21 in the terminal device 20 according to the present embodiment. The display screen 21 of the terminal device 20 illustrated in FIG. 6 has a quantitative evaluation display section 211 and a qualitative evaluation display section 212 . In the quantitative evaluation display section 211, quantitative evaluation values for each of the 14 types of evaluation items in Table 4 are displayed in numbers and graphs. The qualitative evaluation display section 212 displays a brief comment fed back (FB) from the information processing apparatus 30 described above. The display screen 21 may display the data of the prosodic feature amount, the language feature amount, and the action feature amount, which are the basis of the analysis result including the evaluation value, and the information of the analysis model (machine-learned model).

As described above, according to the present embodiment, analysis results including evaluation values obtained by analyzing multimodal feature amounts such as the linguistic feature amount, the prosodic feature amount, and the action feature amount of the presentation and quantitatively evaluating the predetermined evaluation items of the presentation can be estimated and output, it is possible to quantitatively evaluate presentations from multiple perspectives. Moreover, the linguistic feature amount, prosodic feature amount, and action feature amount used for evaluation of the presentation can be extracted from audio data and video data, and no special hardware such as a viewing direction detection device is required. The hurdles for implementation in terminal devices used by users are low.

Further, according to the present embodiment, by using only voice data and moving image data having a predetermined quality for extraction of the linguistic feature amount, the prosodic feature amount, and the motion feature amount, it is possible to improve the extraction accuracy of each feature amount. . In particular, speech data with a predetermined sound pressure and noise below a predetermined level is used to extract the linguistic feature amount and the prosody feature amount, thereby increasing the extraction accuracy of the linguistic feature amount and the prosodic feature amount, and using it for the extraction of the action feature amount. By using the moving image data that includes the body part of the subject whose coordinates are to be acquired and has an angle of view within a predetermined angle range for extracting the motion feature amount, the extraction accuracy of the motion feature amount can be improved. Presentations that can be evaluated by this system or presentations that are suitable for such evaluation by displaying (outputting) an advisory message regarding the acquisition of audio data and video data when the audio data and video data do not have the specified quality. Presenters can be reminded to do

Further, according to the present embodiment, by using an analysis model that outputs an analysis result including a quantitative evaluation value by processing an input including a linguistic feature amount, a prosody feature amount, and an action feature amount with a predetermined algorithm, , a linguistic feature amount, a prosody feature amount, and a motion feature amount.

Further, according to the present embodiment, each of the linguistic feature amount, the prosodic feature amount, and the action feature amount includes a plurality of types of feature amounts, so that more multifaceted evaluation of the presentation becomes possible. In addition, by using an analysis model that does not use as input feature amounts that require a predetermined amount of time or more to extract patterns from audio data and video data among multiple types of feature amounts, the quantitative evaluation value of the presentation can be obtained. Real-time feedback from the information processing device 30 to the terminal device 20 of the included analysis result is enabled.

Further, according to the present embodiment, an analysis model selected from a plurality of types of analysis models with mutually different algorithms is used for analysis of the prosodic feature amount, the language feature amount, and the motion feature amount, so that the , which enables a highly accurate evaluation of the presentation. In particular, by using an analysis model selected based on at least one of the linguistic feature amount, the prosodic feature amount, and the action feature amount, highly accurate evaluation according to the feature amount becomes possible. Also, by using an analysis model selected based on at least one of the attributes of the presenter (target person) of the presentation and the type of presentation, highly accurate evaluation according to the attributes of the presenter and the type of presentation is possible. . In addition, by using a trained model created by machine learning using supervised learning data including linguistic feature values, prosodic feature values, motion feature values, and correct evaluation value data obtained in advance for multiple presentations , input the linguistic feature amount, the prosodic feature amount, and the action feature amount of the presentation to be evaluated, and output a quantitative evaluation of the presentation from multiple viewpoints.

In particular, according to this embodiment, evaluation values evaluated from the viewpoint of linguistic feature amounts regarding at least one of the number of fillers, the number of nouns, the number of verbs, the number of interjections, the number of verb repetitions, and the number of noun repetitions in the full sentence of the presentation are output. be able to. In addition, it is possible to output an evaluation value evaluated from the viewpoint of prosodic features regarding at least one of pitch, intensity, sound pressure, intonation, speech speed, speech length, silence length, and speech ratio in speech of a presentation. Also, it is possible to output an evaluation value evaluated from the viewpoint of the motion feature amount regarding at least one of the amount of motion of each part of the presenter's body during the entire presentation and the amount of motion of each part of the presenter's body during speech.

In addition, according to the present embodiment, it is possible to output a multifaceted evaluation that includes coverage of the purpose of the presentation, logicality of the content, how it looks and sounds, and effective production elements.

In addition, according to the present embodiment, by including in the analysis result a brief comment generated based on the evaluation value and the determination coefficient when estimating the evaluation value, the presenter of the presentation or the user of the terminal device 20 can intuitively Able to give clear and concise evaluations.

Further, according to the present embodiment, with a simple operation of acquiring the audio data and video data of the presentation given by the presenter 10, who is the target person, with the terminal device 20, quantitative data can be obtained from multiple viewpoints of the presentation. It is possible to output the analysis result including the evaluation value to the terminal device 20 .

It should be noted that the processing steps and components of the presentation evaluation system, terminal device, and information processing device described herein can be implemented by various means. For example, these processes and components may be implemented in hardware, firmware, software, or any combination thereof.

For hardware implementation, the above steps and components in an entity (e.g., computer device, server, cloud service platform (cloud computer system), various wireless communication devices, Node B, terminal, hard disk drive device, or optical disk drive device) The means such as processing units used to implement the ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, computers, or , may be implemented in combinations thereof.

Also, for firmware and/or software implementations, means such as processing units used to implement the above components may be programs (e.g., procedures, functions, modules, instructions) that perform the functions described herein. , etc.). In general, any computer/processor readable medium tangibly embodying firmware and/or software code means, such as a processing unit, used to implement the above steps and components described herein. may be used to implement For example, firmware and/or software code may be stored in memory and executed by a computer or processor, such as in a controller. The memory may be implemented within the computer or processor, or external to the processor. The firmware and/or software code may also be, for example, random access memory (RAM), read only memory (ROM), non-volatile random access memory (NVRAM), programmable read only memory (PROM), electrically erasable PROM (EEPROM). ), FLASH memory, floppy disk, compact disk (CD), digital versatile disk (DVD), magnetic or optical data storage devices, etc. good. The code may be executed by one or more computers or processors and may cause the computers or processors to perform certain aspects of the functionality described herein.

Also, the medium may be a non-temporary recording medium. Also, the code of the program may be readable and executable by a computer, processor, or other device or apparatus machine, and its format is not limited to a particular format. For example, the program code may be source code, object code, or binary code, or may be a mixture of two or more of these codes.

Moreover, the description of the embodiments disclosed herein is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to this disclosure will be readily apparent to those skilled in the art, and the general principles defined herein are applicable to other variations without departing from the spirit or scope of this disclosure. This disclosure, therefore, is not to be limited to the examples and designs described herein, but is to be accorded the broadest scope consistent with the principles and novel features disclosed herein.

10: Presenter 20: Terminal device 21: Display screen 30: Information processing device 40: User 100: Skeleton detection images 101 to 119: Detection point 201: Data acquisition unit 202: Data confirmation unit 203: Data acquisition advice display unit 204: Data transmission unit 205 : Analysis result reception unit 206 : Analysis result display unit 211 : Quantitative evaluation display unit 212 : Qualitative evaluation display unit 301 : Data reception unit 302 : Feature amount extraction unit 303 : Analysis model determination unit 304 : Inference unit 305: Analysis model DB
306: Analysis result transmission unit 321: Voice analysis unit 322: Language analysis unit 323: Motion analysis unit 3211: Transcription unit 3212: Prosody analysis unit

Claims (20)

A system for evaluating presentations, comprising:
a data acquisition unit that acquires voice data and video data of a target person who is giving a presentation;
a feature amount extracting unit that extracts the linguistic feature amount and the prosodic feature amount of the presentation from the audio data, and extracts the motion feature amount of the subject during the presentation from the moving image data;
analyzing the linguistic feature amount, the prosodic feature amount, and the action feature amount to generate an analysis result including an evaluation value quantitatively evaluated for each of evaluation items regarding a plurality of evaluation contents regarding major items for evaluating the presentation; an inference unit that estimates;
and an analysis result output unit that outputs the analysis result,
The reasoning unit stores a plurality of comment data as comment candidates for the presentation, and generates the plurality of comments based on the evaluation values of the plurality of evaluation items and a coefficient of determination indicating the accuracy of estimation of the evaluation values. A system characterized by selecting one or more pieces of comment data from data and including comments generated using the selected comment data in the analysis result . The system of claim 1, wherein
further comprising a data confirmation unit for confirming whether the audio data and the video data acquired by the data acquisition unit have a predetermined quality;
The audio data and video data having the predetermined quality are used to extract the language feature amount, the prosodic feature amount, and the action feature amount, and the audio data and video data not having the predetermined quality are used to extract the language feature amount. , is not used for extracting the prosody feature amount and the motion feature amount. The system of claim 2, wherein
The quality to be confirmed is that the sound pressure of the audio data is within a predetermined range, that the magnitude of noise in the audio data is equal to or less than a threshold, and that the image in the moving image data is used for extracting the motion feature amount. and that the angle of view of the moving image capturing direction with respect to the front direction of the subject is within a predetermined angle range. , a system characterized by: In the system of claim 2 or 3,
A data acquisition advice output unit for outputting an advice message regarding acquisition of the audio data and the video data when the audio data and the video data acquired by the data acquisition unit do not have a predetermined quality. characterized system. In the system according to any one of claims 1 to 4,
The inference unit uses an analysis model that outputs an analysis result including the quantitative evaluation value by processing an input including the linguistic feature amount, the prosodic feature amount, and the action feature amount with a predetermined algorithm. A system characterized by The system of claim 5, wherein
the linguistic feature amount, the prosodic feature amount, and the action feature amount each include a plurality of types of feature amounts,
The analysis model used in the inference unit is an analysis model that does not use, as the input, a feature amount that requires a predetermined time or more for patterning and extracting the audio data and the moving image data among the plurality of types of feature amounts. A system characterized by: In the system of claim 5 or 6,
The system according to claim 1, wherein the inference unit uses a plurality of types of analysis models with mutually different algorithms for the analysis. In the system of claim 5 or 6,
The system according to claim 1, wherein the inference unit has a plurality of types of analysis models with mutually different algorithms, and uses an analysis model selected from the plurality of types of analysis models for the analysis. 9. The system of claim 8, wherein
The inference unit selects an analysis model to be used for the analysis from the plurality of types of analysis models based on at least one of the linguistic feature amount, the prosodic feature amount, and the action feature amount. system. The system of claim 8 or 9,
The system, wherein the inference unit selects an analysis model to be used for the analysis from the plurality of types of analysis models based on at least one of an attribute of a target person of the presentation and a type of the presentation. In the system according to any one of claims 5 to 10,
The analysis model is a learning created by machine learning using supervised learning data including the linguistic feature amount, the prosodic feature amount, the action feature amount, and the correct answer data of the evaluation value acquired for a plurality of presentations. A system characterized in that it is a finished model. The system according to any one of claims 1 to 11,
The linguistic feature amount includes a feature amount related to at least one of the number of fillers, the number of nouns, the number of verbs, the interjection, the number of verb repetitions, and the number of noun repetitions in the full sentence of the presentation,
The prosodic feature amount includes a feature amount related to at least one of pitch, intensity, sound pressure, intonation, speech speed, speech length, silence length, and speech ratio in speech of the presentation,
A system, wherein the motion feature quantity includes a feature quantity relating to at least one of motion amounts of each part of the body of the subject during the entire presentation and motion of each part of the body of the subject during speech. In the system of any one of claims 1-12,
A system according to claim 1, wherein major items for evaluating said presentation include coverage of purpose of said presentation, logicality of content, how it looks and sounds, and effective production elements . In the system according to any one of claims 1 to 13,
Equipped with a terminal device and an information processing device that can communicate with each other via a communication network,
The terminal device includes the data acquisition unit, a data transmission unit that transmits the audio data and the video data to the information processing device, an analysis result reception unit that receives the analysis result from the information processing device, and the analysis a result output unit;
The information processing device includes a data receiving unit that receives the audio data and the video data from the terminal device, the feature extraction unit, the inference unit, and an analysis result transmission that transmits the analysis result to the terminal device. and a system. A trained model program for operating a computer or processor to evaluate a presentation,
Verbal features, prosodic features, and motion features extracted based on the subject's voice data and video data acquired for a plurality of presentations, and correct data of evaluation values quantitatively evaluated for predetermined evaluation items of the presentations Created by machine learning using supervised learning data including
When there is an input including language features, prosodic features, and motion features extracted based on the subject's voice data and video data acquired for the presentation to be evaluated, a plurality of major items for evaluating the presentation A program code for outputting an analysis result including an evaluation value quantitatively evaluated for each evaluation item regarding the evaluation content ;
storing a plurality of comment data as comment candidates for the presentation, and selecting one or more of the plurality of comment data based on the evaluation values of the plurality of evaluation items and a coefficient of determination indicating the accuracy of estimation of the evaluation value; and a program code for selecting comment data from and including comments generated using the selected comment data in the analysis result . A method of evaluating a presentation, comprising:
Acquiring audio data and video data of a subject who is giving a presentation;
extracting the language feature amount and the prosodic feature amount of the presentation from the audio data, and extracting the motion feature amount of the subject during the presentation from the moving image data;
analyzing the linguistic feature amount, the prosodic feature amount, and the action feature amount to generate an analysis result including an evaluation value quantitatively evaluated for each of evaluation items regarding a plurality of evaluation contents regarding major items for evaluating the presentation; estimating;
outputting the analysis result;
storing a plurality of comment data as comment candidates for the presentation, and selecting one or more of the plurality of comment data based on the evaluation values of the plurality of evaluation items and a coefficient of determination indicating the accuracy of estimation of the evaluation value; and including comments generated using the selected comment data in the analysis results . A terminal device capable of communicating with an information processing device via a communication network,
a data acquisition unit that acquires voice data and video data of a target person who is giving a presentation;
a data transmission unit that transmits the audio data and the video data to the information processing device;
Analysis for receiving, from the information processing device, an analysis result including an evaluation value quantitatively evaluated for each of evaluation items regarding a plurality of evaluation contents regarding major items for evaluating the presentation based on the audio data and the video data. a result receiving unit;
and an analysis result output unit that outputs the analysis result ,
The analysis result received from the information processing device is such that the information processing device, which stores a plurality of comment data as comment candidates for the presentation, compares the evaluation values of the plurality of evaluation items and the estimation accuracy of the evaluation values. one or more comment data is selected from the plurality of comment data based on a coefficient of determination shown, and a comment generated by the information processing device using the selected comment data is included. . An information processing device capable of communicating with a terminal device via a communication network,
a data receiving unit that receives audio data and video data of a target person who is giving a presentation from the terminal device;
a feature amount extracting unit that extracts the linguistic feature amount and the prosodic feature amount of the presentation from the audio data, and extracts the motion feature amount of the subject during the presentation from the moving image data;
analyzing the linguistic feature amount, the prosodic feature amount, and the action feature amount to generate an analysis result including an evaluation value quantitatively evaluated for each of evaluation items regarding a plurality of evaluation contents regarding major items for evaluating the presentation; an inference unit that estimates;
an analysis result transmission unit that transmits the analysis result to the terminal device ;
The reasoning unit stores a plurality of comment data as comment candidates for the presentation, and generates the plurality of comments based on the evaluation values of the plurality of evaluation items and a coefficient of determination indicating the accuracy of estimation of the evaluation values. An information processing apparatus , comprising: selecting one or a plurality of pieces of comment data from data; and including a comment generated using the selected comment data in the analysis result . A program executed in a computer or processor provided in a terminal device capable of communicating with an information processing device via a communication network,
A program code for acquiring audio data and video data of a target person who is giving a presentation;
a program code for transmitting the audio data and the video data to the information processing device;
To receive, from the information processing apparatus, an analysis result including an evaluation value quantitatively evaluated for each of evaluation items regarding a plurality of evaluation contents regarding major items for evaluating the presentation based on the audio data and the video data. program code of
a program code for outputting the analysis result ,
The analysis result received from the information processing device is such that the information processing device, which stores a plurality of comment data as comment candidates for the presentation, compares the evaluation values of the plurality of evaluation items and the estimation accuracy of the evaluation values. A program comprising: selecting one or a plurality of comment data from the plurality of comment data based on a coefficient of determination shown, and including a comment generated by the information processing device using the selected comment data . A program executed in a computer or processor provided in an information processing device capable of communicating with a terminal device via a communication network,
a program code for receiving audio data and video data of a target person who is giving a presentation from the terminal device;
a program code for extracting the linguistic feature amount and the prosody feature amount of the presentation from the audio data, and extracting the motion feature amount of the subject during the presentation from the moving image data;
analyzing the linguistic feature amount, the prosodic feature amount, and the action feature amount to generate an analysis result including an evaluation value quantitatively evaluated for each of evaluation items regarding a plurality of evaluation contents regarding major items for evaluating the presentation; program code for estimating;
a program code for transmitting the analysis result to the terminal device;
storing a plurality of comment data as comment candidates for the presentation, and selecting one or more of the plurality of comment data based on the evaluation values of the plurality of evaluation items and a coefficient of determination indicating the accuracy of estimation of the evaluation value; and program code for selecting comment data from the above and including comments generated using the selected comment data in the analysis result .

Images