JP4774825B2 - Performance evaluation apparatus and method - Google Patents

Description

  The present invention relates to a performance evaluation apparatus and method for evaluating a user performance with respect to a performance required by a system.

  In recent years, game devices that move various application software have been released, but many of them use general-purpose controllers such as joysticks and game pads, and control games with simple modals such as buttons and sticks. Yes.

  On the other hand, for example, a game device that has a camera and controls a game using the input image (for example, Eye Toy (TM): Play of Sony Computer Entertainment Inc. published in Non-Patent Document 1) ) And a game device (for example, OPERATORS SIDE (Sony Computer Entertainment Co., Ltd., published in Non-Patent Document 2)) that has a microphone and controls the game using its input sound. Also in these game devices, the game is controlled in a single modal, such as a motion area in an image and a text as audio content.

http://www.playstation.jp/land/soft/pickup/eyetoy_play.html http://www.playstation.jp/scej/title/operatorsside/01.html

  However, a game device that controls a game in a single modal can only supply a game with a simple configuration content to a user, for example, a user performs a performance and advances a story by the evaluation. The game having the configuration contents could not be supplied to the user.

  The present invention has been proposed in view of such a conventional situation, and provides a performance evaluation apparatus and method that realizes evaluating a user's performance with respect to a performance required by the system and providing the evaluation result. For the purpose.

In order to achieve the above-described object, the performance evaluation device according to the present invention is a performance evaluation device that evaluates a user's performance in a predetermined scene , and a recognition unit that recognizes the user's performance through a plurality of modals, The performance evaluation value for each modal is calculated using the recognition result data for each modal obtained by the recognition means and the teacher data for each modal, and all the performance evaluation values for each modal are calculated. And a performance evaluation means for evaluating the performance of the user in the current scene by calculating a comprehensive evaluation value for comprehensively evaluating the user's performance using the performance evaluation value and a reference value for the performance evaluation value Comparison means for comparing the first evaluation value and the second evaluation value as a reference value of the comprehensive evaluation value and the comprehensive evaluation value If it is equal to or greater than the second threshold, the user's performance content in the next scene is notified to the user, and if the overall evaluation value is less than the second threshold, the performance evaluation value is Informing means for informing the user of the current scene performance correction point for a modal that is less than the first threshold .

In order to achieve the above-described object, the performance evaluation method according to the present invention is a performance evaluation method for evaluating a user's performance in a predetermined scene, and a recognition step of recognizing the user's performance through a plurality of modals. And calculating the performance evaluation value for each modal using the recognition result data for each modal obtained in the recognition step and the teacher data for each modal, and the performance evaluation value for each modal. A performance evaluation step for calculating a comprehensive evaluation value for comprehensively evaluating the user's performance using all of the above and evaluating the performance of the user in the current scene, and the performance evaluation value and a reference value for the performance evaluation value A comparison step of comparing the first evaluation threshold value and the second evaluation threshold value as a reference value of the total evaluation value, and the total evaluation value If it is equal to or greater than the second threshold, the user's performance content in the next scene is notified to the user, and if the overall evaluation value is less than the second threshold, the performance evaluation value is And a notifying step of notifying the user of the correction point of the current scene performance regarding the modal that is less than the first threshold .

  According to the performance evaluation apparatus and method according to the present invention, a user's performance can be recognized through a plurality of modals. Moreover, a user's performance can be evaluated based on the recognition result of each modal.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings.

  First, the structure of the performance evaluation apparatus in this Embodiment is demonstrated.

  As shown in FIG. 1, the performance evaluation apparatus 1 according to the present embodiment includes a camera image input device 10, a microphone sound input device 11, a face recognition device 12, a line-of-sight recognition device 13, and a facial expression recognition device 14. The hand gesture recognizer 15, the speech recognizer 16, the prosody recognizer 17, the performance evaluator 18, the scenario controller 19, the agent controller 20, and the agent 21 are included.

  The camera image input device 10 inputs an image of a user who performs an action, and supplies the input image to the face recognizer 12 and the hand gesture recognizer 15.

  The microphone voice input device 11 inputs the voice of the user performing the performance and supplies the input voice to the voice recognizer 16 and the prosody recognizer 17.

  When the image is supplied from the camera image input device 10, the face recognizer 12 recognizes a face image. Then, the center position (x, y) of the face is detected using the supplied image as an orthogonal coordinate plane in which the horizontal direction is the X axis and the vertical direction is the Y axis. In addition, the face direction is detected at the position, and the detected face direction is expressed using the roll angle, the pitch angle, and the yaw angle. Thereafter, the face recognizer 12 generates data F (t) {x (t), y (t), roll (t), pitch (t), yow indicating the center position and direction of the face in the image at time t. (t)} is supplied to the performance evaluator 18. Further, the face recognizer 12 supplies the face image to the line-of-sight recognizer 13 and the expression recognizer 14. Regarding the technology to detect faces, for example, “Learning a real-time arbitrary posture detector using pixel difference features (Kotaro Sabe, Kenichi Hidai)” (http://face.pdp.crl.sony.co .jp / index.html # mview_face_detection).

  When the face image is supplied from the face recognizer 12, the line-of-sight recognizer 13 detects the line-of-sight direction of the right eye and the line-of-sight direction of the left eye from the face image. The line-of-sight recognizer 13 expresses the detected right-eye line-of-sight direction (θr, φr) and left-eye line-of-sight direction (θl, φl) using a three-dimensional polar coordinate expression. Thereafter, the line-of-sight recognizer 13 supplies data G (t) {θr (t), φr (t), θl (t), φl (t)} indicating the direction of the line of sight at time t to the performance evaluator 18. . For example, Japanese Patent Publication No. 3-51407 discloses a technique for detecting a gaze direction in a predetermined coordinate system based on a face image.

  When the facial image is supplied from the facial recognizer 12, the facial expression recognizer 14 detects 18 feature points 43 of the eyebrows 40, eyes 41, and mouth 42 from the facial image, as shown in FIG. Subsequently, the facial expression recognizer 14 calculates a distance 44 between feature points indicated by 16 arrows of the eyebrows 40, eyes 41, and mouth 42 in order to specify the facial expression of the user. Thereafter, the facial expression recognizer 14 supplies data E (t) {Ei (t)} (i = 0 to 15) indicating the distance 44 between the feature points at time t to the performance evaluator 18. Regarding the technology for recognizing facial expressions using distances between feature points, “Emotion recognition by integrating facial image information and speech information for use in robots, 22nd Annual Conference of the Robotics Society of Japan, 3D14, Sep. 2004 (Shohei Matsumoto, Ken Yamaguchi, Kazunori Komatani, Tetsuya Ogata, Hiroshi Okuno) (http://winnie.kuis.kyoto-u.ac.jp/okuno-lab-bib-j.html) .

  When an image is supplied from the camera image input device 10, the hand shake recognizer 15 sets the center position (x, y) of the hand as an orthogonal coordinate plane in which the horizontal direction is the X axis and the vertical direction is the Y axis. To detect. Further, the rotation angle with the axis passing through the position (x, y) and being perpendicular to the palm as the rotation axis is detected as the hand direction α, and the hand size s and the finger state l ( For example, the number of standing fingers among the five fingers is detected. Thereby, the hand gesture recognizer 15 can specify the state of the hand gesture performed by the user. Thereafter, the hand gesture recognizer 15 uses the data H (t) {x (t), y (t), s (t), α (t), l (t)} indicating the hand shaking state at time t as the performance evaluator. 18 is supplied. Regarding hand gesture recognition technology, “Lars Bretzner, Ivan laptev and Tony Lindberg. Hand Gesture recognition using Multi-Scale Color Features, Hierarchical Models and Particle Filtering. 煤 ihttp: //cg.cs.uni-bonn.de/docs /teaching/2003/SS/cv-seminar/documents/papers/bretzner02hand.pdf).

  When the voice is supplied from the microphone voice input unit 11, the voice recognizer 16 converts the voice into text and supplies it to the performance evaluator 18 as text data T.

  When a voice is input from the microphone voice input unit 11, the prosody recognizer 17 analyzes the voice and detects the height, strength, and speed of the voice. Thereafter, the prosody recognizer 17 supplies the performance evaluator 18 with data PRO (t) {pitch (t), power (t), speed (t)} indicating the prosody at time t.

  The performance evaluator 18 uses the data supplied from each recognizer and the user performance teacher data, which is a model of the user's performance supplied from the scenario controller 19, to calculate using a distance between weighted data. Thus, the performance evaluation value and the overall evaluation value for each modal in the user's performance are calculated. Here, the comprehensive evaluation value is calculated using the performance evaluation values for all the modals, and comprehensively evaluates the user performance and determines the progress of the story. Thereafter, the performance evaluator 18 supplies the performance evaluation value data and the comprehensive evaluation value data to the scenario controller 19 as performance evaluation result data.

  Hereinafter, a method in which the performance evaluator 18 calculates a performance evaluation value for each modal at time t using the weighted data distance will be described.

  The performance evaluator 18 uses the data F (t) supplied from the face recognizer 12 and the user performance teacher data Ft (t) supplied from the scenario controller 19 to calculate the time according to the following equation (1). The performance evaluation value Fscore (t) at t is calculated.

Fscore (t) = Σexp (−Wi | Fi−Fti |) / N (1)
Here, i = 0 to 4, F0 = x, F1 = y, F2 = roll, F3 = pitch, and F4 = yow. Therefore, the number of data N is N = 5. Fti is user acting teacher data for Fi, and Wi is a data weighting factor.

  Further, the performance evaluator 18 uses the data G (t) supplied from the line-of-sight recognizer 13 and the user performance teacher data Gt (t) supplied from the scenario controller 19 according to the following equation (2). The performance evaluation value Gscore (t) is calculated.

Gscore (t) = Σexp (−Wi | Gi−Gti |) / N (2)
Here, i = 0 to 3, G0 = θr, G1 = φr, G2 = θl, and G3 = φl. Therefore, the number of data N is N = 4. Gti is user acting teacher data for Gi, and Wi is a data weighting coefficient.

  The performance evaluator 18 uses the data E (t) supplied from the facial expression recognizer 14 and the user performance teacher data Et (t) supplied from the scenario controller 19 according to the following equation (3). Then, the performance evaluation value Escore (t) is calculated.

Escore (t) = Σexp (−Wi | Ei−Eti |) / N (3)
Here, Ei is data indicating the distance between feature points, and i = 0 to 15 as shown in FIG. Therefore, the number of data N is N = 16. Eti is user acting teacher data for Ei, and Wi is a data weighting coefficient.

  The performance evaluator 18 uses the data H (t) supplied from the hand gesture recognizer 15 and the user performance teacher data Ht (t) supplied from the scenario controller 19 according to the following equation (4). The performance evaluation value Hscore (t) is calculated.

Hscore (t) = Σexp (−Wi | Hi−Hti |) / N (4)
Here, i = 0 to 4, H0 = x, H1 = y, H2 = s, H3 = α, and H4 = l. Therefore, the number of data N is N = 5. Hti is user acting teacher data for Hi, and Wi is a data weighting coefficient.

  Further, the performance evaluator 18 uses the text data T supplied from the speech recognizer 16 and the user teacher data Tt supplied from the scenario controller 19 to perform the performance evaluation value Tscore (t ) Is calculated.

Tscore (t) = exp (−W * Levenshtein distance) (5)
Here, the Levenshtein distance is the minimum value that must be replaced, inserted, or deleted in order to convert the text data T supplied from the speech recognizer 16 into the user acting teacher data Tt supplied from the scenario controller 19. Is the number of characters. W is a weighting factor of data.

  The performance evaluator 18 uses the data PRO (t) supplied from the prosody recognizer 17 and the user performance teacher data PROt (t) supplied from the scenario controller 19 according to the following equation (6). Then, the performance evaluation value PROscore (t) is calculated.

PROscore (t) = Σexp (−Wi | PROi−PROti |) / N (6)
Here, i = 0 to 2, PRO0 = pitch, PRO1 = power, PRO2 = speed. Therefore, the number of data is N = 3. PROti is user acting teacher data for PROi, and Wi is a data weighting factor.

  Thereafter, the performance evaluator 18 calculates a performance evaluation value to be supplied to the scenario controller 19 using the performance evaluation value at time t obtained by the above mathematical formulas (1) to (6).

  Here, the performance evaluator 18 defines a scene so that the period between the start and end of performance evaluation is one scene, and the speech recognition result fits into one scene, and the performance is supplied to the scenario controller 19 every time one scene ends. An evaluation value is calculated.

  Specifically, the performance evaluator 18 integrates the performance evaluation value at time t with the scene time for modals other than voice. For voice, the performance evaluation value at time t is averaged by the number of times data is input to one scene. The value obtained by such calculation is used as the performance evaluation value supplied to the scenario controller 19.

  Further, the performance evaluator 18 uses the average value or the weighted average value of the performance evaluation values for all the modals calculated as described above as the overall evaluation value.

  Thereafter, the performance evaluator 18 supplies the scenario controller 19 with performance evaluation value data and comprehensive evaluation value data for each modal as performance evaluation result data.

  The scenario controller 19 supplies user performance teacher data to the performance evaluator 18 when the performance evaluator 18 performs performance evaluation.

  Further, the scenario controller 19 has a certain threshold value for the performance evaluation value and the comprehensive evaluation value for each modal. When the performance evaluation result data is supplied from the performance evaluator 18, the performance evaluation value is obtained. And the threshold value, and the comprehensive evaluation value and the threshold value are compared.

  In response to this comparison result, the scenario controller 19 supplies agent operation data indicating what operation the agent 21 should perform to the user to the agent controller 20 and performs the performance evaluation of the next user performance teacher data. Supply to the vessel 18.

  When the agent operation data is supplied from the scenario controller 19, the agent controller 20 controls the agent 21 based on the agent operation data.

  The agent 21 operates based on the agent operation data under the control of the agent controller 20.

  Next, the operation in which the performance evaluation apparatus 1 having the above-described configuration evaluates the user's performance will be described separately for the performance practice mode and the stage performance mode.

  The action evaluation apparatus 1 performs an operation for evaluating a user action in the order shown in the flowcharts of FIGS. 3 and 4, for example.

  The operation of the performance evaluation apparatus 1 in the performance practice mode is shown in FIG.

  First, in step S1, the scenario controller 19 supplies the agent controller 20 with agent action data for transmitting the contents of the user's performance in the first scene of the story to the user. The scenario controller 19 supplies user performance teacher data corresponding to the first scene of the story to the performance evaluator 18.

  Next, in step S2, the agent 21 receives the control of the agent controller 20 and informs the user of the performance contents of the first scene as a stage director.

  In step S3, the camera image input unit 10 inputs an image of the user's performance received by the agent 21, and the microphone audio input unit 11 inputs audio, and the input image and audio are recognized as described above. Supply to the vessel.

  Thereafter, in step S4, when each recognizer performs image processing and sound processing, it supplies data of recognition results of each modal to the performance evaluator 18.

  In step S5, the performance evaluator 18 calculates a user performance evaluation value using the data supplied from each recognizer and the user performance teacher data supplied from the scenario controller 19, and the performance evaluation value data And the comprehensive evaluation value data are supplied to the scenario controller 19 as performance evaluation result data.

  In step S6, when the performance evaluation result data is supplied, the scenario controller 19 compares the performance evaluation value with its threshold value, and compares the overall evaluation value with its threshold value. If the overall evaluation value is equal to or greater than the threshold value, the process proceeds to step S7. On the other hand, when the comprehensive evaluation value is less than the threshold value, the process proceeds to step S9.

  When the comprehensive evaluation value is equal to or greater than the threshold value, in step S7, the scenario controller 19 supplies the agent controller 20 with agent action data for conveying the contents of the user's performance in the next scene to the user. The scenario controller 19 supplies user performance teacher data corresponding to the next scene to the performance evaluator 18.

  In step S8, the agent 21 receives the control of the agent controller 20 and informs the user of the performance content of the next scene as a stage director.

  On the other hand, if the overall evaluation value is less than the threshold value, in step S9, the scenario controller 19 sends agent action data for conveying the correction points of the current performance to the user for the modal whose performance evaluation value is less than the threshold value. The agent controller 20 is supplied. The scenario controller 19 supplies user performance teacher data corresponding to the current scene to the performance evaluator 18 again.

  In step S10, the agent 21 receives the control of the agent controller 20 and informs the user of the correction point of the current performance.

  As described above, in the performance practice mode, it is determined whether the user performs the next scene or again performs the current scene, based on the comparison result between the comprehensive evaluation value and the threshold value.

For example, in the acting practice mode,
Agent A (stage director): “So, look up at the sky and say,“ Oh, Romeo, why are you Romeo? ”
User: “Looking at the sky in front of the camera image input device and microphone audio input device, oh, Romeo, why are you Romeo?”
Performance evaluator: Fscore = 0.9, Gscore = 0.9, Escore = 0.8, Hscore = 0.8, Tscore = 0.5, PROscore = 0.3
Scenario controller: Score _Threshold (80)> Score (70) Then Repeat
Agent A (stage director): “I currently have 70 performances. The expressions and poses are good, but the dialogue is not good. Try again.”
User: “(Looking up at the sky in front of the camera image input device and microphone audio input device)“ Oh, Romeo, why are you Romeo? ”
Performance evaluator: Fscore = 0.9, Gscore = 0.9, Escore = 0.9, Hscore = 0.9, Tscore = 1, PROscore = 0.9
Scenario controller: Score _Threshold (80) <Score (92) Then Go To Next
Agent A (stage director): “I did quite well. 92 points! Then let's go to the next scene?”
The story is advanced by the interaction between the performance evaluation device 1 and the user.

  On the other hand, the operation of the performance evaluation device 1 in the stage performance mode is shown in FIG.

  First, in step S11, the scenario controller 19 supplies agent operation data for controlling the operation of the agent in the first scene of the story to the agent controller 20. The scenario controller 19 supplies user performance teacher data corresponding to the first scene of the story to the performance evaluator 18.

  Next, in step S12, the agent 21 receives the control of the agent controller 20 and performs the first scene of the story as the user's opponent.

  Subsequently, in step S13, when the user performs the action of the first scene in response to the acting of the partner agent 21, the camera image input device 10 displays the user's performance image and the microphone audio input device 11 as well. Inputs the voice of the user's performance and supplies the input image and voice to each of the above-described recognizers.

  Thereafter, in step S <b> 14, when each recognizer performs image processing and sound processing, it supplies data of recognition results of each modal to the performance evaluator 18.

  In step S15, the performance evaluator 18 calculates the user performance evaluation value using the data supplied from each recognizer and the user performance teacher data supplied from the scenario controller 19, and the performance evaluation value data And the comprehensive evaluation value data are supplied to the scenario controller 19 as performance evaluation result data.

  In step S <b> 16, the scenario controller 19 supplies the agent controller 20 with agent action data for controlling the action of the agent in the next scene and agent action data for showing the audience reaction corresponding to the comprehensive evaluation value to the user. . The scenario controller 19 supplies user performance teacher data corresponding to the next scene to the performance evaluator 18.

  In step S17, the agent 21 under the control of the agent controller 20 shows the user the reaction to the user performance of the current scene as a spectator.

  In step S18, the agent 21 under the control of the agent controller 20 performs the next scene as the user's opponent.

For example, in stage production mode,
Agent A (Romeo): "If you touch this and get dirty, I'm a blush pilgrimage, so let me kiss you for compensation."
User (Juliet): (In front of the camera image input device and microphone audio input device) I ’m happy to hear from you, but you should n’t kiss. ”
Performance evaluator: Fscore = 0.9, Gscore = 0.9, Escore = 0.9, Hscore = 0.9, Tscore = 1, PROscore = 0.9
Scenario controller: Score _Threshold (80) <Score (92) Then Go To Next
Agent B (audience): The story is advanced by the interaction between the performance evaluation apparatus 1 and the user such as a big applause.

  As mentioned above, although the structure and operation | movement of the performance evaluation apparatus 1 in the case where the performance evaluation device 18 calculates a performance evaluation value using the distance between weighted data were demonstrated, as a method by which the performance evaluation device 18 calculates a performance evaluation value. In addition to the method using the weighted data distance, there is a method using a Bayesian network.

  FIG. 5 shows a configuration of the performance evaluator 18 using a Bayesian network.

  In FIG. 5, each recognizer and the scenario controller 19 are also shown.

  The performance evaluator 18 is composed of a node having the recognition result of each modal recognizer at each time as a state variable and a node having a scene type as the state variable, and connects the causal relationships of the nodes with an effective graph. It has a Bayesian network structure.

  Here, each node has a conditional probability distribution (CPD) or a conditional probability table (CPT).

  A Bayesian network structure may be designed by a designer or acquired by learning from sample data. The latter structure collects time-series data in each recognizer when a user actually performs a certain performance. Sample data and obtained using the K2 algorithm, MCMC method, or the like.

  Using the Bayesian network, the performance matching probability indicating how much the user has performed to the current scene is calculated as follows.

First, the scenario controller 19 supplies user performance teacher data (SCENE _ID ) indicating the ID of the current scene to the performance evaluator 18.

  After each modal recognizer performs image processing and voice processing, the recognition result data other than the voice recognizer 16 is supplied to the corresponding node at each time. The speech recognizer 16 supplies the recognition result data to the corresponding node at the time when the recognition result data is supplied.

Subsequently, the performance evaluator 18 performs inference by a method such as π-λ method, junction tree algorithm, Loopy · BP, etc., when the recognition result data of the recognizer at each time is supplied to each node. The performance matching probability {Prob. (SCENE _ID )} is calculated as an evaluation value for the performance of each scene.

  Thereafter, the performance evaluator 18 supplies the scenario controller 19 with performance evaluation value data expressed as the performance suitability probability as performance evaluation result data.

  The scenario controller 19 has a certain threshold value for the performance evaluation value. When the performance evaluation result data is supplied from the performance evaluation unit 18, the scenario controller 19 compares the performance evaluation value with the threshold value.

In response to the comparison result, the scenario controller 19 supplies agent operation data indicating what operation the agent 21 should perform to the user to the agent controller 20 and indicates the ID of the current or next scene. User performance teacher data (SCENE _ID ) is supplied to the performance evaluator 18.

For example, in the stage production mode of the scene with ID = 3,
Agent A (Romeo): "If you touch this and get dirty, I'm a blush pilgrimage, so let me kiss you for compensation."
User (Juliet): (In front of the camera image input device and microphone audio input device) I ’m happy to hear from you, but you should n’t kiss. ”
Performance evaluator: Prob. (SCENE _ID = 3) = 0.9
Scenario controller: Prob _Threshold <Prob. (SCENE _ID = 3) Then Go To Next
Agent B (audience): The story is advanced by the interaction between the performance evaluation apparatus 1 and the user such as a big applause.

  It should be noted that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

  For example, in the above-described embodiments, the method using the weighted data distance and the method using the Bayesian network have been described as methods by which the performance evaluator 18 calculates the performance evaluation value. However, it is also possible to configure a performance evaluation apparatus that integrates a performance evaluator that uses a distance between weighted data and a performance evaluator that uses a Bayesian network. Specifically, a distance between weighted data may be calculated from data supplied from several recognizers, and the result may be supplied to a Bayesian network to calculate an overall performance evaluation value. Alternatively, data supplied from several recognizers may be supplied to a Bayesian network, and a comprehensive performance evaluation value may be calculated from the results using the distance between weighted data.

  Moreover, as a modal used when each recognizer recognizes a user's performance, in addition to the modal used in the above-described embodiment, for example, the movement of the foot, the shape of the mouth, the shape of the eyes, the shape of the eyebrows, and the posture Etc. may be used.

It is a block diagram which shows typically the structure of the performance evaluation apparatus in this Embodiment. It is a figure which shows the feature point of the facial expression detected by the facial expression recognizer in this Embodiment, and the distance between feature points. It is a flowchart which shows the operation | movement order at the time of the performance practice mode of a performance evaluation apparatus. It is a flowchart which shows the operation | movement order in the stage performance mode of a performance evaluation apparatus. It is a figure which shows the performance evaluator using a Bayesian network.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Performance evaluation apparatus, 10 Camera image input device, 11 Microphone voice input device, 12 Face recognition device, 13 Eye-gaze recognition device, 14 Expression recognition device, 15 Hand movement recognition device, 16 Speech recognition device, 17 Prosody recognition device, 18 Performance evaluation 19 Scenario controller 19, 20 Agent controller, 21 Agent, 40 Eyebrows, 41 eyes, 42 mouths, 43 Feature points, 44 Distance between feature points

Claims (4)

A performance evaluation device for evaluating a user's performance in a predetermined scene ,
Recognition means for recognizing the user's performance through a plurality of modals;
The performance evaluation value for each modal is calculated using the recognition result data for each modal obtained by the recognition means and the teacher data for each modal, and all the performance evaluation values for each modal are calculated. a performance evaluation means for evaluating the performance of the current scene of the user to calculate the overall evaluation value for comprehensively evaluating the performance of the user using,
Comparing means for comparing the performance evaluation value with a first threshold value serving as a reference value for the performance evaluation value and comparing the comprehensive evaluation value with a second threshold value serving as a reference value for the comprehensive evaluation value;
When the comprehensive evaluation value is equal to or greater than the second threshold, the user's performance content in the next scene is notified to the user, and when the comprehensive evaluation value is less than the second threshold, the performance evaluation value is first the current scene of Ru performance evaluation device and a notification means for notifying to the user the modification point acting about the modal is less than the threshold. The recognition result data and the teacher data are represented by vectors,
2. The performance evaluation device according to claim 1 , wherein the performance evaluation means calculates the performance evaluation value using a distance between vectors of the recognition result data and the teacher data. The recognizing means, as modal, includes the position and direction of the face, the direction of the line of sight, facial expressions, hand and foot movement, mouth shape, eye shape, eyebrow shape, posture, utterance content, and prosody, acting evaluation apparatus according to claim 1, wherein the Ru using at least two or more. A performance evaluation method for evaluating a user's performance in a predetermined scene ,
A recognition process for recognizing the user's performance through a plurality of modals;
The performance evaluation value for each modal is calculated using the recognition result data for each modal obtained in the recognition step and the teacher data for each modal, and all the performance evaluation values for each modal are calculated. A performance evaluation step of evaluating a performance in the current scene of the user by calculating a comprehensive evaluation value for comprehensively evaluating the performance of the user using
A comparison step of comparing the performance evaluation value with a first threshold value serving as a reference value for the performance evaluation value and comparing the comprehensive evaluation value with a second threshold value serving as a reference value for the comprehensive evaluation value;
When the comprehensive evaluation value is equal to or greater than the second threshold, the user's performance content in the next scene is notified to the user, and when the comprehensive evaluation value is less than the second threshold, acting evaluated how having a a notifying step of the performance evaluation value reports the fixes of the acting current scene for the modal is less than the first threshold value to the user.

Images