JP7260826B2 - Learning device and evaluation information output device - Google Patents

Description

The present invention relates to a learning device and an evaluation information output device.

2. Description of the Related Art Conventionally, the productivity of a conference is estimated using the behavior and the amount of speech of conference attendees obtained using images (Patent Document 1 (Japanese Patent Application Laid-Open No. 2019-211962)).

When estimating the productivity of a group activity such as a meeting, the evaluation based on the behavior and the amount of speech of the attendees of the meeting is limited to the quantitative evaluation of the group activity, and the qualitative evaluation is not sufficient. In addition, even if the quantitative evaluation of group activities is small, there are cases where high-quality group activities are carried out, so qualitative evaluation is important, but it is difficult to qualitatively evaluate group activities such as meetings. I have a problem.

A learning device according to a first aspect is a learning device for learning one or more human nodding motions, and includes a learning unit. The learning unit receives information about the nodding motion and generates a trained model that outputs nodding motion evaluation information. The learning unit learns the information about the nodding motion and the evaluation information of the nodding motion as a learning data set to generate a trained model.

With this learning device, by learning information about nodding motions, which are reactions of attendees of meetings, etc., and evaluation information of nodding motions, as a learning data set, it is possible to perform qualitative evaluation of group activities such as meetings. You can generate a trained model that can

An evaluation information output device according to a second aspect is an evaluation information output device that outputs evaluation information related to nodding motions based on one or more human nodding motions, comprising a first information acquisition unit and an output unit. , provided. The first information acquisition unit acquires information about the nodding motion. The output unit has a trained model that receives information about the nodding motion and outputs nodding motion evaluation information. The output unit estimates and outputs evaluation information of the nodding motion from information about the nodding motion using the trained model. The trained model is obtained by learning information about the nodding motion and evaluation information of the nodding motion as a learning data set.

In this evaluation information output device, a trained model is used to estimate evaluation information of nodding motions from information about nodding motions that are reactions of attendees of meetings, etc., thereby performing a qualitative evaluation of group activities such as meetings. be able to.

The evaluation information output device of the third aspect is the device of the second aspect, further comprising a second information acquisition section. The second information acquisition unit acquires information about an object that causes a nodding motion. The object causing the nodding motion includes at least one of speech during a meeting, data or objects presented in a presentation, or a human or animal performance. The output unit outputs the information about the target that causes the nodding motion and the evaluation information of the nodding motion in association with each other.

In this evaluation information output device, by linking and outputting information about an object that causes a nodding motion and evaluation information of the nodding motion, the evaluation information of the nodding motion caused by the specific target is output. can be obtained.

The evaluation information output device of the fourth aspect is the device of the second aspect or the third aspect, and the evaluation information of the nodding motion is information related to human emotion. The information about human emotion includes information about human emotions, or information about whether or not a human being is satisfied with an object that causes a nodding motion. The nodding motion evaluation information can be obtained by analysis of human face image data or voice data analysis.

This evaluation information output device can determine the quality of a nod by estimating the intention of the human nodding motion from the information on the nodding motion.

The evaluation information output device according to the fifth aspect is the device according to the second aspect or the third aspect, and the nodding motion evaluation information is an evaluation of an object that causes the nodding motion. The evaluation of the subject responsible for the nodding motion is the result of a manual questionnaire given to humans.

With this evaluation information output device, the quality of the nodding motion can be determined by obtaining an evaluation of the object that causes the nodding motion in a questionnaire.

An evaluation information output device according to a sixth aspect is the device according to any one of the second aspect to the fifth aspect, wherein the information regarding the nodding motion includes at least one of the number of nodding motions, the period, and the amplitude.

With this evaluation information output device, it is possible to obtain information about the reactions of attendees at a meeting or the like by measuring the number of nodding motions, the period, or the amplitude.

The evaluation information output device according to the seventh aspect is the device according to any one of the second aspect to the fifth aspect, wherein the information about the nodding motion is the timing of the nodding motion, the strength of the nodding motion, or the number of persons among the plurality of people. , the presence or absence of a person with the same nodding motion timing, or at least one of the number of people with the same nodding motion timing.

This evaluation information output device measures the timing of nodding motions, the strength of nodding motions, the presence or absence of a person nodding at the same time among a plurality of people, or the number of people nodding at the same time. It is possible to obtain information on the reaction of

The evaluation information output device of the eighth aspect is the device of the third aspect, and the information about the nodding motion includes the time from the speech during the meeting to the occurrence of the nodding motion.

This evaluation information output device can determine the quality of a nodding motion by measuring the time from the time a speech is made during a meeting until the nodding motion occurs.

The evaluation information output device of the ninth aspect is the device of any one of the second aspect to the fifth aspect, wherein the information about the nodding motion is at least one of the frequency of the nodding motion and the frequency of the nodding motion per unit time. including.

With this evaluation information output device, it is possible to determine the quality of the nodding motion by obtaining the frequency of the nodding motion or the frequency of the nodding motion per unit time.

The evaluation information output device according to the tenth aspect is the device according to any one of the second aspect to the ninth aspect, wherein the first information acquisition unit comprises a seat sensor, motion capture, acceleration sensor, depth camera, or human moving image. is used to obtain information about the nodding motion obtained by recognizing the nodding motion.

This evaluation information output device uses a seat sensor, a motion capture, an acceleration sensor, a depth camera, or a moving image of a person to easily acquire information about a nodding gesture, which is a gesture during a meeting.

2 is a functional block diagram of the learning device 1; FIG. 3 is a functional block diagram of the evaluation information output device 100; FIG. FIG. 4 is a diagram showing an example of learning data; It is a figure for demonstrating a learning process. 4 is a flowchart of the evaluation information output device 100; 2 is a functional block diagram of an evaluation information output device 200; FIG.

(1) Overall Configuration of Evaluation Information Output Device FIG. 2 shows an evaluation information output device 100 of this embodiment. The evaluation information output device 100 is implemented by a computer. The evaluation information output device 100 includes a first information acquisition section 101 and an output section 102 . The first information acquisition unit 101 acquires information 11 regarding nodding motions. The output unit 102 has a trained model 40 and uses the trained model 40 to estimate evaluation information 21 of nodding motions from information 11 related to nodding motions and outputs the estimated nodding motion evaluation information 21 .

(2) Detailed Configuration of Evaluation Information Output Device (2-1) First Information Acquisition Unit The first information acquisition unit 101 acquires information 11 regarding a nodding motion. The information 11 on the nodding motion is the number of nodding motions. In this embodiment, the first information acquisition unit 101 acquires the number of nodding motions using a seat surface sensor (not shown).

The number of nodding movements is measured using a seat surface sensor (pressure sensor). The participants of the conference are sitting in chairs on which seat sensors are installed.

First, the position of the center of gravity P and the total load WT are obtained based on the magnitude of the pressure applied to the seat surface sensors installed at the four corners of the chair. The seat surface sensor is made to learn changes in the center of gravity P and the magnitude of the total load WT by machine learning, and it is detected whether or not it corresponds to a nodding motion. The seat sensor detects the timing of a person's nodding from the change in the weight of the center of gravity.

The seat sensor divides the time during the meeting into a plurality of frames and calculates the amount of nodding, which is the number of nodding movements. Whether or not each participant in the conference is nodding is determined on a frame-by-frame basis. For participant k, the nodding amount akt at a certain time _t is calculated by the following equation 1, where _Sk is the total number of nodding frames of participant k.
Formula 1:

For example, if there are six participants, the amount of nodding for that frame is calculated by adding up the frames assigned to each participant for all six participants. The nodding amount A _t at a certain time t is calculated by the following equation 2.
Equation 2:

As shown in Equation 1, by weighting the reciprocal of the total number of nodding frames for each participant, when calculating the amount of nodding in Equation 2, the nodding of people who often nod has little effect, and the nods of people who do not often nod. The nod is making the impact bigger.

(2-2) Output Unit The output unit 102 has a trained model 40 and uses the trained model 40 to estimate and output nodding motion evaluation information 21 from the nodding motion related information 11 . A processor such as a CPU or GPU can be used for the output unit 102 . The output unit 102 reads a program stored in a storage device (not shown) and performs predetermined image processing and arithmetic processing according to the program. Furthermore, the output unit 102 can write the calculation result to the storage device and read the information stored in the storage device according to the program. A storage device can be used as a database.

(3) Learning Process The trained model 40 used by the evaluation information output device 100 will be described. The trained model 40 uses an estimation model that has been trained in advance using the learning apparatus 1 with a training data set that includes information about the nodding motion and evaluation information of the nodding motion.

FIG. 1 shows a learning device 1 of this embodiment. The learning device 1 is implemented by a computer. The learning device 1 includes a learning section 2 .

The learning unit 2 learns information 10 related to the nodding motion and evaluation information 20 of the nodding motion as a learning data set 30 . In this embodiment, the information 10 on the nodding motion is the number of nodding motions (amount of nodding). The nodding motion evaluation information 20 is an evaluation of the object that causes the nodding motion. The object that causes the nodding motion is the speech during the meeting. Speech during the meeting shall relate to the ideas generated during the meeting.

In the learning process, information 10 related to the nodding motion is obtained using the seat surface sensor. The nodding motion evaluation information 20 is obtained by a manual questionnaire.

The trained model 40 is obtained by learning the information 10 on the nodding motion and the nodding motion evaluation information 20 using the learning data set 30 .

FIG. 3 shows an example of learning data. In this example, the number of nodding movements (amount of nodding) of the participants of the conference when an idea is generated during the conference is used as the information on the nodding movements. The average value of the results of the questionnaire regarding the ideas generated during the meeting is used as the evaluation information of the nodding motion. A questionnaire is given to the participants of the conference to evaluate their ideas on a scale of five.

As shown in FIG. 3, the nodding amount _A1 for the idea _X1 generated at time _t1 during the meeting is input, and learning is performed using a teacher data set whose output is the average value "2" of the questionnaire results. In addition, learning is performed using a teacher data set in which the amount of nodding A ₂ for idea X ₂ generated at time t ₂ during the meeting is input, and the average value of the questionnaire results "3" is output. In addition, learning is performed using a teacher data set in which the amount of nodding A ₃ for idea X ₃ generated at time t ₃ during the meeting is input, and the average value of the questionnaire results "4" is output. In addition, learning is performed using a teacher data set in which the amount of nodding An for an idea Xn generated at time _tn during the meeting is input, and the average value "2" of the questionnaire results is output.

FIG. 4 shows an example of a scatter diagram prepared with the vertical axis representing the total amount of nodding, which is the total number of frames in which all participants nodded, and the horizontal axis representing the average value of the questionnaire results.

In this embodiment, 13 groups of 6 participants are participating in the conference. The total nodding amount for each group is simply calculated by summing all the nodding frames of the participants for each group. We also conduct a questionnaire on whether the ideas are impressive. The questionnaire is evaluated in five stages.

As shown in FIG. 4, there is a positive correlation between the average value of the questionnaire results and the total number of nods. Correlation analysis using Spearman's rank correlation coefficient for the average value of the questionnaire results and the total number of nods shows p>.05. The absolute value of the correlation coefficient was 0.49, indicating a correlation. Since there is a correlation between the average value of the questionnaire results and the total number of nods, it is shown that there is not only an instantaneous relationship between the quality of ideas and the number of nods, but also a medium- to long-term relationship.

(4) Overall Operation of Evaluation Information Output Device A flowchart of the evaluation information output device 100 is shown in FIG. In this embodiment, a case where the evaluation information output device 100 is used in a conference will be described.

First, a conference is started in step S1. Conference attendees are filmed on camera. Also, conference attendees are sitting in chairs with pressure sensors installed. The first information acquisition unit 101 acquires the number of nodding motions (amount of nodding) of the participants in the conference (step S2). Using the trained model 40, the output unit 102 estimates the average value of the questionnaire results from the number of nodding actions of the participants in the meeting (step S3). Next, the output unit 102 outputs the average value of the questionnaire results estimated in step S3 to a display (not shown) (step S4). The display shows the estimated mean value of the questionnaire results.

(5) Features (5-1)
The learning device 1 according to the present embodiment is a learning device 1 that learns nodding motions of one or more people, and includes a learning unit 2 . The learning unit 2 generates a trained model 40 that receives information about the nodding motion and outputs nodding motion evaluation information. The learning unit 2 learns the nodding motion information 10 and nodding motion evaluation information 21 as a learning data set 40 to generate a trained model 30 .

In this learning device 1, by learning information 10 on nodding motions, which are reactions of attendees of a meeting, etc., and evaluation information 20 of the nodding motions as a learning data set 30, qualitative evaluation of group activities such as meetings is performed. can generate a trained model 40 that can perform

(5-2)
The evaluation information output device 100 according to the present embodiment is an evaluation information output device 100 that outputs evaluation information related to nodding motions based on one or more human nodding motions, and includes a first information acquisition unit 101 and , and an output unit 102 . The first information acquisition unit 101 acquires information 11 regarding nodding motions. The output unit 102 has a trained model 40 that receives information 11 on a nodding motion and outputs nodding motion evaluation information 21 . The output unit 102 uses the trained model 40 to estimate the nodding motion evaluation information 21 from the nodding motion related information 11 and outputs the estimated nodding motion evaluation information 21 . The trained model 40 is obtained by learning the information 10 on the nodding motion and the nodding motion evaluation information 20 as the learning data set 30 .

This evaluation information output device 100 uses a trained model 30 to estimate the evaluation information of the nodding motion from the information 11 on the nodding motion, which is the reaction of the attendees of the meeting or the like. can be evaluated.

The output unit 102 of the evaluation information output device 100 can present the intellectual productivity score of the meeting as nodding motion evaluation information based on the number of nodding motions along the elapsed time of the meeting. The evaluation information output device 100 evaluates the intellectual productivity of the conference based on the nodding motions of the participants and their timing.

Also, the synchrony of the participant may be evaluated based on the timing of the participant's nod and the length of the nod. By evaluating the nod after the speech, the degree of contribution of the speech in the meeting can be evaluated.

(5-3)
In the evaluation information output device 100 according to the present embodiment, the nodding motion evaluation information 21 is an evaluation of the object that causes the nodding motion. The evaluation of the subject responsible for the nodding motion is the result of a manual questionnaire given to humans.

With this evaluation information output device 100, the quality of the nodding motion can be determined by obtaining an evaluation of the object that causes the nodding motion in a questionnaire.

(5-4)
In the evaluation information output device 100 according to the present embodiment, the information 11 related to the nodding motion includes the number of nodding motions.

This evaluation information output device 100 measures the number of nodding motions and calculates the amount of nodding, thereby obtaining information on the reactions of attendees at a conference or the like.

(5-5)
In the evaluation information output device 100 according to the present embodiment, the first information acquisition unit 101 acquires the information 11 regarding the nodding motion obtained by recognizing the nodding motion using the seat surface sensor.

This evaluation information output device 100 can easily measure the number of times 11 of nodding motions, which are gestures during a meeting, by using a seat surface sensor.

(6) Modification (6-1) Modification 1A
Although the evaluation information output device 100 according to the present embodiment has been described as having the first information acquisition section 101 and the output section 102, it may further comprise a second information acquisition section.

FIG. 6 shows an evaluation information output device 200 of Modification 1A. The evaluation information output device 200 is implemented by a computer. The evaluation information output device 200 includes a first information acquisition section 201 , a second information acquisition section 203 and an output section 202 . The first information acquisition unit 201 acquires the information 11 regarding the nodding motion. The second information acquisition unit 203 acquires the information 50 regarding the object that causes the nodding motion.

The information 50 about the object that causes the nodding motion is the speech during the meeting. In the evaluation information output device 200 of Modification A, the second information acquisition unit 203 shoots a moving image using a camera (not shown), and acquires ideas, which are utterances during the meeting, from the moving image.

The output unit 202 outputs, as output data 60, data in which the information 50 regarding the object that causes the nodding motion and the evaluation information 21 of the nodding motion are linked. Also, the output unit 202 may output evaluation data in which the minutes information of the meeting developed in chronological order and the intellectual productivity score, which is the evaluation information 21 of the nodding motion, are combined.

In this evaluation information output device 200, the information 50 related to the target causing the nodding motion and the evaluation information 21 of the nodding motion are output in association with each other. evaluation information can be obtained.

(6-2) Modification 1B
In the evaluation information output device 100 according to the present embodiment, the information 11 regarding the nodding motion is the number of nodding motions, but it may be the period or the amplitude of the nodding motion.

The first information acquisition unit 101 may acquire information related to nodding, including the timing at which the nodding motion was performed. By acquiring the timing of the nodding motion, it is possible to grasp the number of participants who nodded at the meeting. The greater the number of people who nodded, the higher the possibility that an idea was created before the nodding motion. Nodding is more common when ideas are generated than when there is no dialogue.

In addition, the information 11 related to the nodding motion includes the timing of the nodding motion, such as whether the participant nods immediately after speaking during the meeting or nods later, the strength of the nodding motion, the presence or absence of people nodding at the same time, and the number of such people. good.

In addition, it is possible to determine the characteristics of a nod by using the time from the idea to the occurrence of the nodding motion, the length of the nodding motion (frequency), the number of nodding motions (frequency per unit time), etc. . Since the time from coming up with an idea to nodding is captured in a video, it is digitized by extracting the time difference from the utterance timing. Also, the frequency of the nodding motion is digitized by extracting the frequency contained in the image.

Accordingly, by determining the characteristics of the nodding, it is possible to determine the degree of concentration of the participants during the conference and the degree of excitement of the conference.

(6-3) Modification 1C
In the evaluation information output device 100 according to the present embodiment, the number of nodding motions, which is the information 11 related to nodding motions, is the total amount of nodding for each group of participants in the conference, but the present invention is not limited to this. The number of nodding motions performed by each participant in the conference may be used as the information 11 regarding nodding motions.

(6-4) Modification 1D
In the evaluation information output device 100 according to the present embodiment, the evaluation information 21 of the nodding motion is the result of a manual questionnaire given to humans, and a questionnaire was conducted as to whether the ideas generated during the meeting were impressive. Although the case has been described, it is not limited to this. A questionnaire may be conducted as to whether the ideas generated during the meeting are original, convincing, or socially acceptable.

(6-5) Modification 1E
In the evaluation information output device 100 according to the present embodiment, the object causing the nodding motion has been described as being an idea, which is an utterance during a meeting, but the present invention is not limited to this. The object that causes the nodding motion may include at least one of data or objects presented in the presentation, or a human or animal performance.

(6-6) Modification 1F
In the evaluation information output device 100 according to the present embodiment, the case where the evaluation information 21 of the nodding motion is the evaluation regarding the object causing the nodding motion has been described, but information regarding human emotions may also be used. Information about human emotions includes information about whether or not a person is satisfied with an object that causes human emotions or a nodding motion. The nodding motion evaluation information can be obtained by analysis of human face image data or voice data analysis.

Thus, by estimating the intention of the human nodding motion from the information 11 related to the nodding motion, it is possible to determine the quality of the nodding motion.

(6-7) Modification 1G
In the evaluation information output device 100 according to the present embodiment, the case where the first information acquisition unit 101 acquires the information about the nodding motion from the seat surface sensor has been described, but the present invention is not limited to this. Information about the nodding motion obtained by recognizing the nodding motion may be acquired using motion capture, an acceleration sensor, a depth camera, or a human video.

Motion capture can estimate head movement using a motion capture device such as OptiTrack. Moreover, JIN Meme etc. are mentioned as an acceleration sensor. An acceleration sensor can be used to capture head movements. A depth camera or kinect sensor can also be used to measure the number of nodding movements. Also, nodding motions can be detected from head movements based on the moving images of the participants during the conference.

(6-8) Modification 1H
Although embodiments of the present disclosure have been described above, it will be appreciated that various changes in form and detail may be made without departing from the spirit and scope of the present disclosure as set forth in the appended claims. .

1 learning device 2 learning units 100 and 200 evaluation information output devices 101 and 201 first information acquisition unit 203 second information acquisition units 102 and 202 output units 10 and 11 nodding motion information 20 and 21 nodding motion evaluation information 30 for learning Data set 40 Trained model 50 Information about the object causing the nodding motion

JP 2019-211962 A

Claims (9)

A learning device for learning one or more human nodding actions,
a learning unit (2) that generates a trained model (40) that receives the information (10) about the nodding motion as input and outputs evaluation information (20) of the nodding motion;
with
the evaluation information of the nodding motion is an evaluation of an object that causes the nodding motion;
The learning unit learns information about the nodding motion and evaluation information of the nodding motion as a learning data set (30) to generate the trained model.
A learning device (1). An evaluation information output device for outputting evaluation information regarding a nodding motion based on one or more human nodding motions,
a first information acquisition unit (101, 201) for acquiring information (11) about the nodding motion;
a second information acquisition unit (203) that acquires information (50) about an object that causes the nodding motion;
A trained model that receives information on the nodding motion as an input and outputs evaluation information on the nodding motion, and uses the trained model to estimate the evaluation information (21) on the nodding motion from the information on the nodding motion. and an output unit (102, 202) that outputs
with
the evaluation information of the nodding motion is an evaluation of an object that causes the nodding motion;
The trained model is learned by using information about the nodding motion and evaluation information of the nodding motion as a learning data set,
The object that causes the nodding motion includes at least one of speech during a meeting, data or objects presented in a presentation, or performance of a person or animal,
The output unit outputs information related to an object that causes the nodding motion and evaluation information of the nodding motion in association with each other.
An evaluation information output device (100, 200). the nodding motion evaluation information further includes information about the human emotion,
The information about the emotion of the human includes information about the emotion of the human, or information about whether the human is satisfied with the object that causes the nodding motion,
The nodding motion evaluation information can be obtained by analysis of the human face image data or voice data analysis.
3. The evaluation information output device according to claim 2 . The evaluation of the subject that causes the nodding motion is the result of a manual questionnaire conducted on the human.
3. The evaluation information output device according to claim 2 . the information about the nodding motion includes at least one of the number of nodding motions, the period, or the amplitude;
5. The evaluation information output device according to claim 2. The information about the nodding motion is the timing of the nodding motion, the strength of the nodding motion, or the presence or absence of the person whose nodding motion timing is the same among the plurality of people, or the timing of the nodding motion. includes at least one of the number of persons whose
5. The evaluation information output device according to claim 2. The information on the nodding motion includes the time from when the speech is made during the meeting to when the nodding motion occurs.
3. The evaluation information output device according to claim 2 . The information about the nodding motion includes at least one of the frequency of the nodding motion or the frequency of the nodding motion per unit time.
5. The evaluation information output device according to claim 2. The first information acquisition unit acquires information about the nodding motion obtained by recognizing the nodding motion using a seat sensor, motion capture, acceleration sensor, depth camera, or video of the human.
The evaluation information output device according to any one of claims 2 to 8 .

Images