JP7214372B2 - Conference productivity visualization system, conference productivity visualization device, visualization method and program - Google Patents

Description

TECHNICAL FIELD Embodiments of the present invention relate to a conference productivity visualization system, a conference productivity visualization device, a visualization method, and a program.

Meetings occupy a large amount of time in daily work, and it is important to improve the productivity of meetings in order to improve work efficiency. Conventionally, there has been known a technique of measuring and accumulating the meeting time of each attendee, and totaling the meeting time for each meeting, each project, and each individual. Also, there is known a technique of calculating the wasteful cost of the conference from silent time, interruption time, etc., and further calculating individual contribution cost according to the amount of speech of the individual. On the other hand, there is known a technique that enables evaluation of the entire conference including the attitudes of the attendees by estimating the attitudes of the attendees (face down, supine, etc.) using a camera. In addition, there is known a technique for calculating the degree of contribution of a speaker for each phase of a conference by analyzing the content of the conference itself using speech recognition technology.

However, with conventional technology, the quality of the conference and the contribution costs of individual participants are calculated independently. It is difficult to create an atmosphere that creates In addition, since individual contribution costs cannot be accumulated and visualized, it is difficult to identify persons who contribute to high-quality conferences. Furthermore, if a particular attendee continues to speak, the degree of contribution will be high, and the quality of the meeting measured will be high, even though there is a high possibility that they are obstructing other attendees from speaking. There is a problem of storage. In addition, the attitudes of other attendees while speaking are only used to assess the quality of the meeting itself or the person taking the attitude. Furthermore, the method using speech recognition has many problems such as recognition rate, and it is difficult to specify the content of the speech and the conference phase itself.

JP-A-2005-31944 JP 2011-60054 A Japanese Patent Application Laid-Open No. 2013-8114 JP-A-2000-99570 Japanese Patent Application Laid-Open No. 2013-6511 JP-A-9-307868

T. Watanabe et al.: Co-occurrence histograms of oriented gradients for pedestrian detection, 2009 Yuasa et al., Digital makeup system based on high-precision facial feature point detection, SSII10, 2004 T. Kozakaya et al.: Face Recognition by Projection-based 3D Normalization and Shading Subspace Orthogonalization, 2006

The problem to be solved by the present invention is to identify conference attendees using sensors, accumulate the conference attendance time of each attendee, and estimate the utterance state, facial expression, line of sight (non-verbal information), etc. of the attendees. It is to visualize the meeting attendance time and productivity score for each organization, meeting, and attendee by scoring the productivity of the meeting and distributing and accumulating the productivity score according to the contribution of each attendee. .

In order to achieve the above object, the conference productivity visualization system of the embodiment identifies conference attendees based on images, and analyzes the behavior of other attendees while a certain attendee is speaking. a meeting time calculation unit that calculates the meeting attendance time of each attendee as a personal meeting time and calculates the meeting time from the sum of the personal meeting time of all attendees; and the analysis of the behavior analysis unit Based on the results and the calculated meeting time, the meeting score calculation unit calculates the meeting score for evaluating the productivity of the meeting, and the contribution of each attendee to the meeting is calculated based on the analysis results of the behavior analysis unit. a contribution calculation unit for calculating a contribution, a personal score calculation unit for calculating a personal score based on the calculated meeting score and the contribution, and a meeting productivity visualization unit for graphically displaying the calculated meeting score or the individual score.

1 is a block diagram showing a conference productivity visualization system according to this embodiment; FIG. The first half of the operation of the conference productivity visualization system according to the present embodiment. An example of an installation method of the image acquisition device 100 according to the present embodiment. Detailed operation of behavior analysis according to the present embodiment. An example of a meeting history DB 240a according to the present embodiment. An example of a user table 240b according to this embodiment. An example of a meeting table 240c according to the present embodiment. The second half of the operation of the conference productivity visualization system according to the present embodiment. Detailed operation of the meeting score calculator 230 according to the present embodiment. An example of a correction value table 240d according to this embodiment. Detailed operation of the contribution calculation unit 230 according to the present embodiment. An example of the calculation image of the individual score which concerns on this embodiment. An example of a method for illustrating meeting productivity according to the present embodiment.

Embodiments for carrying out the invention will be described below. FIG. 1 is a block diagram showing a conference productivity visualization system according to this embodiment. In the conference productivity visualization system of this embodiment, an image acquisition device 100 and a conference productivity visualization device 200 are connected via a network.

The image acquisition device 100 is, for example, a camera, an infrared sensor, or the like, and includes an image acquisition unit 110 that acquires images of conference attendees and the like. The conference productivity visualization device 200 includes a behavior analysis unit 210 , a conference time calculation unit 220 , a conference score calculation unit 230 , a storage unit 240 , a contribution calculation unit 250 , a personal score calculation unit 260 and a conference productivity visualization unit 270 .

Behavior analysis section 210 includes attendee identification section 211 , speaker identification section 212 , speech amount calculation section 213 , gaze direction estimation section 214 , nodding detection section 215 , motion amount estimation section 216 , and smile estimation section 217 .

The attendee identification unit 211 detects attendees from the image acquired by the image acquisition unit 110 . The speaker identification unit 212 detects the mouth portion of the attendee identified by the attendee identification unit 211, and identifies the speaker based on the open/close state of the mouth. The speech amount calculation unit 213 calculates the speaking time of the attendee based on the opening/closing state of the mouth of the attendee identified by the speaker identification unit 212 .

The gaze direction estimation unit 214 detects the head of the attendee identified by the attendee identification unit 211, and estimates the gaze direction of each attendee from the head position and head direction. The nodding detection unit 215 detects the nodding state of each attendee, which will be described later, from the vertical movement and amplitude of the head or face of the attendee identified by the gaze direction estimation unit 214 . The motion amount estimator 216 measures the motion amount of the head or face of the attendee identified by the gaze direction estimator 214 .

The smile estimation unit 217 calculates the degree of smile, which will be described later, for the facial regions of the attendees identified by the attendee identification unit 211 . The behavior analysis unit 210 only needs to be able to determine the behavior of each attendee during the conference. All of the unit 215, the motion amount estimation unit 216, and the smile estimation unit 217 may not be included.

The meeting time calculator 220 calculates the meeting time based on the analysis result of the behavior analysis unit 210 . The meeting score calculation unit 230 calculates a meeting score, which is an index of the productivity of the entire meeting, based on the analysis result of the behavior analysis unit 210 and the meeting time calculated by the meeting time calculation unit 220 . The contribution calculation unit 250 calculates the contribution of each attendee to the conference based on the analysis result of the behavior analysis unit 210 . The personal score calculator 260 calculates the personal score of each attendee based on the conference score calculated by the conference score calculator 230 and the contribution of each attendee calculated by the contribution calculator 250 . The storage unit 240 stores a data table referred to when the behavior analysis unit 210 or the like performs analysis, and the calculation results of the personal score calculation unit 260 or the like. The conference productivity visualization unit 270 is, for example, a liquid crystal monitor, display, etc., and displays the calculation results of the individual score calculation unit 260 and the like.

Next, the first half of the operation of the conference productivity visualization system according to this embodiment will be described with reference to FIG. First, the image acquisition unit 110 of the image acquisition device 100 acquires images of attendees (S201). The image acquisition device 100 is, for example, a camera, an infrared sensor, a terminal with an image capturing function, or the like.

FIG. 3 shows an example of an installation method of the image acquisition device 100 according to this embodiment. In FIG. 3, the image acquisition device 100 is placed on the desk so that as many attendees as possible can take pictures. Here, it is desirable to install the image acquisition device 100 at a position where all attendees can be photographed. Also, the number of image acquisition devices 100 is not limited to one, and a plurality of devices may be provided. When a plurality of image acquisition apparatuses 100 are provided, it is preferable to merge the imaging results by attaching the imaging time and the identification ID of the image acquisition apparatus to the acquired image data.

Next, the behavior analysis unit 210 analyzes the behavior of the attendee in the acquired image (S202). Details of a specific example of behavior analysis will be described later.

Subsequently, the behavior analysis unit 210 determines whether or not the conference has ended (S203). If the conference has not ended (No in S203), the process returns to S201. In this embodiment, when the input signal from the image acquisition unit 110 stops, for example, when the power of the image acquisition device 100 is turned off, it is determined that the conference has ended. Note that the behavior analysis unit 210, when it can be estimated that the movement of the attendees has changed from the acquired image, for example, when all the participants stand up, when it is determined that the inside of the conference room is cleaned up, etc. may be determined to have ended. If the conference has ended (Yes in S203), the process proceeds to the second half (A) of the operation of the conference productivity visualization system according to the present embodiment.

Here, detailed operation of behavior analysis according to the present embodiment will be described with reference to FIG. First, the attendee identifying unit 211 identifies a person area in the acquired image (S401). In this embodiment, a person area (face area) is detected from an image by using a known face detection technique. In face area detection, a face image and an image other than a face are prepared and a model is learned, and the similarity between the learned model and the detected image is calculated to calculate the likelihood of a face. There are well-known technologies for face region detection technology, and for example, (Non-Patent Document 1) may be used.
(Non-Patent Document 1) T. Watanabe et al.: Co-occurrence histograms of oriented gradients for pedestrian detection, 2009

Next, the attendee identification unit 211 identifies attendees for each detected face area (S402). In this embodiment, attendees are specified by performing matching processing with face dictionary data stored in the storage unit 240 in advance. The face dictionary data is stored in association with attendees of the conference and facial features of the attendees. In the matching process, the position of the feature point of the person's face area in the acquired image is specified, the feature amount is calculated from the feature point position, and then the feature amount is compared with the feature amount of the attendant's face stored in advance. It can be realized by There are known techniques for face matching processing, and for example, (Patent Document 5) may be used.
(Patent Document 5) JP-A-2013-6511 "Face Recognition Apparatus and Face Recognition Method"
In this embodiment, the user ID of the identified attendee is stored in the meeting history DB 240 a in the storage unit 240 . Also, the acquisition time of the image specified by each attendee is stored as the "start time" in association with the user ID of each attendee. Note that the time at which the attendees are identified, rather than the time at which the image was acquired, is set as the "start time". you can use any time.

Here, FIG. 5 shows an example of the meeting history DB 240a. In this embodiment, a meeting history table is generated for each meeting. For example, the conference history of the conference ID "P001" includes the "user ID" of the attendee identified as the attendee of the conference, the "start time" when each attendee started attending the conference, and the "end time" at which each attendee finished the meeting, "total amount of speech", "total number of times watched", "total number of nods", "total number of movements", "total number of smiles received", each attendance It has columns for storing "individual meeting time", "individual score", etc., which indicate the attendance time of the person to the meeting. Note that a meeting history table may be generated for each attendee (user) rather than for each meeting.

6 shows an example of the user table 240b in the storage unit 240. As shown in FIG. In this embodiment, attendees who can be attendees of the conference are registered in advance and given user IDs. The user table 240b stores the assigned "user ID", attendee's "name", attendee's attribute (belonging "organization", participating "project", participating "regular meeting", etc.). Have columns. The names and attributes of the attendees may be input by the administrator of the conference productivity visualization apparatus 200 or the like, or may be automatically referenced from another system such as an in-house telephone directory. Also, from the registered attributes, attendees may be specified by linking with the conference name, or attendance may be managed.

Furthermore, FIG. 7 shows an example of a conference table 240c in the storage unit 240. As shown in FIG. In this embodiment, the details of the conference are registered in advance. The conference table 240c stores the assigned "meeting ID", "meeting name", details of the meeting ("keyword" indicating the contents of the meeting, "period" of the meeting, required "attendees", etc.). have a column to The conference name and details may be input by the administrator of the conference productivity visualization device 200. For example, by text-analyzing a conference invitation email, etc., "meeting name", "keyword", "period", " It is also possible to register by estimating "attendees" or the like. Further, by referring to the user table 240b from the "meeting name" and "keyword", the "attendees" may be specified, and the attendance may be managed. It should be noted that it is not necessary to record the details of the conference by only assigning the conference ID to each conference.

Subsequently, the speaker identifying unit 212 extracts a mouth area from the face area detected in S301, and identifies the speaker based on the open/closed state of the mouth of the person (S403). Known techniques can also be used to extract the mouth region and the open/closed state of the mouth. For example, using [2], mouth opening/closing information may be obtained and the vertical distance between the upper and lower lips may be thresholded to identify the attendee who is speaking.
(Non-Patent Document 2) Yuasa et al., Digital makeup system based on high-precision facial feature point detection, SSII10, 2004
Also, using (Patent Document 6), it may be determined whether or not the person in the image is speaking.
(Patent Document 6) JP-A-9-307868 "Communication Device and Communication Method"

Next, the speech volume calculation unit 213 calculates the speech volume of the speaker identified by the speaker identification unit 212 (S404). In this embodiment, the utterance time of each speaker is measured from the utterance start time and utterance end time of each speaker, the measurement result is added for each speaker, and the addition result is stored in the storage unit 240 (S409). . The result of the measurement and addition is stored in the "total speech volume" column of the relevant conference history table in the conference history DB 240 in association with the user ID of the relevant attendee. As for the update timing (addition timing) of the speech volume, even when the speech of the speaker ends, the speech volume may be updated sequentially without waiting for the speech of the speaker to finish.

Also, the gaze direction estimation unit 214 estimates the gaze direction of each attendee from the head position and head direction detected from the image (S405). In the present embodiment, the position and direction of the head are estimated by applying the facial feature point positions used for facial feature amount calculation in S402 to a human 3D coordinate model. The position of the face detected by the attendee identification unit 211 may be substituted for the position of the head. Furthermore, (Non-Patent Document 3) may be used to estimate the position and orientation of the head more accurately.
(Non-Patent Document 3) T. Kozakaya et al.: Face Recognition by Projection-based 3D Normalization and Shading Subspace Orthogonalization, 2006
In this embodiment, the person that each attendee gazes at is estimated, and the number of times each attendee has been gazed at by other attendees is counted as the "total number of gazes". Here, instead of the number of gazes, the gaze time may be measured.

Next, the nodding detection unit 215 detects a nod for each attendee from the detected vertical movement amount and amplitude of the head or face (S406). Subsequently, the motion amount estimation unit 216 measures the detected motion amount of the head or face (S407). In this embodiment, the optical flow around the head region in the image is calculated, and the nodding and motion amount of the attendee are estimated from the sum of the lengths of the flow vectors of the head region. An inter-frame subtraction method or a background subtraction method may be used. Here, the nodding detection unit 215 and the motion amount estimation unit 216 may use the same algorithm for detecting the amount of motion, and the nod detection unit 215 mainly detects vertical motion of the face (that is, motion corresponding to a nod). , the number of times each attendee was nodded at another attendee is counted as the "total number of nods received". Whether or not each attendee was nodded by another attendee may be judged from the direction of gaze of the other attendee who nodded, You can judge that you have slipped. The motion amount estimator 216 also detects other motions and counts the number of times for each attendee. Note that the motion amount estimation unit 216 may include the function of the nodding detection unit 215 .

Finally, the measurement result is stored in the storage unit 240 (S409). The counted "total number of times of attention" is stored in the "total number of times of attention" column of the corresponding meeting history table in the meeting history DB 240 in association with the user ID of the corresponding attendee. In addition, in this embodiment, it is estimated that there was a "nod" and "movement" when a motion amount equal to or greater than a predetermined amount is detected, and the number of "nods" and "movements" is counted. The counted number of "nods" and "movements" are stored in the "total number of nods received" and "total number of movements" columns of the corresponding conference history table in the conference history DB 240 in association with the user ID of the attendee. Here, instead of the number of "nods" and "movements", a specific numerical value such as the sum of Flow vectors may be calculated, stored, or weighted. The timing for storing the "total number of gazes", "total number of nods received", and "total number of movements" is determined by the gaze direction estimator 214, the nod detector 215, and the motion amount estimator 216. It may be a degree, or it may be collectively stored at the end.

Furthermore, by calculating the similarity of the detected face region to the smile estimation model, it is estimated whether or not the face is a smile (S408). With known technology, it is possible to learn a model from a smiling face image and non-smiling face images, and to estimate whether or not the face is a smiling face by the same method as face area detection. In this embodiment, the number of times each attendee makes other attendees smile is counted and stored in the storage unit 240 (S409). The number of times each attendee made other attendees smile is counted as the "total number of smiles received." Whether or not each attendee made other attendees smile can be judged from the gaze direction of the other attendees when the attendee smiles, or when a certain attendee speaks when the attendee smiles. It may be judged that the attendees who are in attendance have smiled. The counted number of smiles is stored in the "total number of smiles received" column of the corresponding meeting history table in the meeting history DB 240 in association with the user ID of the corresponding attendee. Here, it is possible to count the number of times of responding with a smile instead of the number of times of making the person smile. In addition, it is also possible to calculate and store specific numerical values, such as estimating the degree of a smile as well as whether or not it is a smile, or weighting.

This concludes the detailed operation of the behavior analysis. Note that the above processing is performed on a detected person each time an image is input from the image acquisition unit 110 . In this way, if a large number of attendees are nodding their heads or staring at the speaker when a particular person is speaking, it is likely that the person is making a significant contribution to the conference. Conceivable. If it is possible to detect the states and attitudes of other attendees when a specific person is speaking, such as "total number of gazes", "total number of nods", "total number of smiles", etc., the productivity of the meeting will be more accurate. can be visualized.

Next, the second half of the operation of the conference productivity visualization system according to this embodiment will be described with reference to FIG. First, the meeting time calculation unit 220 refers to the meeting history DB 240a in the storage unit 240 and calculates the meeting time of the meeting (S801). In this embodiment, the meeting attendance time of the attendee is calculated by taking the difference between the "start time" and the "end time" of the meeting history DB 240a of each attendee. The calculated attendance time is stored in the "personal meeting time" column of the meeting history DB 240a. In addition, in this embodiment, the total sum of "personal meeting time" of all attendees is calculated as the meeting time.

Subsequently, the meeting score calculation unit 230 refers to the meeting history DB 240a in the storage unit 240 and evaluates the productivity of the meeting (S802). In this embodiment, the meeting score is calculated based on the meeting time. The conference score is a score obtained by converting the productivity of the conference based on the behavior of each attendee during the conference. The conference score calculation unit 230 evaluates the productivity of the conference from the behavior of the attendees analyzed by the behavior analysis unit 210 .

Here, detailed operation of the meeting score calculation unit 230 in S802 will be described using FIG. In S802, the meeting score is calculated based on the following procedure/index.

First, the total amount of utterances of attendees is calculated, and the conference score is corrected based on the ratio of the total amount of total utterances to the conference time (S901). In the present embodiment, the sum total of the speech volume of each attendee is calculated by referring to the meeting history DB 240a and calculating the sum of the "total speech volume" column of each attendee. Further, in the present embodiment, it is determined that active discussion is taking place when the sum of the amount of remarks for the meeting time is equal to or greater than the threshold. Therefore, the meeting time is multiplied by a correction value to correct the meeting score in the increasing direction. Conversely, when the sum of the amount of remarks for the meeting time is smaller than the threshold, it is determined that there is no lively discussion. Therefore, the meeting time is multiplied by a correction value to correct the meeting score in the decreasing direction. Here, by setting a threshold value, two types of correction values are prepared for "active discussion" and "no active discussion". A plurality of correction values may be used by preparing an interval size such as "discussed mediocrely", "not actively discussed", and the like. Further, the correction may be made based on the absolute amount of the sum total of the total speech volume, instead of the ratio to the conference time. Note that a separate table is stored in the storage unit 240 regarding the correspondence relationship between the ratio of the total speech volume to the conference time and the correction value.

Here, FIG. 10 shows an example of a correction value table in which correction values are set. FIG. 10 is a correction value table 240d showing the correspondence relationship between the ratio of the total speech volume to the conference time and the correction value. In the present embodiment, if the ratio of the total amount of speech to the meeting time is "less than 0.3", it is "0.5", if it is "0.3 to 0.8", it is "1.0" "greater than 0.8" indicates "2.0". As long as the productivity of the meeting can be shown, the threshold value and the step size of the correction value table may be set in any way. Also, depending on the item, it is not necessary to provide a correction value.

Subsequently, the conference score is further corrected based on the bias of the speech volume of each attendee (S902). In this embodiment, the conference score is corrected based on the ratio of the total speech volume of each attendee to the total speech volume. When the percentage of the total amount of speech of each attendee does not change greatly for each attendee, that is, when the bias in the amount of speech of each attendee is small, each attendee actively participates in the conference. , the meeting score is corrected in the increasing direction. Conversely, when the ratio of the speech volume of some of the attendees is greater than that of the other attendees, it is determined that the speech is biased, and the conference score is corrected downward. A separate table is also prepared for the correspondence relationship between the bias and the correction value.

Next, the conference score is further corrected based on the degree of smile of the attendee (S903). In this embodiment, the sum total of the total number of smiling faces of each attendee in the meeting history DB 240a is calculated as the degree of smile. If the degree of smile is equal to or greater than the threshold, it is determined that the meeting is being held in a good atmosphere, and the meeting score is corrected to increase. On the other hand, if it is lower than the threshold, the conference score is corrected downward. Here, instead of the number of smiles, the sum may be calculated from specific numerical values relating to the degree of smiles. Also, instead of the sum, the average of attendees may be used as the degree of smile. A separate table is also prepared for the correspondence between the sum or average value and the correction value.

Subsequently, the conference score is further corrected based on the degree of attention of the attendees (S904). In the present embodiment, the sum total of the number of times each attendee has been watched in the meeting history DB 240 is calculated as the degree of attention. A separate table is also prepared for the correspondence relationship between the gaze degree and the correction value. If the total number of gazes is greater than or equal to the threshold, it is determined that the attendees are concentrating on the meeting, and the meeting score is corrected to increase. On the other hand, if it is less than the threshold, the conference score is corrected downward. Here, the meeting score may be corrected using the degree of matching of the line-of-sight direction of each attendee as the degree of gaze. The degree of coincidence is calculated, for example, by finding the intersection or the shortest distance between straight lines in the three-dimensional space generated in the head direction with the head position of each attendee as the starting point. Alternatively, the 2D Hough transform may be extended to 3D, the straight lines may be voted in a 3D space, and the maximum vote value may be calculated as the matching score. In this case, in addition to the meeting history DB 240a of each attendee, a gaze degree table (not shown) is prepared. The degree-of-gaze table stores, for example, the degree of matching of line-of-sight directions based on time.

Next, the total amount of movement of each attendee is calculated, and the conference score is corrected (S905). In the present embodiment, the total number of times of movement in the meeting history DB 240a is calculated and used as the total amount of movement. When the sum is large, it is determined that the activity level of the meeting is high, and the meeting score is corrected to increase, and when the sum is small, the meeting score is corrected to decrease. Here, the sum may be calculated from a specific numerical value of the amount of motion instead of the number of motions. Also, instead of the sum, the average of the attendees may be used as the sum of the motion amounts. A separate table is also prepared for the correspondence relationship between the amount of motion and the correction value.

The above is the detailed operation of the meeting score calculation unit 230 in S802. Note that the threshold value or step size of the correction value may be normalized by the meeting time. For example, when the correction value is 100, if you set "meeting time" = "meeting score", the correction value calculated by each indicator will be calculated as the correction value / 100 x meeting score. Calculations allow correction of the actual values. Also, the weight of each index can be freely set by the administrator of the conference productivity visualization device 200 or the like. For example, if it is determined that the total amount of remarks by attendees is important, set the weight to "2" and set the weight to "1" for the other four indicators. If you take , you can double the importance of the total amount of remarks compared to other indicators. In addition, when all the above indices are standard values, "meeting score"="meeting time". Note that the meeting score may be corrected in any manner as long as it can indicate the productivity of the meeting.

Next, the contribution calculation unit 250 refers to the meeting history DB 240a in the storage unit 240, and evaluates the contribution of each attendee to the meeting (S803). Contribution is a numerical representation of how much each attendee contributed to the conference. In this embodiment, the contribution of each attendee is evaluated from the behavior of the attendee analyzed by the behavior analysis unit 210 .

Here, detailed operation of the contribution calculation unit 250 in S803 will be described with reference to FIG. In S803, the degree of contribution is calculated based on the following procedure/index.

First, the degree of contribution of each attendee is set to the "utterance rate" (S1101). The utterance rate is the ratio of the total utterance volume of each attendee to the total total utterance volume of all attendees. Alternatively, it may be the ratio of the total speech volume of each attendee to the meeting time.

Subsequently, the degree of contribution is corrected based on the number of times each attendee nodded at the other attendees (S1102). In this embodiment, the degree of contribution of each attendee is corrected based on the “total number of nods received” of each attendee in the meeting history DB 240 . If a certain attendee's "total number of nods received" is equal to or greater than a threshold, the attendee's degree of contribution is corrected to increase. On the other hand, if it is less than the threshold, the contribution is corrected downward. A separate table is also prepared for the correspondence relationship between the total number of nods received and the correction value.

Next, the degree of contribution is further corrected based on the number of times each attendee made other attendees smile (S1103). In this embodiment, the degree of contribution of each attendee is corrected based on the "total number of smiles received" of each attendee in the meeting history DB 240 . When the "total number of smiles received" of a certain attendee is equal to or greater than the threshold, since more attendees have smiled, the degree of contribution of the attendee is corrected in an increasing direction. On the other hand, if it is less than the threshold, the contribution is corrected downward. Here, instead of the number of smiles, correction may be performed based on a specific numerical value relating to the degree of smile. A separate table is also prepared for the correspondence relationship between the total number of smiles received and the correction value.

Finally, the degree of contribution is further corrected based on the number of times each attendee has been watched by other attendees (S1104). In this embodiment, the degree of contribution of each attendee is corrected based on the "total number of times watched" of each attendee in the meeting history DB 240 . When a certain attendee's "total number of times of attention" is equal to or greater than a threshold value, the attendee's degree of contribution is corrected in an increasing direction, assuming that the attendee has received more attention. On the other hand, if it is less than the threshold, the contribution is corrected downward. Here, instead of the number of gazes, the correction may be made based on the gaze time. A separate table is also prepared for the correspondence relationship between the total number of smiles received and the correction value.

The above is the detailed operation of the contribution calculation unit 250 in S803. Note that in the present embodiment, the correction value is defined as a multiplying factor for the initially set speech rate. For example, if the speech rate is set to "0.5" and the correction value is set to "1.5", the contribution will be "0.75". If it is a standard evaluation result, "contribution"="speech rate" remains unchanged. Also, the weight of each index can be freely set by the administrator of the conference productivity visualization device 200 or the like. For example, if it is determined that the number of times nodded is important, the weight is set to "2", and for the other two indicators, the weight is set to "1", and the weighted average of each correction value is calculated using the above weights. For example, it is possible to make the importance of the total amount of speech “double” that of other indicators. Note that the contribution may be corrected in any way as long as it can indicate the production of each attendee's meeting.

Subsequently, the personal score calculator 260 calculates the personal score of each attendee (S804). The individual score is a score indicating how much each attendee contributed to the conference. In this embodiment, the personal score of each attendee is calculated by multiplying the "meeting score" by the ratio of each attendee's "contribution".

FIG. 12 shows an example of a calculation image of the personal score in this embodiment. The graph shown in FIG. 12 is a calculation image of the personal score of the meeting ID "P001", (a) is "meeting time", (b) is "meeting score", and (c) is "personal score". there is First, for the "meeting time" of the meeting ID "P001", the meeting history in the meeting history DB 240a is referred to, and the "personal meeting time" of the user IDs "S001", "D004", "S002", and "M003" who are attendees. By adding them together, "4h" is calculated. (a) The graph of the meeting time is corrected by the meeting score calculator 230, and the calculated meeting score is shown in the graph (b). In this embodiment, the "meeting score" is corrected to "8.0", which is 2.0 times the "meeting time". Further, graph (c) shows the personal score of each attendee calculated by multiplying the "meeting score" by the ratio of "contribution". In this embodiment, the contributions of "Taro Toshiba", "Hanako Nishishiba", "Jiro Kitashiba" and "Saburo Minamishiba" are calculated as "2.5", "4.0", "0.5" and "1.0", respectively. For example, the individual score of “Hanako Nishishiba” with the user ID “D004” is the conference score “8.0” and the contribution ratio “4.0/(2.5+4.0+0.5+1.0)”. )” is calculated as “4.0”. The calculated "personal score" is stored in the "personal score" column of the meeting DB 240a in association with the user ID. The personal score may be set so that "meeting score" = "sum of individual scores of each attendee", or how it is calculated if it is possible to show the productivity of each attendee's meeting. I don't mind.

In this embodiment, the meeting time, meeting score, contribution, and personal score are calculated after the meeting ends. may be sequentially updated, and the individual score may be updated at the timing when the behavior analysis result is updated. In that case, the behavior analysis result of the behavior analysis unit 210 does not have to be stored in the storage unit 240 .

Finally, the meeting productivity visualization unit 270 visualizes the calculated meeting time, individual score, etc. by graphically displaying them (S805). In this embodiment, visualization is performed by drawing a graph as shown in FIG. 13 . FIG. 13 is a graph of meeting time (that is, individual meeting time) and meeting score (that is, individual score) for each user with reference to the meeting history DB 240a. In this example, the cumulative meeting hours and meeting scores for the first half of fiscal year 2017 are graphed, but the cumulative period may be set freely. Also, the meeting time and meeting score may be displayed for each meeting, and the meeting time and meeting score totaled for each organization or each participating project may be displayed by referring to the user table 240b. In this way, a graph is created for each individual, each meeting, each organization, each project, or the like, for each attribute of attendees and each detail of the meeting.

The methods described in the above embodiments can be applied to programs that can be executed by a computer, such as magnetic disks (floppy (registered trademark) disks, hard disks, etc.), optical disks (CD-ROM, DVD, etc.), magneto-optical disks ( MO), or stored in a storage medium such as a semiconductor memory for distribution.

Here, the storage medium may be of any form as long as it can store the program and is readable by the computer.

In order to realize this embodiment, the OS (operating system) running on the computer based on the instructions of the program installed on the computer from the storage medium, MW (middleware) such as database management software, network software, etc. You may perform a part of each process of .

Furthermore, the storage medium in this embodiment is not limited to a medium independent of a computer, and includes a storage medium that stores or temporarily stores a downloaded program transmitted via a LAN, the Internet, or the like.

In addition, the storage medium is not limited to one, and a case where the processing in this embodiment is executed from a plurality of media is also included in the storage medium in this embodiment, and the medium configuration may be any configuration.

Note that the computer in this embodiment executes each process in this embodiment based on a program stored in a storage medium. Any configuration such as a system or the like may be used.

Further, each storage device of this embodiment may be implemented by one storage device or may be implemented by a plurality of storage devices.

In addition, the computer in this embodiment is not limited to a personal computer, but includes arithmetic processing units, microcomputers, etc. included in information processing equipment, and is a general term for equipment and devices that can realize the functions of this embodiment by a program. ing.

Although several embodiments of the invention have been described above, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the gist of the description. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

DESCRIPTION OF SYMBOLS 100... Image acquisition apparatus 110... Image acquisition part 200... Conference productivity visualization apparatus 210... Behavior analysis part 211... Attendee identification part 212... Speaker identification part 213... Speech amount calculation part 214... Gaze direction estimation part 215... Nodding detection Unit 216: Motion amount estimation unit 217: Smile estimation unit 220: Meeting time calculation unit 230: Meeting score calculation unit 240: Storage unit 240a: Meeting history DB
240b... User table 240c... Conference table 240d... Correction value table 250... Contribution calculation unit 260... Individual score calculation unit 270... Conference productivity visualization unit

Claims (17)

A meeting productivity visualization device for visualizing the productivity of a meeting from an image,
a behavior analysis unit that identifies attendees of the conference based on the image, and analyzes the behavior of other attendees when a certain attendee is speaking;
a meeting time calculation unit that calculates the attendance time of each attendee to the meeting as personal meeting time, and calculates the meeting time from the sum of the personal meeting times of all attendees;
a meeting score calculation unit that calculates a meeting score for evaluating the productivity of the meeting based on the analysis result of the behavior analysis unit and the calculated meeting time;
a contribution calculation unit that calculates the contribution of each attendee to the conference based on the analysis result of the behavior analysis unit;
a personal score calculation unit that calculates a personal score based on the calculated meeting score and contribution;
and a meeting productivity visualization unit that illustrates the calculated meeting score or personal score ,
The meeting score calculation unit corrects the meeting score based on the smile level of the attendees.
Conference productivity visualization device. A meeting productivity visualization device for visualizing the productivity of a meeting from an image,
a behavior analysis unit that identifies attendees of the conference based on the image, and analyzes the behavior of other attendees when a certain attendee is speaking;
a meeting time calculation unit that calculates the attendance time of each attendee to the meeting as personal meeting time, and calculates the meeting time from the sum of the personal meeting times of all attendees;
a meeting score calculation unit that calculates a meeting score for evaluating the productivity of the meeting based on the analysis result of the behavior analysis unit and the calculated meeting time;
a contribution calculation unit that calculates the contribution of each attendee to the conference based on the analysis result of the behavior analysis unit;
a personal score calculation unit that calculates a personal score based on the calculated meeting score and contribution;
and a meeting productivity visualization unit that illustrates the calculated meeting score or personal score,
The contribution calculation unit corrects the contribution based on the degree of smile of other attendees.
Conference productivity visualization device. A meeting productivity visualization device for visualizing the productivity of a meeting from an image,
a behavior analysis unit that identifies attendees of the conference based on the image, and analyzes the behavior of other attendees when a certain attendee is speaking;
a meeting time calculation unit that calculates the attendance time of each attendee to the meeting as personal meeting time, and calculates the meeting time from the sum of the personal meeting times of all attendees;
a meeting score calculation unit that calculates a meeting score for evaluating the productivity of the meeting based on the analysis result of the behavior analysis unit and the calculated meeting time;
a contribution calculation unit that calculates the contribution of each attendee to the conference based on the analysis result of the behavior analysis unit;
a personal score calculation unit that calculates a personal score based on the calculated meeting score and contribution;
and a meeting productivity visualization unit that illustrates the calculated meeting score or personal score,
The conference score calculation unit corrects the conference score based on at least one of the bias in the amount of speech of each attendee and the sum of the amount of movement of each attendee.
Conference productivity visualization device. A meeting productivity visualization device for visualizing the productivity of a meeting from an image,
a behavior analysis unit that identifies attendees of the conference based on the image, and analyzes the behavior of other attendees when a certain attendee is speaking;
a meeting time calculation unit that calculates the attendance time of each attendee to the meeting as personal meeting time, and calculates the meeting time from the sum of the personal meeting times of all attendees;
a meeting score calculation unit that calculates a meeting score for evaluating the productivity of the meeting based on the analysis result of the behavior analysis unit and the calculated meeting time;
a contribution calculation unit that calculates the contribution of each attendee to the conference based on the analysis result of the behavior analysis unit;
a personal score calculation unit that calculates a personal score based on the calculated meeting score and contribution;
and a meeting productivity visualization unit that illustrates the calculated meeting score or personal score,
The contribution calculation unit corrects the contribution based on at least one of whether or not other attendees nodded to a certain attendee and the degree of attention of other attendees to a certain attendee.
Conference productivity visualization device. The conference productivity visualization device includes a storage unit,
The behavior analysis unit
an attendee identifying unit that identifies attendees by performing matching processing between a person detected from the received image and face dictionary data stored in advance in the storage unit;
a speaker identifying unit that detects the movement of the mouth of the attendee from the received image and identifies the attendee who is speaking;
a utterance volume calculation unit that calculates the utterance volume during the meeting time of each identified attendee;
5. The gaze direction estimation unit for estimating each gaze direction from the head position and head direction of each attendee identified by the attendee identification unit. Conference productivity visualization device. The conference productivity visualization device includes a storage unit,
The behavior analysis unit
an attendee identifying unit that identifies attendees by performing matching processing between a person detected from the received image and face dictionary data stored in advance in the storage unit;
a speaker identifying unit that detects the movement of the mouth of the attendee from the received image and identifies the attendee who is speaking;
a utterance volume calculation unit that calculates the utterance volume during the meeting time of each identified attendee;
5. The conference productivity visualization device according to any one of claims 1 to 4 , further comprising a nod detection unit that detects whether or not a nod is made from time displacement of head direction of each attendee. The conference productivity visualization device includes a storage unit,
The behavior analysis unit
an attendee identifying unit that identifies attendees by performing matching processing between a person detected from the received image and face dictionary data stored in advance in the storage unit;
a speaker identifying unit that detects the movement of the mouth of the attendee from the received image and identifies the attendee who is speaking;
a utterance volume calculation unit that calculates the utterance volume during the meeting time of each identified attendee ;
The conference productivity visualization device according to any one of claims 1 to 4 , further comprising a motion estimating unit that estimates the amount of motion of the attendee from the temporal displacement of the head position of each attendee. The conference productivity visualization device includes a storage unit,
The behavior analysis unit
an attendee identifying unit that identifies attendees by performing matching processing between a person detected from the received image and face dictionary data stored in advance in the storage unit;
a speaker identifying unit that detects the movement of the mouth of the attendee from the received image and identifies the attendee who is speaking;
a utterance volume calculation unit that calculates the utterance volume during the meeting time of each identified attendee;
The conference productivity visualization device according to any one of claims 1 to 4 , further comprising a smile estimation unit that estimates whether each attendee is smiling. The conference productivity visualization unit
Aggregate personal meeting time or personal score for each attendee, each meeting, each attribute of attendee, or each detail of meeting, and display the total result in a graph;
The conference productivity visualization device according to any one of claims 1 to 8 . A meeting productivity visualization method for visualizing the productivity of a meeting from an image,
identifying attendees of the meeting based on the image, and analyzing the behavior of other attendees when one attendee is speaking;
calculating the meeting attendance time of each attendee as a personal meeting time, and calculating the meeting time from the sum of the personal meeting times of all attendees;
calculating a meeting score for evaluating the productivity of the meeting based on the behavior analysis result and the calculated meeting time;
calculating the degree of contribution of each attendee to the conference based on the analysis result of the behavior;
calculating a personal score based on the calculated meeting score and contribution;
Graphing the calculated conference score or and individual score;
The step of calculating the conference score includes a process of correcting the conference score based on the degree of smile of the attendees,
Meeting productivity visualization method. A meeting productivity visualization method for visualizing the productivity of a meeting from an image,
identifying attendees of the meeting based on the image, and analyzing the behavior of other attendees when one attendee is speaking;
calculating the meeting attendance time of each attendee as a personal meeting time, and calculating the meeting time from the sum of the personal meeting times of all attendees;
calculating a meeting score for evaluating the productivity of the meeting based on the behavior analysis result and the calculated meeting time;
calculating the degree of contribution of each attendee to the conference based on the analysis result of the behavior;
calculating a personal score based on the calculated meeting score and contribution;
Graphing the calculated conference score or and individual score;
The step of calculating the contribution includes a process of correcting the contribution based on the degree of smile of other attendees,
Meeting productivity visualization method. A meeting productivity visualization method for visualizing the productivity of a meeting from an image,
identifying attendees of the meeting based on the image, and analyzing the behavior of other attendees when one attendee is speaking;
calculating the meeting attendance time of each attendee as a personal meeting time, and calculating the meeting time from the sum of the personal meeting times of all attendees;
calculating a meeting score for evaluating the productivity of the meeting based on the behavior analysis result and the calculated meeting time;
calculating the degree of contribution of each attendee to the conference based on the analysis result of the behavior;
calculating a personal score based on the calculated meeting score and contribution;
Graphing the calculated conference score or and individual score;
The step of calculating the conference score includes a process of correcting the conference score based on at least one of the bias in the amount of speech of each attendee and the sum of the amount of movement of each attendee.
Meeting productivity visualization method. A meeting productivity visualization method for visualizing the productivity of a meeting from an image,
identifying attendees of the meeting based on the image, and analyzing the behavior of other attendees when one attendee is speaking;
calculating the meeting attendance time of each attendee as a personal meeting time, and calculating the meeting time from the sum of the personal meeting times of all attendees;
calculating a meeting score for evaluating the productivity of the meeting based on the behavior analysis result and the calculated meeting time;
calculating the degree of contribution of each attendee to the conference based on the analysis result of the behavior;
calculating a personal score based on the calculated meeting score and contribution;
Graphing the calculated conference score or and individual score;
The step of calculating the degree of contribution corrects the degree of contribution based on at least one of whether or not other attendees have nodded at the attendee and the degree of attention of other attendees toward the attendee. including processing to
Meeting productivity visualization method. A program executed by a conference productivity visualization device that visualizes conference productivity from images,
a behavior analysis function that identifies the attendees of the conference based on the image and analyzes the behavior of other attendees while a certain attendee is speaking;
A meeting time calculation function that calculates the attendance time of each attendee to the meeting as personal meeting time and calculates the meeting time from the sum of the personal meeting times of all attendees;
a meeting score calculation function for calculating a meeting score for evaluating the productivity of the meeting based on the analysis result of the behavior analysis function and the calculated meeting time;
a contribution calculation function that calculates the contribution of each attendee to the conference based on the analysis result of the behavior analysis function;
a personal score calculation function that calculates a personal score based on the calculated meeting score and contribution;
a meeting productivity visualization function that graphically displays the calculated meeting score or personal score, and
The meeting score calculation function corrects the meeting score based on the degree of smiling of attendees.
Meeting productivity visualization program. A program executed by a conference productivity visualization device that visualizes conference productivity from images,
a behavior analysis function that identifies the attendees of the conference based on the image and analyzes the behavior of other attendees while a certain attendee is speaking;
A meeting time calculation function that calculates the attendance time of each attendee to the meeting as personal meeting time and calculates the meeting time from the sum of the personal meeting times of all attendees;
a meeting score calculation function for calculating a meeting score for evaluating the productivity of the meeting based on the analysis result of the behavior analysis function and the calculated meeting time;
a contribution calculation function that calculates the contribution of each attendee to the conference based on the analysis result of the behavior analysis function;
a personal score calculation function that calculates a personal score based on the calculated meeting score and contribution;
a meeting productivity visualization function that graphically displays the calculated meeting score or personal score, and
The contribution calculation function corrects the contribution based on the degree of smile of other attendees.
Meeting productivity visualization program. A program executed by a conference productivity visualization device that visualizes conference productivity from images,
a behavior analysis function that identifies the attendees of the conference based on the image and analyzes the behavior of other attendees while a certain attendee is speaking;
A meeting time calculation function that calculates the attendance time of each attendee to the meeting as personal meeting time and calculates the meeting time from the sum of the personal meeting times of all attendees;
a meeting score calculation function for calculating a meeting score for evaluating the productivity of the meeting based on the analysis result of the behavior analysis function and the calculated meeting time;
a contribution calculation function that calculates the contribution of each attendee to the conference based on the analysis result of the behavior analysis function;
a personal score calculation function that calculates a personal score based on the calculated meeting score and contribution;
a meeting productivity visualization function that graphically displays the calculated meeting score or personal score, and
The conference score calculation function corrects the conference score based on at least one of the bias in the amount of speech of each attendee and the sum of the amount of movement of each attendee.
Meeting productivity visualization program. A program executed by a conference productivity visualization device that visualizes conference productivity from images,
a behavior analysis function that identifies the attendees of the conference based on the image and analyzes the behavior of other attendees while a certain attendee is speaking;
A meeting time calculation function that calculates the attendance time of each attendee to the meeting as personal meeting time and calculates the meeting time from the sum of the personal meeting times of all attendees;
a meeting score calculation function for calculating a meeting score for evaluating the productivity of the meeting based on the analysis result of the behavior analysis function and the calculated meeting time;
a contribution calculation function that calculates the contribution of each attendee to the conference based on the analysis result of the behavior analysis function;
a personal score calculation function that calculates a personal score based on the calculated meeting score and contribution;
a meeting productivity visualization function that graphically displays the calculated meeting score or personal score, and
The contribution calculation function corrects the contribution based on at least one of whether or not other attendees nodded to a certain attendee and the degree of attention of other attendees to a certain attendee.
Meeting productivity visualization program.

Images