JP5441071B2 - Face analysis device, face analysis method, and program - Google Patents

Description

  The present invention relates to a face analysis device that recognizes the faces of one or more students in a lecture and analyzes the faces in the lecture.

Conventionally, from the entrance / exit information database server that receives personal identification information transmitted from the students who enter and leave the classroom and the instructor (lecturer), and stores this in the entrance / exit information database as entrance / exit information, and the entrance / exit information database , A Web server having a number confirmation program for acquiring classroom information for each student and instructor (lecturer), and an administrator who manages the corresponding classroom when the corresponding class exists by the number confirmation program There has been known a classroom management system including a mail server that transmits an e-mail to the Internet (for example, see Patent Document 1).

JP 2009-129117 A (first page, FIG. 1 etc.)

  However, in the past, there has been a problem that it is impossible to objectively analyze the attendance status of students (for example, students) during the lecture, and as a result, it is not possible to perform an appropriate analysis on the lecture. It was.

  The face analysis apparatus of the present invention recognizes a student's face from a student image storage unit that stores a student image, which is a moving image obtained by capturing one or more students' faces during a lecture, and a student image. The face analysis apparatus includes a face analysis unit that performs analysis on the recognized face, and an output unit that outputs information related to the analysis result by the face analysis unit.

  With this configuration, it is possible to analyze the face of one or more students in the lecture recognized from the student image and perform appropriate analysis on the lecture. In addition, since the student's face is automatically recognized from the student image and analyzed, it is not necessary to look at the student's face during the analysis, and the student's privacy can be ensured.

  Further, the face analysis device of the present invention further includes a statistical analysis unit that performs a statistical analysis specified in advance using the analysis result acquired by the face analysis unit in the face analysis device, and the output unit performs statistical analysis. It is a face analysis device that outputs a result.

  With this configuration, it is possible to output a statistical analysis result on the analysis result of the faces of one or more students in the lecture.

  Further, the face analysis device of the present invention includes a statistical analysis unit that performs a statistical analysis specified in advance using the analysis result acquired by the face analysis unit in the face analysis device, and a statistical analysis result of the statistical analysis unit. In addition, a lecture evaluation unit that evaluates a lecture is further provided, and the output unit is a face analysis device that outputs an evaluation result of the lecture evaluation unit.

  With this configuration, it is possible to output the evaluation result for the lecture using the statistical analysis result.

  The face analysis device of the present invention is the face analysis device, wherein the face analysis device includes at least one of a lecturer image that is a moving image of a lecturer during a lecture or lecturer voice information that is voice information of a lecturer during a lecture. A lecturer information storage unit for storing lecturer information and a change detection unit for detecting changes in the lecturer information, and the face analysis unit is a class acquired immediately after the change detection unit detects a change. The face analysis apparatus acquires a face analysis result for the raw image, and the output unit outputs information related to the face analysis result for the student raw image acquired immediately after the change detection unit detects the change.

  With this configuration, it is possible to perform a more appropriate analysis regarding the lecture by further utilizing the change in the lecturer information.

  Further, the face analysis device of the present invention is a related statistics for performing statistical analysis specified in advance using the analysis result for the student raw image acquired immediately after the change detection unit detects the change in the face analysis device. The face analysis apparatus further includes an analysis unit, and the output unit outputs a result of statistical analysis of the related statistical analysis unit.

  With this configuration, it is possible to output a statistical analysis result regarding the analysis result of one or more students' faces when a change occurs in the lecturer information.

  Further, the face analysis device of the present invention is a related statistics for performing statistical analysis specified in advance using the analysis result for the student raw image acquired immediately after the change detection unit detects the change in the face analysis device. An analysis unit and a related lecture evaluation unit that evaluates a lecture using the statistical analysis result of the related statistical analysis unit are further provided, and the output unit is a face analysis device that outputs an evaluation result of the related lecture evaluation unit.

  With this configuration, it is possible to output a more appropriate evaluation result for a lecture by using a result of statistical analysis of a face analysis result when a change occurs in lecturer information.

  Further, in the face analysis device of the present invention, in the face analysis device, the face analysis unit acquires the analysis result of the face of the student along the time series, and the output unit displays the analysis result of the face of the face analysis unit. It is a face analyzer that outputs in time series.

  With this configuration, face analysis results can be output in chronological order.

  The face analysis device of the present invention further includes a designation receiving unit that receives designation for the analysis result of the face at one time point that is output by the output unit in the face analysis device, and the output unit is designated by the designation receiving unit. It is a face analysis device that outputs a student raw image at a time corresponding to a received analysis result.

  With this configuration, it is possible to easily output the student image corresponding to the designated analysis result.

  Further, in the face analysis device of the present invention, in the face analysis device, when the output unit outputs the student image, the display mode is such that the student's face cannot be identified by the student. This is a face analysis device that outputs the trained raw images.

  With this configuration, it is possible to protect the student's privacy by preventing the student from being identified from the displayed student image.

  According to the face analysis apparatus and the like according to the present invention, it is possible to perform an appropriate analysis regarding a lecture.

Block diagram of face analysis apparatus according to Embodiment 1 of the present invention Flow chart explaining the operation A diagram schematically showing part of the frame image of the student image Schematic diagram showing a frame image with the same face recognized The figure which shows the same face analysis result management table Figure showing the statistical analysis result management table Figure showing the display example Figure showing the display example Diagram showing the display example Block diagram of face analysis apparatus according to Embodiment 2 of the present invention Flow chart explaining the operation Figure showing a part of the lecturer information Figure showing the same time analysis result management table Figure showing the statistical analysis result management table The figure which shows an example of the external appearance of the computer system in each embodiment of this invention The figure which shows an example of a structure of the computer system

  Hereinafter, embodiments of the face analysis device and the like will be described with reference to the drawings. In addition, since the component which attached | subjected the same code | symbol in embodiment performs the same operation | movement, description may be abbreviate | omitted again.

(Embodiment 1)
FIG. 1 is a block diagram of face analysis apparatus 1 in the present embodiment.

  The face analysis device 1 includes a first photographing unit 11, a student raw image storage unit 12, a face analysis unit 13, a statistical analysis unit 14, a lecture evaluation unit 15, a designation receiving unit 16, and an output unit 17.

  The 1st imaging | photography part 11 image | photographs the face of one or more students in a lecture, and acquires the image of the face of one or more students in a lecture. The face image of one or more students means, for example, an image including at least one student's face. It is preferable that the 1st imaging | photography part 11 acquires the image containing a several student's face. For example, the first photographing unit 11 performs photographing from the front side of one or more students and acquires a moving image. Lecture is a concept including classes at elementary and junior high schools, high schools, universities, cram schools, prep schools, and the like. The student is, for example, a student taking a lecture. A student may be considered a student. In addition, as long as the students are participating in the lecture, it does not matter whether they apply for the lecture or have qualifications. The first imaging unit 11 captures a moving image. Note that a plurality of still images taken continuously at a predetermined or indefinite timing designated in advance may be considered as moving images here. The same applies to the following. The first imaging unit 11 accumulates the captured moving images in, for example, the student raw image storage unit 12. The first imaging unit 11 acquires a moving image associated with a time series. For example, a moving image composed of frame images associated with so-called time codes is acquired. A frame image may be considered as each still image constituting a moving image. The 1st imaging | photography part 11 is realizable with a camera, for example.

  The student raw image storage unit 12 stores student raw images. The student image is a moving image obtained by photographing one or more students' faces during a lecture. For example, the student raw image storage unit 12 stores a student raw image captured by the first imaging unit 11. The student image is preferably an image obtained by photographing a plurality of students' faces. The student image may be composed of a plurality of moving images obtained by dividing a plurality of students into a plurality of areas. Two or more frame images and still images constituting the student image are usually associated with a time series, for example. For example, each frame image or still image constituting the student raw image is associated with, for example, a time code indicating the shooting time. The file format of the student image and the compression method are not limited. Also, the size and aspect ratio of the image do not matter. For example, the student raw image is a moving image in a format such as MPEG2, MPEG4, motion JPEG, AVCHD, or DV. The time here may be an absolute time such as a standard time or a relative time such as an elapsed time from the start of imaging. The same applies to the following. The storage here is a concept including temporary storage. The student raw image storage unit 12 is preferably a non-volatile recording medium, but can also be realized by a volatile recording medium. The same applies to other storage units.

  The face analysis unit 13 recognizes one or more students' faces from the student raw images stored in the student raw image storage unit 12 and analyzes the recognized faces. For example, the face analysis unit 13 analyzes, for example, the face of each student recognized from the student image, and closes the eyes, blinking (flapping) the facial expression, direction, line-of-sight direction of each face. Or information indicating an analysis result of an analysis item such as an eye open. The analysis result acquired here may be considered as a value indicating a feature amount for each analysis item, for example. This value may be a numerical value, a character string indicating the analysis result, or the like. The face analysis result may be, for example, information indicating the facial expression, orientation, line of sight, and blinking obtained by the analysis. The facial expression is, for example, an expression such as laughter, anger, or sadness. The face analysis unit 13 detects, for example, one or more faces for consecutive frame images before or after one time point (for example, time) of the student raw images, and the face images detected at substantially the same place are detected. May be acquired between the frame images to obtain information indicating the facial expression of the student and the blinking situation at one time point (for example, time). In addition, one or more faces are detected for each frame image at one time (for example, time) of the student image, and information indicating the face direction, line of sight, etc. is acquired by analyzing the face image. May be. Processing for recognizing a face from a moving image and processing for analyzing the recognized face to acquire facial expression, orientation, line of sight, and blinking information are known as face detection technology, facial expression recognition technology, etc. Therefore, detailed description is omitted here. For example, please refer to the following non-patent documents ("Sony Japan | Technical Information Sony's Technology | View from Technology / Theme | Facial Image Recognition", [online], [Search August 23, 2011], Internet <URL: http://www.sony.co.jp/SonyInfo/technology/technology/theme/sface_01.html>).

  For example, the face analysis unit 13 acquires the analysis results of the student's face at one or more time points, preferably at a plurality of time points (for example, time points), using the student image. The time point here may be considered to mean a frame image at one time point of the student image. The same applies to the following. For example, the face analysis unit 13 recognizes one or more students' faces for each of the frame images at a plurality of time points constituting the student image, and for each recognized student's face, for each frame image Get analysis results. When acquiring the face analysis result for the frame image at each time point, the frame image before or after each frame may be used as appropriate. The same applies to the following. The plurality of time points are, for example, a plurality of times at predetermined time intervals within a time when the student image is taken, or a plurality of times and frame images specified by the user. For example, the face analysis unit 13 may sequentially acquire the analysis results for every n seconds (n is an integer equal to or greater than 1) for the student image that is a moving image.

  The statistical analysis unit 14 performs statistical analysis designated in advance using the analysis result acquired by the face analysis unit 13. The statistical analysis unit 14 may perform any statistical analysis. For example, the statistical analysis unit 14 may individually perform statistical analysis for each of one or more time points of the student image, or may comprehensively analyze the entire student image corresponding to one lecture or a part thereof. Statistical analysis may be performed. The statistical analysis unit 14 may perform statistical analysis on the analysis result for each analysis item acquired by the face analysis unit 13 or use the analysis result for each analysis item acquired by the face analysis unit 13. Then, a comprehensive analysis result may be acquired and statistical analysis may be performed on the comprehensive analysis result.

  For example, the statistical analysis unit 14 performs, for each analysis item (for example, for each expression, each direction, each line of sight, blinking), for each analysis result acquired by the face analysis unit 13 for each of one or more time points of the moving image. The number of detected faces is counted, and the total result or the ratio of the total result to the detected face and the information specifying each time point (for example, the time corresponding to each time point, Information that associates the elapsed time from the start of the lecture, the frame number, and the like) may be acquired as a statistical analysis result. Note that the result of counting the number of faces may be normalized as appropriate. The same applies to the following.

  In addition, the statistical analysis unit 14 totals, for example, one face (or a part thereof) of faces detected by analysis item at one or more time points of the student image obtained in one lecture. A value or a ratio of the total value to the number of detected faces may be acquired as a statistical analysis result. Alternatively, an analysis result (for example, facial expression, face direction, line-of-sight direction, etc.) for each analysis item for each of a plurality of faces recognized at one or more time points (for example, frame images at each time point) of the student raw image, Convert to numerical values using conversion tables and conversion formulas prepared in advance, and obtain the average value, variance, standard deviation, etc. of these numerical values for each time point or for one lecture (or part of it). May be. As this numerical value, a numerical value used in face analysis may be used.

  In addition, the statistical analysis unit 14 obtains information indicating the time when the count result and the ratio are high from the count result of the number of faces for each analysis item of the analysis result acquired for each time point and the ratio thereof (for example, time and progress from the start of the lecture Time etc.) may be acquired as a statistical analysis result.

  In addition, the statistical analysis unit 14 extracts, for example, face analysis results over a plurality of points in time for each student face recognized by the face analysis unit 13 as a statistical analysis result, or arranges them in time series. Or you may. The student's face analysis results over a plurality of time points are, for example, analysis results obtained at different time points (time) of the student image taken for one lecture. For example, a face analysis result for one or more students is acquired from two or more time points (preferably a plurality of time points) from the student image, and the analysis results are arranged in time series for each student. You may get it. At this time, the face analysis unit 13 acquires information in which each analysis result is associated with information indicating the time when each analysis result is acquired (for example, time information, frame number information, etc.). Also good.

  In addition, the statistical analysis unit 14 determines the number of appearances of the analysis result for each analysis item for each face recognized by the face analysis unit 13 for the analysis results obtained by the face analysis unit 13 for a plurality of time points of the student image. The information may be aggregated, and information in which the aggregation result is associated with information for identifying the recognized face (for example, identification information assigned to the recognized face) may be acquired as a statistical result. Thereby, the statistical result which shows the change of the analysis result of the time series of each student's face recognized by the student image can be acquired.

  Further, the face analysis unit 13 may perform cluster analysis or multivariate analysis using the value of the analysis result for each analysis item as a multivariate.

  In addition, the statistical analysis unit 14, for example, adds a value obtained by converting the detection result of the analysis item for each face into a numerical value or a value obtained by weighting and adding the analysis item for each comprehensive face for each recognized face. It may be acquired as a feature value that is an analysis result, and an average value, variance, deviation value, or the like may be acquired using this feature value.

  The analysis result acquired by the face analysis unit 13 may be an analysis result acquired by the face analysis unit 13 accumulated in a storage medium (not shown) by the output unit 17 described later.

  The lecture evaluation unit 15 evaluates the lecture using the statistical analysis result of the statistical analysis unit 14. The lecture evaluation performed here is, for example, an evaluation related to the lecture. The evaluation of the lecture performed by the lecture evaluation unit 15 is, for example, an evaluation of how to advance the lecture or the content of the lecture. Further, the evaluation may be an evaluation for a student who takes a lecture, or an evaluation for a lecturer who gives a lecture. For example, an evaluation for determining whether a lecture is good or bad, an evaluation for determining whether or not a lecture is appropriate, and the good or appropriateness of a lecture are indicated by numerical values or indicators. Alternatively, it may be an evaluation of whether the lecture is easy to understand, whether the lecture is interesting, and whether it is easy to concentrate on the content of the lecture.

  For example, the lecture evaluation unit 15 evaluates a lecture using the result of statistical analysis acquired by the statistical analysis unit 14 for the entire lecture. For example, it is determined whether or not a statistical analysis result (for example, the number of face detections for each analysis item aggregated in one entire lecture) for a whole lecture satisfies a predetermined condition. In this case, an evaluation result prepared in advance in association with this condition is acquired. An evaluation result prepared in advance for not satisfying the condition or not satisfying the condition may be acquired. This condition is, for example, a condition that the value of an analysis item designated in advance prepared for each analysis item exceeds a threshold value or does not exceed a threshold value. Alternatively, this condition may be a condition such that all the conditions individually prepared for each of the plurality of classification items are satisfied, not satisfied, or only a part thereof is satisfied.

  In addition, the lecture evaluation unit 15 determines, for example, whether or not the result of the statistical analysis acquired by the statistical analysis unit 14 satisfies a predetermined condition for each time point of the student image. May be performed. For example, it is determined whether or not the number of detected faces (or a normalized value) for each analysis item acquired by the statistical analysis unit 14 at each time point of the student image exceeds a threshold value specified in advance for each analysis item. When the number of detections of 1 or more exceeds the threshold, it may be determined that the condition is satisfied, and an evaluation result associated with the condition may be acquired.

  Further, the lecture evaluation unit 15 may acquire the evaluation result of one lecture using the evaluation result of the lecture at each time point of the student image of the one lecture acquired as described above. For example, when all or some of the evaluation results of lectures acquired for a plurality of time points of a student image of one lecture satisfy a predetermined condition, the evaluation results associated with this condition are It may be acquired as an evaluation result of one lecture. For example, among the evaluation results of lectures acquired for a plurality of time points, the number of evaluation results specified in advance (for example, the evaluation result indicating that the student's interest is maintained) is greater than or equal to a predetermined threshold value. If the condition is satisfied, it may be determined that the condition is satisfied, and an evaluation result of “quality lecture” associated with the condition may be acquired. Alternatively, evaluation results of a plurality of lectures as a whole may be acquired using evaluation results of a plurality of lectures.

  What kind of student analysis results are used for the evaluation of the lecture may be appropriately changed depending on the purpose of the evaluation of the lecture. When evaluating one lecture, the result of statistical analysis used for evaluation is, for example, the result of statistical analysis acquired by the statistical analysis unit 14 from a student image taken in one lecture. Also, when evaluating lectures conducted on one theme, one subject, one grade, one class, etc., for example, one theme, one subject, one grade, one class, etc. The result of the statistical analysis acquired by the statistical analysis unit 14 from the student images taken in one or more (preferably a plurality of) lectures conducted on is used for evaluation. Further, when evaluating a lecture by one lecturer, for example, a result of statistical analysis acquired by the statistical analysis unit 14 from student images taken in one or more (preferably a plurality of) lectures conducted by one lecturer. Is used for evaluation.

  Note that the statistical analysis result of the statistical analysis unit 14 used by the lecture evaluation unit 15 for the evaluation of the lecture is a statistical analysis result acquired by the statistical analysis unit 14 accumulated in a storage medium (not shown) by the output unit 17 to be lectured. Also good.

  When the face analysis unit 13 acquires a plurality of analysis results of the student's face in time series, the designation receiving unit 16 designates the analysis result of the face at one time point output by the output unit 17 described later. Accept. For example, the designation receiving unit 16 receives one or more designations of the face analysis results output by the output unit 17 in time series. The output here is display, storage in a storage medium (not shown), or the like. The designation of the analysis result may be designation of analysis result identification information, an image showing the analysis result, or the like. Accepting here means accepting information input from input devices such as a keyboard, mouse, touch panel, receiving information sent via a wired or wireless communication line, recording on an optical disc, magnetic disc, semiconductor memory, etc. It is a concept including reception of information read from a medium. The input means for designating may be anything such as a numeric keypad, keyboard, mouse or menu screen. The designation receiving unit 16 can be realized by a device driver for input means such as a numeric keypad or a keyboard, control software for a menu screen, and the like.

  The output unit 17 outputs information related to the analysis result by the face analysis unit 13. The information related to the analysis result by the face analysis unit 13 is, for example, information on the analysis result acquired by the face analysis unit 13. Moreover, the information which has this analysis result information and the information of the acquired time (for example, time), such as a frame image used as the acquisition target of the analysis result, may be associated with each other. The information related to the analysis result by the face analysis unit 13 may be information acquired using the analysis result acquired by the face analysis unit 13, for example. For example, the output unit 17 outputs the result of the statistical analysis acquired by the statistical analysis unit 14 as information related to the analysis result by the face analysis unit 13. The output unit 17 outputs the evaluation result of the lecture evaluation unit 15 as information related to the analysis result by the face analysis unit 13.

  Further, the output unit 17 outputs the face analysis results of the face analysis unit 13 along the time series when the face analysis unit 13 acquires a plurality of analysis results of the face of the student along the time series. It may be. Outputting the analysis results in time series means, for example, outputting the analysis results arranged in time series. For example, the analysis results may be arranged and output according to the order of the time points of the student raw images that are the acquisition targets of the analysis results. Alternatively, the analysis results can be output in association with the time, and the time-series order can be easily grasped, and the time-series order can be easily rearranged. May be. For example, the output unit 17 identifies the analysis result and information identifying the time point of the student raw image from which the analysis result is acquired (for example, the time associated with the frame image of the student raw image from which the analysis result is to be acquired). And the frame image identification information) may be output in association with each other. In addition, when the output is a display, the identification information of each analysis result, the link button associated with the analysis result, and the like can be displayed in association with the time axis, or the analysis result can be output in time series. You may think that

  In addition, when the face analysis unit 13 acquires a plurality of analysis results of the students' faces along the time series, the output unit 17 is a student at the time corresponding to the analysis result received by the designation receiving unit 16 Output an image. For example, the output unit 17 reads out the frame image associated with the time point (for example, the time) corresponding to the analysis result received by the designation receiving unit 16 from the student raw image storage unit 12 and outputs the frame image. Alternatively, a moving image of a period before and after the time corresponding to the analysis result received by the designation receiving unit 16 is read from the student raw image and output as a student raw image at the time corresponding to the analysis result received the designation. Also good. When outputting a student image (for example, a frame image), a student image that is displayed in such a manner that the student's face portion of the student image cannot be identified by the student is output. Anyway. For example, the face portion is displayed in such a manner that the student cannot be identified. For example, the face portion of the student image of the student image is a monochrome image, an image such as an icon prepared in advance, To display images that are unrelated to the student's individual face, such as pattern images, to mask the student's face and hide the face, This is to perform mosaic processing and blur processing on the image. The face part of the student included in the student image is, for example, information indicating the face area recognized by the face analysis unit 13 for the student image when the face is analyzed (for example, the face contour). The coordinates of the pixels and the coordinates of the pixels constituting the face), and when the output unit 17 outputs the student raw image, the process when the face analysis unit 13 performs the face analysis Similarly to the above, it may be detected by recognizing the face in the student image. By doing in this way, it becomes possible to protect each student's privacy by not distinguishing each student's face in a student image.

  Output here means display on a display, projection using a projector, printing on a printer, transmission to an external device, storage on a recording medium, processing results to other processing devices or other programs, etc. It is a concept that includes delivery. The output described here is a concept including temporary storage. The output unit 17 may or may not include an output device such as a display. The output unit 17 can be realized by output device driver software, or output device driver software and an output device. This also applies to the following output units.

  Next, the operation of the face analysis apparatus 1 will be described using the flowchart of FIG. Here, it is assumed that the student raw images taken for one lecture are stored in the student raw image storage unit 12 in advance. Here, it is assumed that this student raw image is a moving image constituting one file. It is assumed that a time code is associated with each frame image of the student image.

  (Step S201) The face analysis apparatus 1 determines whether an instruction to analyze one student raw image stored in the student raw image storage unit 12 has been received via a receiving unit (not shown) or the like. If an instruction has been accepted, the process proceeds to step S202. If not, the process returns to step S201.

  (Step S202) The face analysis unit 13 substitutes 1 for a counter n.

  (Step S203) The face analysis unit 13 determines whether or not a frame image associated with the n-th time to be analyzed exists in the student raw image to be analyzed in the student raw image. The frame image here may not be a continuous frame image. For example, when face analysis is sequentially performed at intervals of 5 seconds from the start of shooting, it is determined whether there is a frame image associated with a time code n × 5 seconds after the start of shooting. If there is a frame image, the process proceeds to step S204; otherwise, the process proceeds to step S210.

  (Step S204) The face analysis unit 13 sequentially recognizes one or more faces in the frame image associated with the nth time. For example, the face analysis unit 13 assigns face identification information to information indicating the contours of one or more recognized faces, and temporarily stores them in a storage medium (not shown).

  (Step S205) The face analysis unit 13 analyzes each face recognized in step S204. For example, for each face recognized in step S204, an analysis is performed on the analysis item designated in advance, and information indicating the analysis result is acquired. When performing face analysis, for example, a frame image before or after the frame image associated with the nth time may be used.

  (Step S206) The face analysis unit 13 associates the identification information of each face, the analysis result for each face, and the time corresponding to the frame image in the analysis mode, and accumulates them in a storage medium (not shown). . As a result, the analysis results are output (in particular, accumulated here) along the time series. This accumulation may be performed by the output unit 17. The face analysis unit 13 performs a face detection process on the frame image associated with the nth time, and performs an analysis as shown in step S205 for each face each time a face is detected. The process of accumulating the analysis results may be repeated until no new face can be detected.

  (Step S207) The statistical analysis unit 14 performs statistical analysis using the analysis result of the face analysis unit 13 for the frame image associated with the nth time. For example, for the analysis result of each face acquired in step S206, the number of faces having the same analysis result is totaled. For example, this total value is a statistical analysis result. For example, the same analysis result means that the analysis item and the analysis value obtained for the analysis item are the same. In addition, when a plurality of analysis results are obtained for one face, a part of the same analysis result may be preferentially aggregated. Alternatively, this one face may be counted redundantly in the aggregation of the respective analysis results. Here, the statistical analysis is performed for each frame image. However, using the analysis result of the face analysis unit 13 for the plurality of frame images, the same statistical analysis as described above is performed for each of the plurality of frame images immediately before step S210. May be performed.

  (Step S208) The statistical analysis unit 14 stores the statistical analysis result acquired in step S207 in a storage medium (not shown) or the like in association with the nth time. This accumulation may be performed by the output unit 17.

  (Step S209) The face analysis unit 13 increments the value of the counter n by 1. Then, the process returns to step S203.

  (Step S210) The lecture evaluation unit 15 evaluates the lecture corresponding to the above-described one student raw image using the statistical analysis result accumulated in step S208. For example, if there is a statistical analysis result that satisfies a previously specified condition among the statistical analysis results accumulated in step S208, information indicating the evaluation of a lecture that is associated with this condition in advance is acquired. For example, the statistical analysis result corresponding to each time is the total number of faces of students who are facing sideways, and the predesignated condition is that the number of faces of students who are facing sideways is designated in advance. If it is a condition that more than a predetermined number of statistical analysis results are detected, and the evaluation of the lecture corresponding to this condition is an evaluation that “concentration is not maintained”, step S208. If the statistical analysis result accumulated in this condition satisfies this condition, the evaluation result of the lecture “Concentration is not maintained” is obtained. Here, a plurality of different evaluations may be performed. Note that the statistical analysis unit 14 may perform a process of counting statistical analysis results in which the number of faces of students facing sideways is equal to or greater than a predetermined number.

  (Step S211) The lecture evaluation unit 15 stores the evaluation results in a storage medium (not shown) or the like in association with the lecture identification information, the student raw images, or the like. This accumulation may be performed by the output unit 17.

  (Step S212) The output unit 17 displays the statistical analysis result accumulated in Step S208 and the evaluation result accumulated in Step S211.

  (Step S213) The output unit 17 determines whether to display the analysis result acquired by the face analysis unit 13 accumulated in Step S206. For example, it is determined whether or not an instruction to display the analysis result is received via a reception unit (not shown) or the like. If the analysis result is displayed, the process proceeds to step S214. If not, the process proceeds to step S219.

  (Step S214) The output unit 17 displays the analysis result accumulated in Step S206.

  (Step S215) The designation receiving unit 16 determines whether designation of one analysis result has been received via a receiving unit (not shown) or the like. The one analysis result here is, for example, an analysis result associated with one time. If accepted, the process proceeds to step S216. If not accepted, the process proceeds to step S220.

  (Step S216) The output unit 17 acquires a frame image at the same time as the time corresponding to the analysis result specified in Step S215 from the student raw image.

  (Step S217) The output unit 17 displays the frame image acquired in step S216. Note that the output unit 17 may continuously reproduce the frame images after the frame image acquired in step S216 of the student raw images. That is, a moving image may be reproduced from the frame image acquired in step S216. When playing back a moving image, the frame image acquired by the output unit 17 in step S216 is a frame image one frame before or one frame after the time corresponding to the analysis result specified in step S215. Alternatively, it may be a frame image of the time before and after the time specified in advance with respect to the time corresponding to the analysis result specified in step S215. Also, when outputting one frame image or continuous frame images, output a frame image in such a display mode that the student's face portion of each frame image cannot be identified by the student. May be.

  (Step S218) The output unit 17 determines whether or not to end the display of the frame image. For example, when an operation for ending the display is received via a reception unit (not shown) or the like, it is determined to end the display. When the display is finished, the display is finished and the process returns to step S215. When the display is not finished, the process returns to step S218.

  (Step S219) The output unit 17 determines whether or not to end the display of the evaluation result and the statistical analysis result. For example, when an operation for ending the display is received via a reception unit (not shown) or the like, it is determined to end the display. When the display is ended, the display is ended and the process returns to step S212. When the display is not ended, the process returns to step S215.

  (Step S220) The output unit 17 determines whether or not to end the display of the analysis result acquired by the face analysis unit 13. For example, when an operation for ending the display is received via a reception unit (not shown) or the like, it is determined to end the display. If the display is to be ended, the display is ended and the process returns to step S201. If the display is not ended, the process returns to step S213.

  In the flowchart of FIG. 2, the process is terminated by powering off or a process termination interrupt.

  Hereinafter, an example of the specific operation of the face analysis apparatus 1 according to the present embodiment will be described.

  FIG. 3 is a diagram schematically showing a part of the frame image constituting the student raw image stored in the student raw image storage unit 12. It is assumed that this student image is a moving image obtained by photographing the entire student participating in one lecture using the first photographing unit 11 from the lecturer side. This student raw image is composed of one file. Here, for the sake of explanation, a time code associated with each frame image is shown below each frame image. The time code represents the time at which each frame image was taken as “hour: minute: second.frame”.

  Assume that the user operates the input device (not shown) such as a mouse and a keyboard to give the face analysis apparatus 1 an instruction to analyze the student raw image shown in FIG. Here, it is assumed that face analysis processing is set in advance for frame images at intervals of 1 second.

  First, the face analysis unit 13 reads out a frame image associated with the time code “10: 30: 30: 01” which is the first frame image (frame image at the start of photographing) of the student raw image. Then, processing for recognizing a face is performed in the read frame image.

  FIG. 4 is a schematic diagram showing a frame image in a state where the face is recognized by the face analysis unit 13. In the figure, it is assumed that regions 401 to 430 surrounded by dotted lines are recognized facial regions. It should be noted that the facial expression and orientation, line-of-sight direction, and blinking situation shown in this figure are schematic and do not necessarily accurately represent the facial expression and direction, line-of-sight direction, and blinking situation, etc. Shall. The same applies to the following.

  As shown in FIG. 4, a plurality of faces are recognized in the frame image by the face recognition process. The face analysis unit 13 associates information indicating the contours of the recognized face regions 401 to 430 with face IDs, which are identification information for identifying faces, and accumulates them in a storage medium (not shown). The information indicating the contour is, for example, a coordinate group of pixels constituting the contour in the frame image. Here, for example, “ID401” to “ID430” are associated with the areas 401 to 430 as face IDs, respectively.

  The face analysis unit 13 sequentially analyzes each face recognized above. First, an analysis is performed on the face with the face ID “ID 401” (that is, the face in the region 401). Here, for example, an analysis item whether or not the facial expression is a smiling face, an analysis item whether or not the gaze direction is facing the front (the lecturer side), and the face direction is the front (the lecturer side) ) Analyze each of the analysis items for whether or not they are facing and the analysis items for the presence or absence of blinking. In addition, since these analysis processes are well-known techniques, specific description is abbreviate | omitted. In each analysis, if necessary, one or more frame images before or after the frame image to be analyzed are read from the student raw image, and the image in the area 401 of the read frame image or its frame image is read. Peripheral images may be used.

  The face analysis unit 13 stores the analysis result for the face with the face ID “ID401” in a storage medium (not shown) in association with the time code “10: 30: 10: 01” and the face ID “ID401”. .

  The face analysis unit 13 performs the same processing for the faces whose face IDs are “ID402” to “ID430”.

  FIG. 5 is a face analysis result management table for managing information indicating the analysis result of the face analysis acquired by the face analysis unit 13 and accumulated in a storage medium (not shown). The face analysis result management table includes items of “time”, “face ID”, “expression”, “gaze direction”, “face direction”, and “blink”. “Time” is the time of the frame image to be analyzed of the student image, and here is the time code of the frame image. “Face ID” is the face ID described above. “Expression” is the analysis result of the facial expression of the student. Here, the value “Smile” indicates that the face is laughing, and the blank indicates that the face is not laughing. “Gaze direction” is the analysis result of the gaze direction of the student's face, where the value “front” indicates that the gaze direction is facing the front and the blank indicates that it is not facing the front . “Face direction” is the analysis result of the student's face direction, and the value “front” indicates that the face is facing the front, and the blank indicates that the face is not facing. “Blink” is the analysis result of whether or not the student has closed eyes, the value “Closed” indicates that the eyes are closed, and the blank indicates that the eyes are open .

  The statistical analysis unit 14 performs statistical analysis using the face analysis result by the face analysis unit 13 with respect to the frame image associated with the time code “10: 30: 30: 01” as illustrated in FIG. . Here, the statistical analysis unit 14 has the number of faces whose “expression” is “smile” and the “gaze direction” is “front” for the frame image whose time code is “10: 30: 00: 00”. The number of faces, the number of faces whose “face direction” is “front”, and the number of faces whose “blink” is “closed” are respectively tabulated. That is, aggregation is performed for each analysis item. Then, these total results are divided by the total number of recognized faces “30” to obtain the ratio of appearance of faces for each analysis item. Instead of calculating the ratio, variance or the like may be calculated. Then, the statistical analysis unit 14 stores the acquired ratio for each analysis item as a statistical analysis result in a storage medium (not shown) in association with the time code “10: 30: 30: 01”.

  The face analysis unit 13 further reads out a frame image associated with the time code “10: 30: 01: 01” corresponding to the time after 1 second of the frame image read out above, and performs the same as described above. Then, the process of recognizing the face is performed, the same analysis as described above is performed on the recognized face, and the analysis result is added to a storage medium (not shown). As a result, an analysis result record in which “time” is associated with “10: 30: 01: 01” is added to the face analysis result management table shown in FIG. Here, as an example, the same face ID as that of the face recognized immediately before is assigned to the face in the recognized face that is substantially in the same position as the face recognized immediately before. A face in substantially the same position is, for example, a face in which the coordinates of pixels included in the region match a predetermined number or more (for example, half or more).

  Further, using this analysis result, the statistical analysis unit 14 performs statistical analysis for each analysis item in the same manner as described above, and accumulates the statistical analysis results.

  The face analysis unit 13 and the statistical analysis unit 14 sequentially repeat the above processing until there is no frame image after one second in the student raw images.

  FIG. 6 is a diagram illustrating a statistical analysis result management table for managing the statistical analysis results acquired and accumulated by the statistical analysis unit 14. The statistical analysis result management table has items of “time”, “expression (smile)”, “line of sight (front), face direction (front)”, and “blink (close)”. “Time” is the time of the frame image to be analyzed, and here is the time code of the frame image. “Facial expression (smile)” is the percentage of faces whose analysis item “expression” is “smile” in the facial analysis results (%), and “Gaze (front)” is the facial analysis result The ratio (%) of faces whose analysis item “line-of-sight direction” is “front” and “face direction (front)” is the value of the analysis item “face direction” in the face analysis result “ The ratio (%) of the face that is “front” and “blink (closed)” are the ratio (%) of the face whose analysis item “blink” is “closed” in the face analysis result.

  Next, the lecture evaluation unit 15 uses the statistical analysis result acquired by the statistical analysis unit 14 to evaluate the lecture corresponding to the student raw images. Here, for example, as a condition for evaluating the lecture, the time when the ratio of ““ line of sight (front) ”is“ 70% or more ”is“ 80% ”of the time when the face analysis is performed. The condition “exceeding” is stored in a storage medium (not shown) in advance, and the value of “highly concentrated lecture” is not shown in association with this condition as a value indicating the evaluation of the lecture when this condition is satisfied. Assume that it is stored in a storage medium. The time here may be considered as each frame image to be analyzed, or may be considered as each record of the statistical analysis result management table shown in FIG.

  The lecture evaluation unit 15 first reads out the above conditions. Next, in the statistical analysis result management table shown in FIG. 6, the number of records whose “line of sight (front)” ratio is “70% or more” is counted. For example, assume that the count number is “1752”. Also, the lecture evaluation unit 15 acquires the total number of records in the statistical analysis result management table shown in FIG. For example, it is assumed that the total number of records is “2386”. The lecture evaluation unit 15 divides the count number “1752” of the record whose “line of sight (front)” is “70% or more” by the total number of records “2382” to obtain the ratio of “line of sight (front)”. “73.6 (%)” is acquired. Then, it is determined whether or not the acquired ratio is “70% or more”. Here, since it is 70% or more, the evaluation result “lecture with high concentration” is acquired. In the case of less than 70%, an evaluation result “lecture with a low degree of concentration” may be acquired. The acquired evaluation result is stored in a storage medium (not shown) in association with the file name of the student raw image, for example.

  The output unit 17 displays the statistical analysis result acquired by the statistical analysis unit 14 and the evaluation result acquired by the lecture evaluation unit 15 on a monitor or the like.

  FIG. 7 is a diagram illustrating a display example of a statistical analysis result and a lecture evaluation result. When the statistical analysis result or the lecture evaluation result does not fit in the display area of the monitor, a scroll bar or the like may be shown so that a portion that cannot be entered by operating the scroll bar may be displayed as appropriate. The same applies to the following.

  Here, assuming that an instruction to display the analysis result of the face acquired by the face analysis unit 13 is received via a reception unit (not shown) or the like, the output unit 17 is managed by the face analysis result management table shown in FIG. The face analysis result along the time series acquired by the face analysis unit 13 is displayed on a monitor or the like. For example, the output unit 17 displays the analysis results side by side along a time series.

  FIG. 8 is a diagram illustrating a display example of a face analysis result. In this face analysis result display screen, the character string of “time” of each analysis result is, for example, a determination area for determining whether or not designation for each analysis result is accepted (for example) (Not shown) is provided.

  Next, if the user operates an input device such as a mouse and presses the character string at one time “10: 30: 02.01” on the screen shown in FIG. Accepts the designation for the face analysis result corresponding to this time. When the designation accepting unit 16 accepts this designation, the output unit 17 reads the student raw image at the time corresponding to the analysis result from the student raw image storage unit 12. Here, the frame image associated with the same time as the time “10: 30: 02.01” of the analysis result that received the designation is read from the student image. Then, the output unit 17 displays the read frame image on the monitor. Note that when outputting the frame image, the output unit 17 may output a frame image in a display mode in which the student's face portion of the frame image cannot be identified by the student. By doing so, it is possible to protect the student's privacy by preventing the student's face from being recognized. For example, the output unit 17 recognizes the contours of the face regions 401 to 430 that the face analysis unit 13 has recognized in the face recognition process as shown in FIG. 4 and accumulated in a storage medium (not shown). Is read, and the face portion in the frame image whose display mode is to be changed is detected. Or the face part newly detected with respect to the frame image output by the output unit 17 by the face analysis unit 13 may be the display mode change target by the output unit 17.

  FIG. 9 is a diagram illustrating a display example of a frame image corresponding to the analysis result of the face whose designation is accepted by the designation receiving unit 16.

  As described above, according to the present embodiment, the face of one or more students in the lecture is automatically recognized and analyzed, the analysis result is statistically analyzed, and the evaluation of the lecture is performed from the result of the statistical analysis. And conduct appropriate analysis on lectures.

  In the above embodiment, the statistical analysis unit 14 performs the statistical analysis of the face analysis results for each frame image. However, the facial expression, the line-of-sight direction, Information indicating changes in analysis results such as face direction and blinking may be acquired as information indicating the results of statistical analysis. Moreover, when the analysis result along such a time series is indicated by a change in a numerical value or the like, a result obtained by performing Fourier transform or the like may be acquired as a result of statistical analysis. As a result, it is possible to represent the situation at the time of attendance of students corresponding to each face as numerical values such as frequency components. In addition, when a frame image analyzed as the above-mentioned blink information detected for one face is “closed” continues for a predetermined time (for example, one minute or longer), The corresponding student may be determined to have fallen asleep. The predesignated time is, for example, a time that is equal to or greater than the maximum time during which the eyelid is considered to be closed during normal blinking.

(Embodiment 2)
The face analysis apparatus 2 according to the second embodiment is the same as the face analysis apparatus 1 according to the first embodiment, in which the image of the lecturer in lecture and the voice information of the lecturer are analyzed for the face of the student and the analysis of the face. It is used for statistical analysis of results and evaluation of lectures.

  FIG. 10 is a block diagram of face analysis apparatus 2 in the present embodiment.

  The face analysis device 2 includes a first photographing unit 11, a student raw image storage unit 12, a second photographing unit 21, a voice acquisition unit 22, a lecturer information storage unit 23, a change detection unit 24, a face analysis unit 25, and a related statistical analysis unit. 26, a related lecture evaluation unit 27, and an output unit 28.

  Since the first imaging unit 11 and the student raw image storage unit 12 are the same as those in the first embodiment, detailed description thereof is omitted here.

  The second photographing unit 21 photographs the lecturer during the lecture and acquires a moving image of the lecturer during the lecture. A lecturer is a person who gives a lecture. There is usually one instructor, but there may be multiple instructors. Shooting the instructor means, for example, obtaining a moving image that shows the instructor's movement. For example, the second photographing unit 21 acquires a moving image obtained by photographing the entire lecturer or the upper body. The second image capturing unit 21 and the moving image acquired by the second image capturing unit 21 have the same configuration as the first image capturing unit 11 described above except that the object to be imaged is different. Is omitted. For example, the second photographing unit 21 accumulates the acquired moving image in the instructor information storage unit 23 as instructor information described later or as a part thereof.

  The voice acquisition unit 22 acquires voice information of the lecturer during the lecture (for example, voice information emitted by the lecturer). For example, the voice acquisition unit 22 acquires information in which voice is associated with timing (for example, time) when the voice is acquired. For example, the voice acquisition unit 22 accumulates the acquired voice information in the lecturer information storage unit 23 as a part of lecturer information described later. The file format, compression method, etc. of lecturer audio information are not limited. Note that either the audio acquisition unit 22 or the second imaging unit 21 has instructor information associated with the audio acquired by the audio acquisition unit 22 and the moving image acquired by the second imaging unit 21 so as to be synchronized. You may make it accumulate | store in the lecturer information storage part 23. FIG. The voice acquisition unit 22 is realized by a microphone, for example. The second imaging unit 21 and the audio acquisition unit 22 may be an imaging unit (camera unit) and an audio acquisition unit (microphone) of one video camera.

  Instructor information storage unit 23 stores instructor information. Instructor information is information including at least one of instructor image or instructor audio information. The instructor image is a moving image of the instructor being lectured. For example, it is a moving image acquired by the second imaging unit 21. The lecturer audio image is information on the audio of the lecturer who is giving a lecture. For example, the voice information of the lecturer acquired by the voice acquisition unit 22. The lecturer image is the same information as the student image, except that the shooting target is different. The lecturer sound image is, for example, information associated with a time series. For example, the lecturer image may be associated with a time code or the like, and the lecturer voice information may be stored in the lecturer information storage unit 23 in synchronization with the lecturer image. The storage here is a concept including temporary storage.

  The change detection unit 24 detects changes in instructor information. For example, the change detection unit 24 detects a predetermined change designated in advance from the lecturer information. The change detection unit 24 may acquire information (for example, the time when the change is detected) specifying the time point when the change is detected as a detection result. The detection of the conversion of the lecturer information is either the detection of the change in the lecturer image included in the lecturer information, the detection of the change in the lecturer audio information included in the lecturer information, or both. For example, the change detection unit 24 detects a change that satisfies a predetermined condition. In the case where the change detection target is a lecturer image, the predesignated condition is, for example, the presence or absence of a motion or a motion greater than the size of a predesignated motion. In addition, when the change detection target is lecturer voice information, the predesignated condition is, for example, voice output of a level specified in advance or higher. The change detection unit 24 may acquire information indicating the content (for example, type) of the detected change or information indicating the amount of change as a detection result.

  For example, the change detection unit 24 performs motion vector detection by comparing the pixels of the frame image with respect to the lecturer image included in the lecturer information, and the presence or absence of motion or a motion vector greater than or equal to a predetermined threshold value. Detection is performed. When a motion that is equal to or greater than a predetermined threshold value (for example, a change in distance that is equal to or greater than a threshold value) is detected, it is determined that a motion is detected. In addition, the instructor image included in the instructor information, like the face analysis unit 25, detects the instructor's face, detects the instructor's hand, etc., and performs motion vector detection of the detected portion, thereby moving The detection of the presence or absence of movement and the movement more than the threshold value designated beforehand is performed.

  Moreover, the change detection part 24 detects the change of lecturer information by the change of the audio | voice level in the lecturer audio | voice information contained in lecturer information, for example. For example, the change detection unit 24 determines that a change in the lecturer information is detected when an output having a level equal to or higher than a predetermined threshold value is detected in the lecturer voice information. Moreover, you may detect the change of lecturer information, when the value of SN ratio of lecturer audio | voice information becomes more than a threshold value. Further, the utterance speed of the lecturer voice information may be detected, and the change in the lecturer information may be detected when the utterance speed exceeds a predetermined threshold value. A technique for detecting the utterance speed is disclosed in, for example, Japanese Patent Application No. 7-307897.

  The face analysis unit 25 recognizes the face of the student from the student image and analyzes the recognized face. In this embodiment, in particular, the face analysis unit 23 acquires the analysis result of the face for the student raw image acquired immediately after the change detection unit 24 detects the change. The term “immediately after here” has the same meaning as just described in the first embodiment. The face recognition process and analysis process performed by the face analysis unit 25 on the student raw image are the same as those of the face analysis device 13 of the first embodiment. Specifically, a frame image) obtains the analysis results of one or more students' faces for student images (frame images) acquired immediately after the change detection unit 24 described later detects a change in instructor information. This is different from the face analysis unit 13 of the first embodiment. Note that the acquisition result of the student's face for the student image (frame image) acquired immediately after the change is detected means that the face analysis unit 25 uses the change detection unit 24 described later as instructor information. The student image acquired immediately after the change is detected may be analyzed for the face to obtain an analysis result, and the face analysis unit 25 may have a plurality of different time points ( The analysis result acquired immediately after the time point when the change is detected in the lecturer information may be acquired from the analysis results of the face acquired at (time), or a process corresponding to this may be performed. The term “immediately after the change is detected” means, for example, a point in time that has passed for a predetermined time from the time (for example, time) at which the change is detected. It may be considered that the time and the number of frames specified in advance are separated. This elapsed time is for considering the time lag between the change of the teacher's movement and the change of the voice and the change of the student's facial expression corresponding to the change. This elapsed time is, for example, a time of 5 seconds or less. This elapsed time may be acquired based on, for example, experimental results in advance. However, it may be the time when the change is detected, or may be considered as the next time (for example, the next frame image) after the change is detected. Immediately after the change is detected, the time immediately after the change is detected may be considered as the time around the time when the change is detected. The same applies to the following. Note that the time here may be considered as the number of frames. Further, immediately after the time when the change is detected, it may be considered as a period of a predesignated length immediately after the time when the change is detected. For example, the face analysis unit 25 acquires the analysis result of the face performed on the frame image at the time when a predetermined number of seconds have elapsed from the time when the change detection unit 24 detects the change in the lecturer information. In the analysis, frame images before and after that may be used as appropriate.

  The related statistical analysis unit 26 performs statistical analysis specified in advance using the analysis result of the face analysis unit 25 on the raw student image acquired immediately after the change detection unit 24 detects the change. The statistical analysis designated in advance may be any statistical analysis. For example, the statistical analysis designated in advance by the related statistical analysis unit 26 is the same statistical analysis as the statistical analysis performed by the statistical analysis unit 14, and the analysis result used for the statistical analysis is changed by the change detection unit 24. This is an analysis result acquired by the face analysis unit 25 with respect to the student raw image acquired immediately after the detection. Further, the statistical analysis designated in advance may be statistical analysis in which information indicating the change target detected by the change detection unit 24, information indicating the content of the change, and the analysis result of the face analysis unit 25 are associated with each other. Good. The change target is a target for which the change detection unit 24 has detected a change, such as a lecturer image or lecturer audio information. The information indicating the content of the change is information indicating the content of the change detected by the change detecting unit 24. For example, the information indicating the change that the instructor raised his hand or the change that the instructor moved sideways. It is information which shows etc. Moreover, the information which shows the change etc. that lecturer audio | voice information became the audio | voice more than the level designated beforehand may be sufficient. The result of the analysis of the face analysis unit 25 with respect to the frame image of the student raw image acquired immediately after each of the plurality of time points when the change is detected is further tabulated according to the analysis item as in the statistical analysis unit 14 described above. Alternatively, the data may be aggregated according to the change target detected by the change detection unit 24 and the content of the change.

  The related lecture evaluation unit 27 evaluates the lecture using the statistical analysis result of the related statistical analysis unit 26. The lecture evaluation performed here is, for example, an evaluation related to the lecture. Similar to the evaluation performed by the lecture evaluation unit 15, the evaluation performed by the related lecture evaluation unit 27 is, for example, an evaluation of how to proceed with the lecture and the content of the lecture.

  The related lecture evaluation unit 27, for example, uses the result of the statistical analysis performed by the related statistical analysis unit 26 included in the result of the statistical analysis acquired by the related statistical analysis unit 26 (hereinafter referred to as the related statistical analysis result). An evaluation result may be acquired, or a lecture evaluation result is obtained from a combination of the content of the change detected by the change detection unit 24 included in the related statistical analysis result and the result of the statistical analysis performed by the related statistical analysis unit 26. May be obtained.

  For example, the related lecture evaluation unit 27 determines whether or not the related statistical analysis result acquired by the related statistical analysis unit 26 from the student image obtained by photographing one lecture satisfies a predetermined condition, for example. If the condition is satisfied, the evaluation result of the lecture prepared in advance associated with this condition is acquired. An evaluation result prepared in advance for not satisfying the condition or not satisfying the condition may be acquired. This condition is, for example, a condition that the value of the analysis item designated in advance exceeds or does not exceed the threshold value prepared for each analysis item. Alternatively, this condition may be a condition such that all the conditions individually prepared for a plurality of classification items are satisfied, or not satisfied, or only a part thereof is satisfied.

  For example, the related lecture evaluation unit 27 has a condition in which a combination of the content of the change detected by the change detection unit 24 included in the related statistical analysis result and the result of the statistical analysis performed by the related statistical analysis unit 26 is specified in advance. If the condition is satisfied, the evaluation result of the lecture prepared in advance associated with the condition is acquired. For example, the number of laughing faces detected from each student image immediately after one or more time points when the level of the lecturer audio information included in the lecturer information is equal to or greater than a predetermined threshold is counted. Is greater than a pre-specified threshold value, it is considered that there are many students laughing at the words spoken by the instructor. The result may be obtained. In this case, for example, since it is considered that there are many students laughing at the words spoken by the lecturer, it is considered that the students are listening to the lecture with interest.

  In addition, the related lecture evaluation unit 27 evaluates the lecture using the result of statistical analysis acquired by the related statistical analysis unit 26 for each student raw image at each time point when the change detection unit 24 detects the change. Also good. For example, the related lecture evaluation unit 27 normalizes the number of face detections by one or more analysis items acquired by the related statistical analysis unit 26 for each time point when a change designated in advance in the lecturer information is detected. It is determined whether or not (value) exceeds a threshold value specified in advance for each analysis item, and if it exceeds, it is determined that the condition is satisfied, and an evaluation result associated with this condition may be acquired.

  Further, the related lecture evaluation unit 27 may acquire the evaluation result of one lecture using the evaluation result of the lecture at each time point of the raw image of the one lecture acquired as described above. For example, when all or some of the evaluation results of lectures acquired for a plurality of time points of a student image of one lecture satisfy a predetermined condition, the evaluation results associated with this condition are It may be acquired as an evaluation result of one lecture. For example, among the evaluation results of lectures acquired for a plurality of time points, the number of evaluation results specified in advance (for example, the evaluation result indicating that the student's interest is maintained) is greater than or equal to a predetermined threshold value. If the condition is satisfied, it may be determined that the condition is satisfied, and an evaluation result of “quality lecture” associated with the condition may be acquired. Alternatively, evaluation results of a plurality of lectures as a whole may be acquired using evaluation results of a plurality of lectures.

  As with the lecture evaluation unit 15, it may be changed as appropriate according to the purpose of the lecture evaluation, etc., as with the lecture evaluation unit 15, what kind of student analysis statistical analysis results are used by the related lecture evaluation unit 27. .

  The statistical analysis result of the related statistical analysis unit 26 used by the related lecture evaluation unit 27 for the evaluation of the lecture is acquired by the related statistical analysis unit 26, and the output unit 28 described later accumulates in a storage medium (not shown). It may be the result of analysis.

  The output unit 28 outputs information related to the analysis result by the face analysis unit 25. In the present embodiment, in particular, the output unit 28 outputs information related to the analysis result of the face analysis unit 25 for the student raw image acquired immediately after the change detection unit 24 detects the change. The information related to the analysis result here is, for example, information on the analysis result acquired by the face analysis unit 25 with respect to the student raw image immediately after the change detection unit 24 detects the change. The analysis result information acquired by the face analysis unit 25 for the student raw image immediately after the change detection unit 24 detects the change, and the instructor information detected by the change detection unit 24 corresponding to the analysis result information. It may be information having at least one of information indicating the contents of the change, or information at the time of acquisition (for example, time) of the frame image or the like that is the acquisition target of the analysis result. Further, the information related to the analysis result by the face analysis unit 25 here is information acquired using the analysis result acquired by the face analysis unit 25 immediately after the change detection unit 24 detects the change. good. For example, the output unit 28 may output the result of statistical analysis acquired by the related statistical analysis unit 26 as information related to the analysis result here. The output unit 28 may output the evaluation result acquired by the related lecture evaluation unit 27 as information related to the analysis result by the face analysis unit 25.

  Further, the output unit 28 may further output information indicating the content of the change in the lecturer information detected by the change detection unit 24 as information related to the analysis result.

  Similarly to the output unit 17 of the first embodiment, when the face analysis unit 25 acquires a plurality of analysis results of the student's face in time series, the output unit 28 of the face analysis unit 25 You may make it output the analysis result of a face along a time series.

  Next, the operation of the face analysis apparatus 2 will be described using the flowchart of FIG. Here, the student image taken in advance for one lecture and the lecturer information including the lecturer image and the lecturer audio information are stored in advance in the student image storage unit 12 and the lecturer information storage unit 23, respectively. It shall be. It is assumed that a time code is associated with each frame image of the student raw image and the lecturer image. In the lecturer information, it is assumed that the lecturer image and the lecturer audio image are synchronized.

  (Step S1101) The face analysis apparatus 2 determines whether an instruction to analyze one student raw image stored in the student raw image storage unit 12 has been received via a receiving unit (not shown) or the like. If an instruction is accepted, the process proceeds to step S1102, and if not accepted, the process returns to step S1101.

  (Step S1102) The change detection unit 17 performs a process of detecting a change location from the lecturer image. For example, the change location is detected in order from the first frame image of the lecturer image.

  (Step S1103) The change detection unit 17 determines whether or not one change point has been detected. If detected, the process proceeds to step S1104. If not detected, the process proceeds to step S1105.

  (Step S1104) The change detection unit 17 associates the content of the detected change with the time corresponding to the frame image where the change is detected, and accumulates it in a storage medium (not shown). Then, the process returns to step S1102.

  (Step S1105) The change detection unit 17 determines whether or not there is a remaining frame image that has not been subjected to change detection processing in the lecturer image. If there is, the process returns to step S1102, and if not, the process proceeds to step S1106.

  (Step S1106) The change detection part 17 detects the location of a change from lecturer audio | voice information. For example, a process of detecting an output portion at a level higher than a predetermined level as a conversion portion from the head of the lecturer voice information is performed.

  (Step S1107) The change detection unit 17 determines whether or not a change point is detected in the lecturer voice information. If a change location is detected, the process proceeds to step S1108. If not detected, the process proceeds to step S1109.

  (Step S1108) The change detection unit 17 stores information (not shown) by associating information indicating the content of the change (for example, information indicating that a sound having a level exceeding the threshold is detected) and the time when the change is detected. Accumulate on media.

  (Step S1109) The change detection unit 17 determines whether there is remaining lecturer voice information for which no change is detected. If there is a remainder, the process returns to step S1106, and if there is no remainder, the process proceeds to step S1110.

  (Step S1110) The face analysis unit 25 substitutes 1 for the counter m.

  (Step S1111) The face analysis unit 25 determines whether there is an mth time among the times at which the changes accumulated in steps S1104 and S1108 are detected. If there is an mth time, the process proceeds to step S1112; otherwise, the process proceeds to step S1118.

  (Step S1112) The face analysis unit 25 sequentially recognizes one or more faces in the frame image of the student raw image corresponding to the mth time. For example, the face analysis unit 25 assigns face identification information to information indicating the outline of one or more recognized faces, and temporarily stores the information in a storage medium (not shown).

  (Step S1113) The face analysis unit 25 analyzes each face recognized in step S1112. For example, for each face recognized in step S1112, an analysis is performed on an analysis item designated in advance, and information indicating the analysis result is acquired. When performing face analysis, for example, a frame image before or after the frame image associated with the mth time may be used.

  (Step S1114) The face analysis unit 25 associates the identification information of each face, the analysis result for each face, and the time corresponding to the frame image in the analysis mode, and accumulates them in a storage medium (not shown). . As a result, the analysis results are output (in particular, accumulated here) along the time series. This accumulation may be performed by the output unit 28. Note that the face analysis unit 25 performs a face detection process on the frame image associated with the nth time, and performs an analysis as shown in step S1113 for each face each time a face is detected. The process of accumulating the analysis results may be repeated until no new face can be detected.

  (Step S1115) The related statistical analysis unit 26 performs statistical analysis using the analysis result of the face analysis unit 25 with respect to the frame image associated with the mth time. For example, for the analysis results of each face acquired in step S1113, the number of faces having the same analysis result is totaled. For example, information in which the total value is associated with information indicating the content of the change accumulated in association with the time corresponding to the frame image is an example of a statistical analysis result here. For example, the same analysis result means that the analysis item and the analysis value obtained for the analysis item are the same. In addition, when a plurality of analysis results are obtained for one face, a part of the same analysis result may be preferentially aggregated.

  (Step S1116) The statistical analysis unit 20 accumulates the related statistical analysis result, which is the result of the statistical analysis acquired in step S1107, in a storage medium or the like not shown in association with the mth time. This accumulation may be performed by the output unit 28.

  (Step S1117) The face analysis unit 25 increments the value of the counter m by 1. Then, the process returns to step S1111.

  (Step S1118) The related lecture evaluation unit 27 evaluates the lecture using the related statistical analysis result accumulated in step S1116. For example, if there is a statistical analysis result that satisfies a predesignated condition among the related statistical analysis results accumulated in step S1116, information indicating the evaluation of the lecture that is associated in advance with this condition is acquired. For example, each related statistical analysis result obtained corresponding to the time when a change was detected in instructor information correlates the total number of faces of laughing students with the content of changes detected in instructor information The change detected for the instructor information is the predesignated condition, the instructor audio information is higher than the predesignated level, and the number of faces of the laughing students is designated in advance. It is a condition that more than a predetermined number of statistical analysis results are detected, and if the evaluation of the lecture corresponding to this condition is an evaluation of “fun lecture”, it is accumulated in step S1108. If the statistical analysis result satisfies this condition, the evaluation result of the lecture “fun lecture” is obtained. Here, a plurality of different evaluations may be performed.

  (Step S1119) The lecture evaluation unit 21 accumulates the evaluation result acquired in step S1118 in a storage medium (not shown) or the like in association with the identification information of the lecture, the student raw image, or the like. This accumulation may be performed by the output unit 28.

  (Step S1120) The output unit 28 displays the statistical analysis result accumulated in step S1116 and the evaluation result accumulated in step S1119.

  (Step S1119) The output unit 28 determines whether or not to end the display of the evaluation result and the statistical analysis result. For example, when an operation for ending the display is received via a reception unit (not shown) or the like, it is determined to end the display. If the display is to be terminated, the display is terminated and the process returns to step S1101. If the display is not to be terminated, the process returns to step S1121.

  Note that the processing is ended by powering off or interruption for aborting the processing in the flowchart in FIG.

Hereinafter, a specific operation of the face analysis apparatus 2 in the present embodiment will be described.
It is assumed that the student raw image storage unit 12 stores student raw images similar to those described with reference to FIG. 3 in the specific example of the first embodiment.

  FIG. 12 is a diagram illustrating a part of the lecturer image 121 and the lecturer voice information 122 which are lecturer information stored in the lecturer information storage unit 23. The lecturer image 121 and the lecturer audio information 122 are the lecturer's moving image and audio information respectively acquired by the second imaging unit 21 and the audio acquisition unit 22 when the student raw image shown in FIG. 3 is captured. Here, a time code is shown below each frame image of the lecturer image 121. In addition, the lecturer image 121 and the lecturer audio information 122 are assumed to be synchronized.

  It is assumed that the user operates an input device (not shown) such as a mouse or a keyboard to give an instruction to analyze the student raw image shown in FIG.

  The change detection unit 24 reads out the lecturer image and the lecturer voice information stored in the lecturer information storage unit 23 from the head, and detects a change for each. Here, for example, for the lecturer image, the frame images are sequentially read out, and motion detection is performed using the difference between the previous and subsequent frame images. Then, it is determined whether or not an object movement (for example, a distance or the like) that is equal to or greater than a movement designated in advance in the lecturer image is detected. When a motion of an object that is equal to or greater than the motion specified in advance is detected, the change detection unit 24 determines that a change has been detected. In addition, regarding the lecturer voice information, it is judged in order from the beginning of the lecturer voice information whether or not there is voice information of a level specified in advance, and in some cases, it is judged that a change has been detected.

  When the change detection unit 24 detects a change in at least one of the lecturer image and the lecturer voice information, the face analysis unit 25 associates the change with the frame image or updated voice information of the part of the lecturer image where the change is detected. Read time code from instructor information. Then, a frame image associated with a time obtained by adding a predetermined time to the time indicated by the read time code is read from the student raw image shown in FIG. As in the specific example of the first embodiment, face recognition and analysis for each recognized face are performed. The pre-designated time is preferably a time suitable for determining whether, for example, the student's face change or the like is due to the instructor's action or the voice uttered by the instructor. Then, for example, it is set in advance to about 1 to 5 seconds. The reason for adding the time is that there is a time lag from when a change occurs in the teacher's action or voice until a change or reaction occurs in the student's face or the like. Here, for example, it is assumed that the time obtained by adding the predesignated time is “10: 35: 38.21”.

  The information indicating the change target detected by the change detection unit 24, the analysis result of each face of the face analysis unit 25, and the time code “10:35:38. 21 ”is stored in association with a storage medium (not shown). Here, it is assumed that the information indicating the change target is, for example, information indicating whether the change detected by the change detection unit 24 is a change for the lecturer image or a change for the lecturer audio information.

  FIG. 13 is a change time analysis result management table for managing the face analysis results and the like accumulated by the change detection unit 24. This change-time analysis result management table is referred to as “change target” which is information indicating the content of the change detected by the change detection unit 24 in the face analysis result management table shown in FIG. 5 in the specific example of the first embodiment. An item is added. Of the values of “change target”, “instructor image” indicates that the change detected by the change detection unit 24 is a change in the instructor image, and “instructor voice information” indicates the change detected by the change detection unit 24. Indicates a change in the lecturer image.

  Next, the related statistical analysis unit 26 statistically analyzes the analysis result of the face analysis unit 25. Here, as shown in FIG. 13, statistical analysis is performed on the analysis result associated with the frame image whose time code is “10: 35: 38.21” and the “instructor image” which is the value of “change target”. I do. Specifically, in the analysis result of the frame image with the time code “10: 35: 38.21”, the number of faces whose “expression” is “smile” and “line-of-sight direction” is “front” The number of faces, the number of faces whose “face direction” is “front”, and the number of faces whose “blink” is “closed” are respectively tabulated. That is, aggregation is performed for each analysis item. Then, these total results are divided by the total number of recognized faces “30” to obtain the ratio of appearance of faces for each analysis item. Instead of calculating the ratio, variance or the like may be calculated. And the related statistical analysis part 26 respond | corresponds with the "instructor image" which is the value of time code "10: 35: 38.21" and the value of "change object" as a statistical analysis result for the acquired ratio for every analysis item. In addition, it is stored in a storage medium (not shown). In addition, when the analysis result to be subjected to statistical analysis is associated with “change object” “instructor image”, the statistical analysis result is associated with “change object” “instructor voice information”. Accumulated.

  After that, the change detection unit 24 repeats the process of detecting changes for the remaining lecturer images and lecturer audio information, and the face analysis unit 25 performs the change detection for each time a change is detected. The face analysis is performed on the frame image of the student image corresponding to the time obtained by adding the time specified in advance, and the related statistical analysis unit 26 performs the statistical analysis.

  FIG. 14 is a diagram showing a statistical analysis result management table for managing the results of statistical analysis accumulated by the related statistical analysis unit 26. The related statistical analysis result management table is obtained by adding the item “change target” as shown in FIG. 13 to the statistical analysis result management table shown in FIG.

  Next, the related lecture evaluation unit 27 evaluates the lecture corresponding to the student image using the statistical analysis result acquired by the related statistical analysis unit 26 as shown in FIG. Here, for example, as a condition for evaluating a lecture, “the number of times when“ change target ”is“ lecturer voice information ”and the ratio of“ expression (smile) ”is“ 60% or more ” However, the condition “is over 40” is stored in advance in a storage medium (not shown), and the value indicating the evaluation of the lecture when this condition is satisfied is called “lecture concentrated on the lecturer's story”. It is assumed that the value is stored in a storage medium (not shown) in association with this condition. The time here may be considered as each frame image to be analyzed, or may be considered as each record in the related statistical analysis result management table shown in FIG.

  The related lecture evaluation unit 27 reads the above conditions. Next, in the related statistical analysis result management table shown in FIG. 14, records whose “change target” is “lecturer voice information” and whose “expression (smile)” ratio is “60% or more” are recorded. Count the number. For example, assume that the count number is “52”. Then, it is determined whether or not the acquired count number is “40 or more”. Here, since it is 40 or more, the evaluation result “lecture concentrated on the lecturer's story” is acquired. When the number is less than 40, an evaluation result “lecture not concentrated on the lecturer's story” may be acquired. The acquired evaluation result is stored in a storage medium (not shown) in association with the file name of the student raw image, for example.

  The output unit 28 displays the statistical analysis result acquired by the related statistical analysis unit 26 and the evaluation result acquired by the related lecture evaluation unit 27 on a monitor or the like as in FIG.

  Further, when the output unit 28 receives an instruction to display the analysis result of the face acquired by the face analysis unit 25, the output unit 28 acquired by the face analysis unit 25 managed by the change analysis result management table shown in FIG. As shown in FIG. 8, the face analysis result is displayed on a monitor or the like in time series. For example, the output unit 17 displays the analysis results side by side along a time series.

  As in the first embodiment, a specification for the analysis result of the face at one point output by the output unit 28 is received, and when the specification is received, a frame image corresponding to the specified analysis result is displayed. Similarly to the specific example of the first embodiment, the output unit 28 may read out from the student image and display it.

  As described above, according to the present embodiment, it is possible to appropriately analyze the lecture by using the analysis result of the lecturer information and the analysis result of the face of the student during the lecture.

  In each of the above embodiments, each process (each function) may be realized by centralized processing by a single device (system), or by distributed processing by a plurality of devices. May be.

  Further, although cases have been described with the above embodiments where the face analysis device is a stand-alone device, the face analysis device may be a stand-alone device or a server device in a server / client system. In the latter case, the output unit or the reception unit receives an input or outputs a screen via a communication line.

  In each of the above embodiments, the constituent elements such as the face analysis unit 13, the statistical analysis unit 14, the lecture evaluation unit 15, the change detection unit 24, the face analysis unit 25, the related statistical analysis unit 26, the related lecture evaluation unit 27, etc. May be configured by dedicated hardware, or may be configured by software and realized by executing a program. For example, each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

  The software that realizes the face analysis device in each of the above embodiments is a program as described below. In other words, this program uses a computer that can access a student image storage unit in which a student image, which is a moving image obtained by capturing one or more students' faces during a lecture, from a student image to a student's face. Is a program for functioning as a face analysis unit that analyzes the recognized face and an output unit that outputs information related to the analysis result by the face analysis unit.

  In the program, the functions realized by the program do not include functions that can be realized only by hardware. For example, a function that can be realized only by hardware such as a modem or an interface card in an acquisition unit that acquires information or an output unit that outputs information is not included in the function realized by the program.

  Further, the computer that executes this program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

  FIG. 15 is a schematic diagram showing an example of the external appearance of a computer that executes the program and realizes the face analysis apparatus according to the embodiment. The above-described embodiment can be realized by computer hardware and a computer program executed on the computer hardware.

  15, a computer system 900 includes a computer 901 including a CD-ROM (Compact Disk Read Only Memory) drive 905, an FD (Floppy (registered trademark) Disk) drive 906, a keyboard 902, a mouse 903, a monitor 904, and the like. Is provided.

  FIG. 16 is a diagram showing an internal configuration of the computer system 900. In FIG. 16, in addition to the CD-ROM drive 905 and the FD drive 906, a computer 901 is connected to an MPU (Micro Processing Unit) 911, a ROM 912 for storing a program such as a bootup program, and the MPU 911. A RAM (Random Access Memory) 913 that temporarily stores program instructions and provides a temporary storage space, a hard disk 914 that stores application programs, system programs, and data, and an MPU 911 and a ROM 912 are interconnected. And a bus 915. The computer 901 may include a network card (not shown) that provides connection to the LAN.

  A program for causing the computer system 900 to execute the functions of the face analysis apparatus and the like according to the above embodiment is stored in the CD-ROM 921 or the FD 922, inserted into the CD-ROM drive 905 or the FD drive 906, and stored in the hard disk 914. May be forwarded. Instead, the program may be transmitted to the computer 901 via a network (not shown) and stored in the hard disk 914. The program is loaded into the RAM 913 when executed. The program may be loaded directly from the CD-ROM 921, the FD 922, or the network.

  The program does not necessarily include an operating system (OS) or a third-party program that causes the computer 901 to execute the functions of the face analysis apparatus according to the above-described embodiment. The program may include only a part of an instruction that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the computer system 900 operates is well known and will not be described in detail.

  The present invention is not limited to the above-described embodiments, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.

  As described above, the face analysis apparatus according to the present invention is suitable as an apparatus for analyzing a lecture, and is particularly useful as an apparatus for performing an analysis on a lecture using an image obtained by photographing a student's face. .

DESCRIPTION OF SYMBOLS 1, 2 Face analyzer 11 1st imaging | photography part 12 Student raw image storage part 13, 25 Face analysis part 14 Statistical analysis part 15 Lecture evaluation part 16 Specification reception part 17, 28 Output part 21 2nd imaging | photography part 22 Voice acquisition part 23 Instructor information storage unit 24 Change detection unit 26 Related statistical analysis unit 27 Related lecture evaluation unit 121 Instructor image 122 Instructor audio information

Claims (9)

A student image storage unit that stores a student image that is a moving image of a face of one or more students during the lecture;
Recognizing a student's face from the student image, performing an analysis on the recognized face, information indicating an analysis result for an analysis item including facial expression, face direction, gaze direction, and blinking, A face analysis unit for acquiring information indicating facial expressions, face orientations, gaze directions, and blinks;
A statistical analysis unit that counts the number of faces for each analysis item including facial expression, face orientation, gaze direction, and blink using information indicating the facial expression, face orientation, gaze direction, and blink ,
A lecture evaluation unit that evaluates the lecture by using the face count result by analysis item including the facial expression, face direction, line-of-sight direction, and blink,
A face analyzer including an output unit for outputting the evaluation result of the lecture evaluation unit ,
A lecturer information storage unit that stores lecturer information that is information including at least one of a lecturer image that is a moving image of a lecturer who is taking a lecture, or lecturer voice information that is voice information of a lecturer being lectured;
A change detector for detecting changes in the lecturer information;
An associated statistical analysis unit that performs a statistical analysis specified in advance using an analysis result for a student raw image acquired immediately after the change detection unit detects a change;
Using a result of statistical analysis of the related statistical analysis unit, further comprising a related lecture evaluation unit for evaluating the lecture,
The face analysis unit is a face analysis result for a student raw image acquired immediately after the change detection unit detects a change, and acquires information indicating an analysis result for a facial expression analysis item,
The output unit is a face analysis device that outputs an evaluation result of the related lecture evaluation unit . The statistical analysis unit
Aggregate the number of appearances of facial expressions of laughter, anger, or sadness,
The lecture evaluation section
The face analysis apparatus according to claim 1, wherein the lecture is evaluated using the number of appearances of the laughing, angry, or sad face. The face analysis apparatus according to claim 1 , wherein the output unit also outputs a result of statistical analysis of the related statistical analysis unit. The change detector is
The level of the lecturer audio information included in the lecturer information is detected at one or more time points that are equal to or higher than a predetermined threshold value,
The related statistical analysis unit
The number of laughing faces respectively detected from the student image immediately after the one or more time points detected by the change detection unit is totalized,
The related lecture evaluation section
The face according to any one of claims 1 to 3 , wherein an evaluation result indicating that the interest is a lecture is acquired when the total number of smiling faces is equal to or greater than a predetermined threshold value. Analysis equipment. The face analysis unit obtains the analysis results of the students' faces in time series,
The output unit, a face analyzer as claimed in any one claims 1 to 4 for outputting in a time series analysis of the face of the face analysis unit. A specification accepting unit that accepts a designation for the analysis result of the face at one time output by the output unit;
The face analysis device according to claim 5 , wherein the output unit outputs a raw student image at a time corresponding to an analysis result received by the designation receiving unit. The face analysis apparatus according to claim 6 , wherein when outputting the student image, the output unit outputs a student image in a display mode in which a student's face cannot be identified. A student image storage unit that stores a student image that is a moving image of the face of one or more students during a lecture, a lecturer image that is a moving image of a lecturer during a lecture, or a lecturer during a lecture Lecturer information storage unit that stores at least one of instructor audio information, which is audio information of the lecturer, a face analysis unit, a statistical analysis unit, a lecture evaluation unit, an output unit, and change detection A face analysis method performed using a department, a related statistical analysis section, and a related lecture evaluation section ,
The face analysis unit recognizes a student's face from the student image, performs analysis on the recognized face, and results of analysis on analysis items including facial expression, face direction, line-of-sight direction, and blinking A face analysis step for acquiring information indicating facial expression, facial orientation, gaze direction, and blinking,
The statistical analysis unit uses the information indicating the facial expression, facial orientation, gaze direction, and blinking to calculate the number of faces for each analysis item including facial expression, facial orientation, gaze direction, and blinking. A statistical analysis step to aggregate,
The lecture evaluation unit performs the evaluation of the lecture using the face count for each analysis item including the facial expression, face direction, gaze direction, and blink,
The output unit is a face analysis method including an output step of outputting an evaluation result in the lecture evaluation step ,
A change detecting step in which the change detecting unit detects a change in the lecturer information;
A related statistical analysis step in which the related statistical analysis unit performs a statistical analysis specified in advance using an analysis result for a student raw image acquired immediately after detecting a change in the change detection step;
The related lecture evaluation unit further includes a related lecture evaluation step for evaluating the lecture using a result of statistical analysis in the related statistical analysis step,
The face analysis step is a face analysis result for a student raw image acquired immediately after detecting a change in the change detection step, and acquires information indicating an analysis result for an analysis item of facial expression,
A face analysis method for outputting an evaluation result in the related lecture evaluation step in the output step . A student image storage unit that stores a student image that is a moving image of the face of one or more students during a lecture, a lecturer image that is a moving image of a lecturer during a lecture, or a lecturer during a lecture A computer that can access a lecturer information storage unit that stores lecturer information that is information including at least one of lecturer voice information that is voice information of
Recognizing a student's face from the student image, performing an analysis on the recognized face, information indicating an analysis result for an analysis item including facial expression, face direction, gaze direction, and blinking, A face analysis unit for acquiring information indicating facial expressions, face orientations, gaze directions, and blinks;
A statistical analysis unit that counts the number of faces for each analysis item including facial expression, face orientation, gaze direction, and blink using information indicating the facial expression, face orientation, gaze direction, and blink ,
A lecture evaluation unit that evaluates the lecture by using the face count result by analysis item including the facial expression, face direction, line-of-sight direction, and blink,
A program for functioning as an output unit that outputs the evaluation result of the lecture evaluation unit,
Computer
A change detector for detecting changes in the lecturer information;
An associated statistical analysis unit that performs a statistical analysis specified in advance using an analysis result for a student raw image acquired immediately after the change detection unit detects a change;
Using the result of statistical analysis of the related statistical analysis unit, further function as a related lecture evaluation unit for evaluating the lecture,
The face analysis unit is a face analysis result for a student raw image acquired immediately after the change detection unit detects a change, and acquires information indicating an analysis result for a facial expression analysis item,
The output unit is a program for causing a computer to function as an output of the evaluation result of the related lecture evaluation unit .

Images