JP2018049480A - Information processing apparatus, evaluation system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract behavior defined as non-language information, for analysis, from an image obtained by imaging a participant, thereby achieving accurate evaluation.SOLUTION: An information processing apparatus includes: an action detection unit 230 which specifies a section of a human body included in video data, to detect an action of the specified section; a non-language information extraction unit 240 which extracts behavior defined to be evaluated in predetermined evaluation items, on the basis of the action of the section of the human body detected by the action detection unit 230; and a response evaluation unit 250 which evaluates each of the evaluation items, on the basis of the behavior extracted by the non-language information extraction unit 240 and evaluation criteria determined in advance for each of the evaluation items.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an information processing apparatus, an evaluation system, and a program.

  Attempts have been made to take pictures of participants in lectures and meetings, and to analyze and evaluate them by image analysis. Patent Document 1 recognizes a student's face from a student image storage unit that stores a student image, which is a moving image obtained by capturing one or more students' faces during a lecture, There has been disclosed a face analysis apparatus for analyzing a lecture by including a face analysis unit for analyzing a recognized face and an output unit for outputting information related to the analysis result by the face analysis unit.

JP 2013-61906 A

  Participants in lectures and gatherings emit non-verbal information in response to various reactions such as gestures and body orientation in addition to facial expressions. Therefore, it is possible to perform analysis and evaluation with higher accuracy by capturing non-linguistic information issued by participants including not only facial analysis but also body movements. Also, if you try to evaluate the movement of the body part you have focused on, it will be evaluated including the meaningless movement that was not issued by the participant as non-verbal information, which may reduce the accuracy of the evaluation. is there.

  An object of the present invention is to realize a highly accurate evaluation by extracting and analyzing an action defined as non-linguistic information from an image obtained by photographing a participant.

An information processing apparatus according to claim 1 of the present invention provides:
An action detection unit that identifies a part of the human body shown in the video data and detects the action of the specified part;
An action extraction unit for extracting an action defined as an evaluation target in a predetermined evaluation item based on the movement of the part of the human body detected by the movement detection unit;
An evaluation unit that performs evaluation for each evaluation item based on the behavior extracted by the behavior extraction unit and an evaluation criterion that is predetermined for each evaluation item;
An information processing apparatus comprising:
In the information processing apparatus according to claim 2, the evaluation unit includes an evaluation item in which the action is an evaluation target based on at least one of the type, the appearance frequency, and the duration of the action defined as the evaluation target. The information processing apparatus according to claim 1, wherein a degree of evaluation is specified.
The information processing apparatus according to claim 3 defines, for a specific evaluation item, an action corresponding to the first classification and an action corresponding to the second classification, which are contradictory evaluations, and the evaluation unit includes the specific evaluation With respect to the item, based on the evaluation based on the appearance of the action corresponding to the first classification and the evaluation based on the appearance of the action corresponding to the second classification, the specific evaluation item is evaluated. An information processing apparatus according to claim 1 or claim 2.
The information processing apparatus according to claim 4, wherein the evaluation unit continuously evaluates the same evaluation item, and generates time-series data indicating a change in the evaluation result over time. An information processing apparatus according to any one of claims 1 to 3.
An evaluation system according to claim 5 is an acquisition means for acquiring moving image data;
Behavior evaluation means for analyzing the video data acquired by the acquisition means and evaluating the behavior of a person shown in the video;
Output means for outputting an evaluation result by the behavior evaluation means,
The behavior evaluation means includes
An action detection unit for identifying a part of a human body shown in the moving image data and detecting an action of the specified part;
An action extraction unit for extracting an action defined as an evaluation target in a predetermined evaluation item based on the movement of the part of the human body detected by the movement detection unit;
An evaluation unit that performs evaluation for each evaluation item based on the behavior extracted by the behavior extraction unit and an evaluation criterion that is predetermined for each evaluation item;
An evaluation system characterized by comprising:
In the evaluation system according to claim 6, with respect to a specific evaluation item, an action corresponding to the first classification and an action corresponding to the second classification that are contradictory evaluations are defined, and the evaluation unit of the action evaluation means includes: Regarding the specific evaluation item, based on the evaluation based on the appearance of the action corresponding to the first classification and the evaluation based on the appearance of the action corresponding to the second classification, the evaluation on the specific evaluation item is performed,
The output means includes a display unit for displaying an image, and compares the evaluation based on the appearance of the behavior corresponding to the first category with the evaluation based on the appearance of the behavior corresponding to the second category. 6. The evaluation system according to claim 5, wherein display is output on the display unit.
In the evaluation system according to claim 7, the output unit includes a display unit that displays an image, and the evaluation result continuously performed on the same evaluation item by the evaluation unit of the behavior evaluation unit is obtained over time. The evaluation system according to claim 5 or 6, wherein an image to be displayed as a graph indicating a change in the evaluation result is displayed and output on the display unit.
A program according to claim 8 is a computer,
A motion detection means for identifying a part of the human body shown in the video data and detecting a motion of the identified part;
Action extracting means for extracting an action defined as an evaluation target in a predetermined evaluation item based on the action of the part of the human body detected by the action detecting means;
Based on the behavior extracted by the behavior extraction means and the evaluation criteria predetermined for each of the evaluation items, functioning as an evaluation means for performing evaluation for each evaluation item;
It is a program characterized by.

According to invention of Claim 1, compared with the structure which evaluates the motion itself of the body part acquired as a moving image, highly accurate evaluation based on non-language information can be performed.
According to the second aspect of the present invention, it is possible to perform a highly accurate evaluation as compared with a configuration in which a binary evaluation of whether or not a specific operation has been performed is performed.
According to the invention of claim 3, it is possible to perform a highly accurate evaluation combining a plurality of evaluation items as compared with a configuration in which a binary evaluation of whether or not a specific operation has been performed is performed.
According to the fourth aspect of the present invention, it is possible to identify a change in evaluation contents with the passage of time and perform highly accurate evaluation as compared with a configuration for simply evaluating whether or not a specific operation has been performed. .
According to invention of Claim 5, compared with the structure which evaluates the motion itself of the body part acquired as a moving image by the acquisition means, highly accurate evaluation based on non-language information can be performed.
According to the sixth aspect of the present invention, it is possible to perform a highly accurate evaluation combining a plurality of evaluation items as compared with a configuration in which a binary evaluation of whether or not a specific operation has been performed is performed.
According to the seventh aspect of the present invention, it is possible to identify a change in evaluation contents with the passage of time and perform highly accurate evaluation as compared with a configuration that simply evaluates whether or not a specific operation has been performed. .
According to the eighth aspect of the present invention, the computer executing the program of the present invention performs a highly accurate evaluation based on non-linguistic information, as compared with the configuration for evaluating the movement of the body part acquired as a moving image. be able to.

It is a figure which shows the structural example of the non-linguistic information evaluation system to which this embodiment is applied. It is a figure which shows the hardware structural example of information processing apparatus. It is a figure which shows the function structure of information processing apparatus. It is a figure which shows the hardware structural example of a terminal device. It is a figure which shows the function structure of a terminal device. FIG. 6A is a diagram for explaining a technique for specifying a region related to a human body using inter-frame feature values, and FIG. 6A is a diagram showing a person sitting sideways on a chair in one frame of a moving image. FIG. 6B is a diagram showing a state where the same person has moved forward in another frame of the moving image. It is a figure which shows the example of the image of the evaluation subject acquired with a video camera in a 1st application scene. It is a figure which shows the example of the image of the evaluation subject acquired with a video camera in a 2nd application scene. It is a figure which shows the example of the output image of an evaluation result. It is a figure which shows the other example of the output image of an evaluation result.

<Configuration of non-linguistic information evaluation system to which this embodiment is applied>
FIG. 1 is a diagram illustrating a configuration example of a non-language information evaluation system to which the present embodiment is applied. As shown in FIG. 1, the non-linguistic information evaluation system 10 according to the present embodiment outputs a video camera 100 as a moving image acquisition device, an information processing device 200 as a moving image analysis device, and an analysis result by the information processing device 200. And a terminal device 300 as an output device. The video camera 100 and the information processing device 200, and the information processing device 200 and the terminal device 300 are connected via the network 20, respectively.

  The network 20 is not particularly limited as long as it can perform information communication between the video camera 100 and the information processing apparatus 200, and between the information processing apparatus 200 and the terminal apparatus 300. For example, the network 20 may be the Internet or a LAN (Local Area Network). A communication line used for information communication may be wired or wireless. The network 20 that connects the video camera 100 and the information processing device 200 and the network 20 that connects the information processing device 200 and the terminal device 300 may be a common network or different networks. . Although not particularly illustrated, the network 20 is appropriately provided with a relay device such as a gateway or a hub for connecting a network or a communication line.

  The non-linguistic information evaluation system 10 of this embodiment analyzes a moving image of a person who is an evaluation object or a person constituting the evaluation object, and extracts and extracts non-language information such as the movement of the person and facial expressions. Evaluate the evaluation target based on non-linguistic information. The non-linguistic information evaluation system 10 according to the present embodiment is an object such as an interview or an evaluation of a participant in a place or scene where many people gather, such as classes, lectures, events, entertainment facilities, and the like. Evaluate the state of individuals in situations where individuals are fixed. The evaluation target, the evaluation item, the evaluation content, and the like are set according to the application target and application scene of the non-linguistic information evaluation system 10. For example, the evaluation target may be an individual person, or may be a set (group, team, etc.) of a plurality of persons (hereinafter, such evaluation target person or evaluation target). The persons who make up the set are called “evaluators”). The evaluation item is set, for example, whether the person or the set of persons to be evaluated is concentrated on something, whether it is actively active, etc. The evaluation content is, for example, a degree suitable for such an evaluation item ( Degree) etc. are judged. These evaluations will be described later with specific application examples.

  In the system shown in FIG. 1, the video camera 100 is an example of a moving image data acquisition unit, and is installed in a classroom, a lecture hall, an event hall, an amusement facility, or the like according to an application target of evaluation according to the present embodiment. Take a picture of the person being evaluated. In the present embodiment, a moving image of an evaluation subject photographed by the video camera 100 is analyzed, and non-linguistic information such as motion and facial expression is extracted. Accordingly, the type and the number of installed video cameras 100 are determined so that the evaluation subject's actions and facial expressions can be identified according to the configuration of the evaluation subject (individual or collective, etc.), the installation location, the shooting range, and the like. Is set. For example, a telephoto camera is used for photographing an individual in a wide place, and a wide-angle camera is used for photographing a plurality of persons existing in a wide range. Further, a plurality of cameras may be installed in various directions in order to photograph various parts of the body of the evaluation subject. Further, by capturing a wide range using a high-resolution camera, a plurality of images may be acquired, and the obtained image may be enlarged so that an individual image is an analysis target. In the present embodiment, the video camera 100 has a function of transmitting a captured moving image as digital data to the information processing apparatus 200 via the network 20.

  The information processing apparatus 200 is an example of a behavior evaluation unit, and is a computer (server) that analyzes a moving image taken by the video camera 100, extracts non-linguistic information about an evaluation target person, and evaluates it. The information processing apparatus 200 may be configured by a single computer or may be configured by a plurality of computers connected to the network 20. In the latter case, the function as the information processing apparatus 200 of the present embodiment to be described later is realized by distributed processing by a plurality of computers.

  FIG. 2 is a diagram illustrating a hardware configuration example of the information processing apparatus 200. As illustrated in FIG. 2, the information processing apparatus 200 includes a central processing unit (CPU) 201 that is a control unit and a calculation unit, a RAM 202 and a ROM 203, an external storage device 204, and a network interface 205. The CPU 201 performs various controls and arithmetic processes by executing programs stored in the ROM 203. The RAM 202 is used as a working memory in the control and arithmetic processing by the CPU 201. The ROM 203 stores various data used in programs executed by the CPU 201 and control. The external storage device 204 is realized by, for example, a magnetic disk device or a non-volatile semiconductor memory that can read and write data, and stores a program that is expanded in the RAM 202 and executed by the CPU 201 and a result of arithmetic processing by the CPU 201. The network interface 205 is connected to the network 20 and transmits / receives data to / from the video camera 100 and the terminal device 300. The configuration example illustrated in FIG. 2 is merely an example of a hardware configuration that implements the information processing apparatus 200 with a computer. The specific configuration of the information processing apparatus 200 is not limited to the configuration example illustrated in FIG. 2 as long as the functions described below can be realized.

  FIG. 3 is a diagram illustrating a functional configuration of the information processing apparatus 200. As illustrated in FIG. 3, the information processing apparatus 200 includes a moving image data acquisition unit 210, a region identification unit 220, an operation detection unit 230, a non-language information extraction unit 240, a reaction evaluation unit 250, and an output unit 260. Is provided.

  For example, in the computer shown in FIG. 2, the moving image data acquisition unit 210 is realized by the CPU 201 executing a program and controlling the network interface 205. The moving image data acquisition unit 210 receives moving image data from the video camera 100 via the network 20. The received moving image data is stored, for example, in the RAM 202 or the external storage device 204 shown in FIG.

  For example, in the computer shown in FIG. 2, the area identification unit 220 is realized by the CPU 201 executing a program. The area identifying unit 220 analyzes the moving image acquired by the moving image data acquiring unit 210, and identifies an area in which the part of the evaluation target person extracted as non-linguistic information by the non-linguistic information extracting unit 240 in the subsequent stage is shown. Specifically, areas where the human body (the whole) is shown, areas where the human head, body, arms, hands, fingers, etc. are reflected, head face, eyes, mouth, nose, ears, etc. The area where the body is reflected, the area where the upper body and the lower body are reflected, and the area where each body feature point is reflected, etc. (Hereinafter, the whole body part or part of the human body is not particularly distinguished. Called). As the parts to be identified, all the predetermined parts may be identified, or parts used for these processes based on the extraction by the subsequent non-linguistic information extraction unit 240 and the evaluation by the reaction evaluation unit 250 Only may be identified.

  For example, in the computer shown in FIG. 2, the operation detection unit 230 is realized by the CPU 201 executing a program. The motion detection unit 230 identifies the body part shown in each region based on the identification result of the region identification unit 220, and detects the motion for each identified region. Specifically, motions such as head movements, face orientations, face component movements (eyes, mouth, etc.), arm and leg movements, body orientations, body movements (walking, etc.) are detected. . As the motion of the detection target, all of the predetermined motions regarding a predetermined portion may be detected as targets, or the content of the evaluation by the non-linguistic information extraction unit 240 in the subsequent stage or the evaluation by the reaction evaluation unit 250 may be used. Based on this, only the movement of the part used for these processes may be detected.

  The non-linguistic information extraction unit 240 is realized by the CPU 201 executing a program in, for example, the computer shown in FIG. The non-linguistic information extraction unit 240 uses, based on the movement of the part detected by the motion detection unit 230, the evaluation target person's behavior (non-linguistic information) used for evaluation for each evaluation item of the reaction evaluation unit 250. Extract. In other words, the non-linguistic information extraction unit 240 is an action extraction unit that extracts an action defined as non-linguistic information issued by the person to be evaluated. Specifically, for example, nodding, moving the face in a specific direction or changing the direction of the face, changing facial expressions, moving the mouth, speaking, yawning, or falling asleep It extracts motions when you are in motion, moving your eyes, raising your hands, writing, typing your keyboard, turning, turning poor, etc.

  For example, in the computer shown in FIG. 2, the reaction evaluation unit 250 is realized by the CPU 201 executing a program. The response evaluation unit 250 evaluates the response of the evaluation target person based on an evaluation criterion predetermined for each evaluation item with respect to the behavior of the non-language information for each evaluation item extracted by the non-linguistic information extraction unit 240. . The evaluation content is set according to the scene (class, lecture, event, etc.) to which the non-linguistic information evaluation system 10 of this embodiment is applied. Specifically, for example, concentration, activity, progress, aggressiveness, responsiveness, etc. are evaluated.

  For example, in the computer shown in FIG. 2, the output unit 260 is realized by the CPU 201 executing a program and controlling the network interface 205. The output unit 260 transmits information on the evaluation result by the reaction evaluation unit 250 to the terminal device 300 via the network 20.

  The terminal device 300 is an example of an output unit, and is an information terminal (client) that outputs an evaluation result by the information processing device 200. As the terminal device 300, for example, a device including an image display unit as an output unit such as a personal computer, a tablet terminal, or a smartphone is used.

  FIG. 4 is a diagram illustrating a hardware configuration example of the terminal device 300. As illustrated in FIG. 4, the terminal device 300 includes a CPU 301, a RAM 302 and a ROM 303, a display device 304, an input device 305, and a network interface 306. The CPU 301 performs various controls and arithmetic processes by executing programs stored in the ROM 303. The RAM 302 is used as a work memory in the control and arithmetic processing by the CPU 301. The ROM 303 stores various data used in programs executed by the CPU 301 and control. The display device 304 is configured by a liquid crystal display, for example, and displays an image under the control of the CPU 301. The input device 305 is realized by an input device such as a keyboard, a mouse, or a touch sensor, for example, and accepts an operator's input operation. As an example, when the terminal device 300 is a tablet terminal or a smartphone, a touch panel in which a liquid crystal display and a touch sensor are combined functions as the display device 304 and the input device 305. The network interface 306 is connected to the network 20 and transmits / receives data to / from the video camera 100 and the terminal device 300. The configuration example illustrated in FIG. 4 is merely an example of a hardware configuration that implements the terminal device 300 with a computer. The specific configuration of the terminal device 300 is not limited to the configuration example illustrated in FIG. 4 as long as the functions described below can be realized.

  FIG. 5 is a diagram illustrating a functional configuration of the terminal device 300. As illustrated in FIG. 5, the terminal device 300 according to the present embodiment includes an evaluation result acquisition unit 310, a display image generation unit 320, a display control unit 330, and an operation reception unit 340.

  For example, in the computer shown in FIG. 4, the evaluation result acquisition unit 310 is realized by the CPU 301 executing a program and controlling the network interface 306. The evaluation result acquisition unit 310 receives evaluation result data from the information processing apparatus 200 via the network 20. The received evaluation result data is stored, for example, in the RAM 302 of FIG.

  For example, in the computer shown in FIG. 4, the display image generating unit 320 is realized by the CPU 301 executing a program. The display image generation unit 320 generates an output image indicating the evaluation result based on the evaluation result data acquired by the evaluation result acquisition unit 310. The configuration and display mode of the generated output image can be set according to the evaluation items, evaluation contents, and the like. Details of the output image will be described later.

  For example, in the computer shown in FIG. 4, the display control unit 330 is realized by the CPU 301 executing a program. The display control unit 330 displays the output image generated by the display image generation unit 320 on, for example, the display device 304 in the computer shown in FIG. In addition, the display control unit 330 receives a command related to display on the display device 304 and performs control such as display switching based on the received command.

  For example, in the computer shown in FIG. 4, the operation receiving unit 340 is realized by the CPU 301 executing a program. The operation receiving unit 340 receives an input operation performed by the operator using the input device 305. Then, in accordance with the operation received by the operation receiving unit 340, the display control unit 330 performs display control of an output image or the like to the display device 304.

<Processing of area identification unit>
Processing performed by the area identification unit 220 of the information processing apparatus 200 will be described. The area identifying unit 220 identifies a part related to the motion of a person shown in the moving image from the moving image captured by the video camera 100. Various existing image analysis techniques may be applied to this part identification. For example, an existing identification method realized by a digital camera or the like may be used to identify a face or a smile. Moreover, based on the part (area | region) of the specific shape currently reflected on the moving image, arrangement | positioning of such a some part, etc., the area | region where the body part is reflected can be specified. Further, as an example, identification based on inter-frame feature values may be performed. Specifically, an inter-frame feature value is obtained based on a difference between two or more consecutive frames of moving image data. Here, as the interframe feature amount, for example, a color boundary (edge), a color change amount, a moving direction or a moving amount of an area specified by these, and the like are used. The inter-frame feature quantity for a preset time is accumulated, and the inter-frame feature quantity is classified and integrated based on the distance and similarity of the inter-frame feature quantity for each frame. Thereby, the area | region which changes in cooperation in a moving image is specified, and the area | region where the part of the body is reflected is identified.

  FIG. 6 is a diagram for explaining a method for specifying a region related to a human body using inter-frame feature values. FIG. 6A shows a person sitting sideways in one frame of the video, and FIG. 6B shows that the same person has moved forward in another frame of the video. It shows a state. In the example illustrated in FIG. 6, the region identification unit 220 identifies a range in which an approximate color is reflected based on the color boundary and the amount of change shown in FIG. Then, the region identification unit 220 compares the frame in FIG. 6A and the frame in FIG. 6B and surrounds the frame with a broken line frame in the image based on the moving direction and moving amount of the corresponding color range. In the area 221, it is recognized that a plurality of color ranges are moving in cooperation, and this area 221 is identified as an area in which the upper body of the human body is reflected. Referring to FIGS. 6A and 6B, the position and size of the region 221 change according to the movement of the color range constituting the human body (upper body). Although the two frames in FIGS. 6A and 6B are compared here, the human body is based on the result of accumulating inter-frame feature quantities such as a change in color range by comparing three or more frames. You may make it identify the area | region where is reflected.

<Processing of motion detection unit>
Processing by the motion detection unit 230 will be described. The motion detection unit 230 analyzes a region in which the body part identified by the region identification unit 220 is reflected, identifies which part is specifically reflected, and detects a motion for each identified part. Various existing image analysis techniques may be applied to this motion detection. The detected motion is a motion that can occur as a physical motion for each identified part. For example, moving your eyes closed or opening your mouth, changing your line of sight, moving your face up and down, left and right, bending your elbows and waving your arms, bending your fingers, stretching your hands and opening and closing your hands. Motion, bending and stretching of the waist, twisting of the body, bending and stretching of the knee, movement of shaking the legs, movement of the body by walking, etc. are detected. Note that these movements are merely examples, and movements that can be detected by the non-language information evaluation system 10 of the present embodiment are not limited to the movements presented above. In the present embodiment, the motion detection unit 230 may detect the movements of all parts identified as regions by the region identification unit 220 or specify the motions extracted by the non-linguistic information extraction unit 240 at the subsequent stage. Therefore, the detection may be limited to the movement for the purpose. For example, if only the nodding motion is extracted by the non-linguistic information extraction unit 240, it is only necessary to detect a head movement that changes the face direction up and down.

<Processing of non-linguistic information extraction unit>
Processing by the non-language information extraction unit 240 will be described. Based on the movement of the part detected by the motion detection unit 230, the non-linguistic information extraction unit 240 extracts meaningful behavior that the evaluation target person has consciously or unconsciously performed as non-linguistic information. For example, a motion of nodding is extracted from a motion of changing the face direction up and down, a motion of utterance or absence is extracted from a motion of moving the mouth, and a motion of raising a hand is extracted from a motion of raising an arm. Extraction of non-linguistic information is not performed based solely on the movement of the part detected by the motion detection unit 230. For example, the movement of the corresponding part before and after the detected movement, the surrounding part, or another person This is also performed in consideration of information such as movements, scenes where the movements are detected, and context (background). As a specific example, when a motion that continuously changes the orientation of the face up and down is performed within a specific time, this motion is extracted as a motion of nodding. On the other hand, when the face is directed upward, and after a certain amount of time has passed, it moves downward and returns to its original state, and this movement is extracted as an action looking up upward for thinking. Also, when the face is facing down and a certain amount of time has elapsed, this action is extracted as an action indicating that the person is dozing. In addition, these operation | movement and consideration information are only illustrations, and the operation | movement and consideration information which can be extracted as non-language information in the non-language information evaluation system 10 of this embodiment are not limited to the operation | movement and information which were shown above.

<Processing of reaction evaluation unit>
Processing by the reaction evaluation unit 250 will be described. The reaction evaluation unit 250 evaluates the evaluation subject's reaction according to the scene to which the non-linguistic information evaluation system 10 is applied, based on the non-linguistic information extracted by the non-linguistic information extraction unit 240. For example, when evaluating the responses of students in a lecture, the degree of concentration on the lecture is an evaluation item. In the case of a participatory class, the degree of concentration and aggressiveness of each student, the activity level of the entire class, and the like are evaluation items. Moreover, you may make it evaluate simply whether it is a positive (positive) reaction or a negative (negative) reaction with respect to the objective of the scene where the non-linguistic information evaluation system 10 is applied. The evaluation result can be determined in various forms according to the evaluation item, the purpose of the evaluation, and the like. For example, a binary evaluation such as whether or not he / she spoke during the class may be performed, or a multi-value that specifies the achievement level (evaluation level) of evaluation items such as concentration and aggressiveness in stages Evaluation may be performed. The reaction evaluation unit 250 may perform continuous evaluation over a certain period of time during which a lecture or lesson is performed, and generate time-series evaluation information that changes with the passage of time.

  When performing multivalued evaluation, according to the evaluation items evaluated in the response evaluation unit 250, the types of actions (hereinafter referred to as reaction actions) defined (extracted) as non-linguistic information used for the evaluation and The appearance mode of the reaction behavior is set. In other words, even if the reaction behavior is the same, the evaluation is different depending on the appearance mode. For example, the evaluation differs between a case where a specific reaction action extracted as non-linguistic information is performed once and a case where the specific reaction action is repeated a plurality of times or continued for a certain time or more.

  Furthermore, when performing multi-level evaluation, for example, the degree of evaluation may be specified based on the type of reaction behavior, the appearance frequency, the duration, and the like in the evaluation item. As an example, let us consider a case where a nodding action and a writing action are defined as reaction behaviors to be evaluated for evaluating the degree of concentration. The nodding operation represents a higher degree of concentration than the writing operation. In this case, when a nodding action appears as a type of reaction action, it is evaluated that the degree of concentration is higher (the degree of evaluation is higher) than when a writing action appears. Further, as the appearance frequency of the reaction behavior, it is evaluated that the degree of concentration is higher (the degree of evaluation is higher) than the case where there are many nodding motions within a certain time. Moreover, it is evaluated that the degree of concentration is higher (the degree of evaluation is higher) when the duration of the writing operation within a certain time is longer than when the duration is shorter. Note that these evaluation items and evaluation methods are merely examples, and the evaluation items and evaluation methods that can be taken by the non-linguistic information evaluation system 10 of the present embodiment are not limited to the evaluation items and evaluation methods presented above.

<Application example>
Here, assuming specific application scenes, the extraction of non-linguistic information and reaction evaluation will be further described. The first application scene is a scene where speakers and students (listeners) are clearly separated, such as lectures and lectures, and almost only the speakers speak. It can be said that this is a scene (case) in which information transmission in one direction (in this case, the direction from the speaker to the student) is large. In this case, the student is evaluated as an evaluation target person.

  FIG. 7 is a diagram illustrating an example of an image of the evaluation subject acquired by the video camera 100 in the first application scene. In the example shown in FIG. 7, a situation is shown in which the students who are arranged in four rows and four columns in one direction are photographed from the speaker side. Therefore, each student is facing the video camera 100 in principle. In this example, each of 16 students is an evaluation subject.

  In the information processing apparatus 200, when the moving image data acquisition unit 210 receives the moving image data acquired by the video camera 100, the region identification unit 220 identifies the region where each student (evaluation target person) is shown from the acquired moving image. To do. Here, regions of the upper body region, face region, eye region, mouth region, face direction, head, and other parts are identified. Then, the motion detection unit 230 detects the movement of each student's part based on the region identified by the region identification unit 220.

  Next, the non-linguistic information extraction unit 240 extracts a specific action for each student as a reaction action representing the non-linguistic information based on the movement of each part of each student detected by the motion detection unit 230. For example, a nodding action, distraction, an eye closing action, a nagging action, a laughing action, and the like are extracted as reaction actions.

  Next, the reaction evaluation unit 250 evaluates the reaction behavior as the non-linguistic information extracted by the non-linguistic information extraction unit 240. As an evaluation method, for example, with respect to a specific evaluation item, an action corresponding to the first classification and an action corresponding to the second classification, which are contradictory evaluations, are defined. The evaluation based on the evaluation and the evaluation based on the appearance of the action corresponding to the second classification are integrated to obtain an evaluation result of this evaluation item. As an example, a case where positive reaction behavior and negative reaction behavior are defined and evaluated will be described. For example, the above-mentioned nodding action and laughing action are evaluated as positive responses. In each operation, the higher the appearance frequency or the longer the appearance time, the higher the evaluation. On the other hand, the absence, the closing of the eyes, the nagging, etc. are evaluated as negative reactions. And it is set as low evaluation, so that appearance frequency is large in each operation | movement, or appearance time is long.

  Here, an action defined as a positive reaction action is always evaluated as a positive reaction when the action appears. For example, when the nodding motion is defined as a positive reaction behavior as described above, when the evaluation subject performs the nodding motion, it is always evaluated as a positive response. The degree of positive evaluation changes depending on the mode of nodding operation. For example, if nodding is repeated many times or nodding with a large and slow motion, the positive degree is evaluated as high. On the other hand, when nodding lightly with a small action, it is evaluated as a positive reaction, but the degree is evaluated as low. Conversely, an action defined as a negative reaction behavior is always evaluated as a negative reaction when the action appears. For example, when the action of closing the eyes is defined as a negative reaction behavior as described above, if the evaluation subject performs an action of closing the eyes, the action is always evaluated as a negative reaction. And the degree of negative evaluation changes according to the mode of operation of closing eyes. For example, when the closed state continues for a long time, it is evaluated that the negative degree is high. On the other hand, if the closed eyes only last for a short time, it is evaluated as a negative reaction, but the degree is evaluated as low. In addition to the dual evaluation of positive (high evaluation) or negative (low evaluation), the evaluation is intermediate as an evaluation when there are not many positive and negative response actions. May be added.

  In addition to the above example, define the response behavior for evaluating aggressiveness and the response behavior for evaluating depolarization, and depending on the frequency and appearance time of the corresponding response behavior, You may evaluate whether it is passive (low evaluation) or intermediate. Also, meaningful and meaningless actions are defined, depending on the appearance frequency and appearance time of meaningful actions, the appearance frequency and appearance time of meaningless actions, high evaluation, low evaluation, intermediate You may make an evaluation.

  Furthermore, instead of the above-described evaluation based on the dual contents, it may be evaluated whether or not a specific characteristic is strong. For example, the reaction behavior for evaluating the degree of understanding is defined, and if there is no corresponding reaction behavior, the understanding level is 0 (zero), and the appearance frequency of the corresponding reaction behavior is large or the appearance time is long. Assess that the level of understanding is high. Similarly, a reaction behavior for evaluating the activity is defined, and it is evaluated that the activity is higher as the appearance frequency of the corresponding reaction behavior is higher or the appearance time is longer. Further, a reaction behavior for evaluating the degree of facilitate is defined, and it is evaluated that the degree of facilitation is higher as the appearance frequency of the corresponding reaction behavior is higher or the appearance time is longer. Also, reaction behavior for evaluating the degree of concentration is defined, and it is evaluated that the degree of concentration is higher as the appearance frequency of the corresponding reaction behavior is higher or the appearance time is longer. Further, a reaction behavior for evaluating the degree of calm is defined, and it is evaluated that the degree of calmness is higher as the appearance frequency of the corresponding reaction behavior is higher or the appearance time is longer. Note that, when an evaluation target person is evaluated by a plurality of evaluation items, an action as one non-language information may be defined as a reaction action in a plurality of evaluation items.

  Next, an application scene different from the above example will be described. The second application scene is a scene where a plurality of participants speak each other, such as a meeting. It can be said that it is a scene (case) where two-way information transmission is performed. Here, it is assumed that each participant is evaluated as an evaluation subject.

  FIG. 8 is a diagram illustrating an example of an image of the evaluation subject acquired by the video camera 100 in the second application scene. In the example shown in FIG. 8, a situation is shown in which a scene in which five participants are sitting around a table is photographed. Accordingly, the participants face each other regardless of the position of the video camera 100. In this example, each of the five participants is an evaluation subject.

  In the information processing apparatus 200, when the moving image data acquisition unit 210 receives the moving image data acquired by the video camera 100, the region identification unit 220 identifies the region where each student (evaluation target person) is shown from the acquired moving image. To do. Here, regions of the upper body region, face region, eye region, mouth region, face orientation, head, body, arm, hand, leg, and other parts are identified. Then, the motion detection unit 230 detects the movement of each student's part based on the region identified by the region identification unit 220.

  Next, the non-linguistic information extraction unit 240 extracts a specific action for each student as a reaction action representing the non-linguistic information based on the movement of each part of each student detected by the motion detection unit 230. For example, a nodding action, a speaking action, a gesture, a hand gesture, a blinking action, a writing action, a keyboard key hitting action, a poor motion, a stretch, an eye closing action, and the like are extracted as reaction actions. Further, when the opening / closing operation of the mouth is performed for a certain time, it may be detected as a speech operation.

  Next, the reaction evaluation unit 250 evaluates the reaction behavior as the non-linguistic information extracted by the non-linguistic information extraction unit 240. As an example, a case will be described in which a reaction behavior indicating that a participant is actively participating and a reaction behavior indicating that a participant is actively participating are defined and evaluated. For example, the above-described nodling action, speaking action, gesture, hand gesture, blinking action, writing action, and keyboard keying action are evaluated as reaction actions indicating active participation. In each operation, the higher the appearance frequency or the longer the appearance time, the higher the evaluation. On the other hand, the lack of action, the action of closing the eyes, and the poor prayer are evaluated as reaction behaviors indicating negative participation. And it is set as low evaluation, so that appearance frequency is large in each operation | movement, or appearance time is long. Similar to the evaluation in the first application scene described above, an intermediate evaluation may be added in addition to a binary evaluation of positive (high evaluation) or passive (low evaluation).

  In addition to the above example, define positive response behavior and negative response behavior, and whether it is positive (high evaluation) or negative (low evaluation) depending on the frequency and time of appearance of the corresponding response behavior An intermediate evaluation may be performed. Also, meaningful and meaningless actions are defined, depending on the appearance frequency and appearance time of meaningful actions, the appearance frequency and appearance time of meaningless actions, high evaluation, low evaluation, intermediate You may make an evaluation.

  Furthermore, as in the evaluation in the first application scene described above, it may be evaluated whether or not the specific characteristic is strong. That is, reaction behaviors for evaluating the degree of understanding, activity, facilitating, concentration, calmness, etc. are defined, respectively, and if there is no corresponding reaction behavior, the degree of the evaluation target characteristic is 0 (zero) It is evaluated that the degree of the characteristic of the evaluation target is higher as the appearance frequency of the corresponding reaction action is higher or the appearance time is longer. Note that, when an evaluation target person is evaluated by a plurality of evaluation items, an action as one non-language information may be defined as a reaction action in a plurality of evaluation items.

  As described above, depending on the specific scene where the non-linguistic information evaluation system 10 of the present embodiment is applied, the evaluation item, the evaluation content, the target behavior used for the evaluation for each evaluation item, and how each target behavior is determined. The evaluation method such as whether or not to evaluate is individually set. In the above example, the evaluation method for individual participants in lectures and classes (first application scene) and meetings (second application scenes) has been explained. Based on the evaluation of all participants, lectures and classes, The degree of concentration, activity, aggressiveness, etc. of the scene itself such as a meeting may be evaluated. The evaluation of the entire participant may be performed using, for example, a cumulative value or a representative value (average value, median value, etc.) of evaluation values of individual participants. The evaluation of the scene itself is variously defined according to the purpose of evaluation, such as giving a high evaluation to a lecture with many highly concentrated participants, or a high evaluation to a meeting with many active participants. You can go.

<Example output of evaluation results>
An example of an evaluation result output by the terminal device 300 will be described. The result of the evaluation performed by the reaction evaluation unit 250 in the information processing device 200 is sent to the terminal device 300 by the output unit 260. In the terminal device 300, the evaluation result acquisition unit 310 receives the evaluation result data transmitted from the information processing device 200. The display image generation unit 320 generates an output image that visually indicates the evaluation result based on the acquired evaluation result data. The display control unit 330 causes the display device 304 to display the generated output image.

  FIG. 9 is a diagram illustrating an example of an output image of the evaluation result. In the example shown in FIG. 9, the evaluation during the 50-minute lecture for the participant A, the participant B, and the participant C is shown in time series. Referring to FIG. 9, for example, Participant A has a positive reaction behavior for about 15 minutes from the start of the lecture, and an intermediate in which neither positive reaction behavior nor negative reaction behavior is performed from 15 to 35 minutes. It turns out that there was a positive reaction behavior again until the end of the lecture (50 minutes). Participant B has a positive reaction behavior from the start of the lecture to about 20 minutes and becomes an intermediate state in which neither positive reaction behavior nor negative reaction behavior is performed from about 20 minutes to about 30 minutes. It can be seen that the positive reaction action was performed again from about 30 minutes to about 40 minutes, and further that the speech was made from about 40 minutes to the end of the lecture (50 minutes). Participant C has a positive reaction behavior from the start of the lecture to about 10 minutes, and is in an intermediate state where neither positive reaction behavior nor negative reaction behavior is performed from about 10 minutes to about 30 minutes. It can be seen that after a negative reaction behavior was performed until around 40 minutes, the state returned to an intermediate state from around 40 minutes until the end of the lecture (50 minutes).

  As described above, when the evaluation results of each participant are arranged in time series, the state of the entire lecture over time can be inferred based on characteristics common to each participant. In the example shown in FIG. 9, each participant performs a positive reaction behavior immediately after the start of the lecture, but the positive reaction behavior gradually decreases after a certain period of time has elapsed from the start. This is presumed to be caused by, for example, a decrease in the concentration and attention of the participants over time. Then, when more time passes and the end of the lecture is near, each participant will react positively again. This is presumed to be due to the fact that, for example, due to the near end of the lecture, the participants turn their concentration and attention to the lecture.

  FIG. 10 is a diagram illustrating another example of the output image of the evaluation result. In the example illustrated in FIG. 10, regarding the participant A who is an evaluation target, it is shown how positive reaction behavior and negative reaction behavior appear with the passage of time. In the example of FIG. 9, the evaluation result of the participant himself / herself obtained by integrating the evaluation results based on the individual reaction behavior of the participant is shown as time series information, whereas in the example of FIG. Shown as information. The upper part of the display image shown in FIG. 10 shows the strength of the positive reaction behavior, and the lower part shows the strength of the negative reaction behavior. Here, the strength of the positive reaction behavior indicates the degree of evaluation specified by the mode of action that is the reaction behavior. For example, in the case where it is evaluated that a positive degree is high when nodding is repeated many times or when nodding is performed with a large and slow movement, the strength of positive reaction behavior is shown in the upper part of FIG. Plotted on the stronger side (upper) (dotted). On the other hand, when it is evaluated that the positive degree is low when nodding lightly, in the upper part of FIG. 10, the strength of the positive reaction behavior is plotted on the weaker side (downward). Similarly, in the case where it is evaluated that the negative degree is high according to the mode of the action that is the reaction behavior, in the lower part of FIG. In the lower part of FIG. 10, the negative reaction behavior strength is plotted on the lower side (downward).

  In the example shown in FIG. 10, when the upper graph and the lower graph are compared, when the strength of the positive reaction behavior is strong, the strength of the negative reaction behavior is also strong (the graph in the figure is a mountain shape). Where) That is, during this time period, the participant A who is the evaluation target performs an action extracted as a strong positive reaction action and an action extracted as a strong negative reaction action, and is very active. It is assumed that he participated in the lecture positively.

<Other configuration examples>
The non-linguistic information evaluation system 10 according to the present embodiment has been described above, but the specific configuration of the present embodiment is not limited to the above. For example, in the above configuration, the information processing apparatus 200 processes a moving image acquired by the video camera 100, and the terminal apparatus 300 as an output unit displays and outputs the obtained evaluation result. On the other hand, the information processing apparatus 200 may be configured to also serve as an output unit. That is, the information processing apparatus 200 and the terminal apparatus 300 are not divided, and for example, the information processing apparatus 200 itself may be configured to include a display device such as a liquid crystal display and display the evaluation result. In the above embodiment, the image of the person to be evaluated is acquired by shooting with the video camera 100. However, the information processing apparatus 200 may analyze and evaluate separately prepared image data. For example, the image data separately taken and stored in a storage device such as a magnetic disk device may be read and evaluated.

DESCRIPTION OF SYMBOLS 10 ... Non-language information evaluation system, 20 ... Network, 100 ... Video camera, 200 ... Information processing apparatus, 201 ... CPU, 202 ... RAM, 203 ... ROM, 204 ... External storage device, 205 ... Network interface, 210 ... Movie Data acquisition unit, 220 ... region identification unit, 230 ... motion detection unit, 240 ... non-linguistic information extraction unit, 250 ... reaction evaluation unit, 260 ... output unit, 300 ... terminal device, 301 ... CPU, 302 ... RAM, 303 ... ROM 304 display device 305 input device 306 network interface 310 evaluation result acquisition unit 320 display image generation unit 330 display control unit 340 operation accepting unit

Claims (8)

An action detection unit that identifies a part of the human body shown in the video data and detects the action of the specified part;
An action extraction unit for extracting an action defined as an evaluation target in a predetermined evaluation item based on the movement of the part of the human body detected by the movement detection unit;
An evaluation unit that performs evaluation for each evaluation item based on the behavior extracted by the behavior extraction unit and an evaluation criterion that is predetermined for each evaluation item;
An information processing apparatus comprising:   The evaluation unit specifies a degree of evaluation in an evaluation item for which the action is an evaluation target, based on at least one of the type of action defined as the evaluation target, the appearance frequency, and the duration. The information processing apparatus according to claim 1.   With respect to a specific evaluation item, an action corresponding to the first classification and an action corresponding to the second classification are defined as conflicting evaluations, and the evaluation unit corresponds to the first classification with respect to the specific evaluation item The evaluation according to the specific evaluation item is performed based on the evaluation based on the appearance of the action and the evaluation based on the appearance of the action corresponding to the second classification. Information processing device.   4. The evaluation unit according to claim 1, wherein the evaluation unit continuously evaluates the same evaluation item, and generates time-series data indicating a change in the evaluation result over time. The information processing apparatus described. An acquisition means for acquiring video data;
Behavior evaluation means for analyzing the video data acquired by the acquisition means and evaluating the behavior of a person shown in the video;
Output means for outputting an evaluation result by the behavior evaluation means,
The behavior evaluation means includes
An action detection unit for identifying a part of a human body shown in the moving image data and detecting an action of the specified part;
An action extraction unit for extracting an action defined as an evaluation target in a predetermined evaluation item based on the movement of the part of the human body detected by the movement detection unit;
An evaluation unit that performs evaluation for each evaluation item based on the behavior extracted by the behavior extraction unit and an evaluation criterion that is predetermined for each evaluation item;
An evaluation system comprising: With respect to a specific evaluation item, an action corresponding to the first classification and an action corresponding to the second classification are defined as conflicting evaluations, and the evaluation unit of the behavior evaluation means Based on the evaluation based on the appearance of the action corresponding to the first classification and the evaluation based on the appearance of the action corresponding to the second classification, the evaluation on the specific evaluation item is performed,
The output means includes a display unit for displaying an image, and compares the evaluation based on the appearance of the behavior corresponding to the first category with the evaluation based on the appearance of the behavior corresponding to the second category. 6. The evaluation system according to claim 5, wherein display is output on the display unit.   The output unit includes a display unit that displays an image, and the evaluation result continuously performed on the same evaluation item by the evaluation unit of the behavior evaluation unit is a graph showing a change in the evaluation result over time. The evaluation system according to claim 5 or 6, wherein an image to be displayed is displayed and output on the display unit. Computer
A motion detection means for identifying a part of the human body shown in the video data and detecting a motion of the identified part;
Action extracting means for extracting an action defined as an evaluation target in a predetermined evaluation item based on the action of the part of the human body detected by the action detecting means;
Based on the behavior extracted by the behavior extraction means and the evaluation criteria predetermined for each of the evaluation items, functioning as an evaluation means for performing evaluation for each evaluation item;
A program characterized by

Images