JP2006079533A - Information processor, information processing method, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To evaluate internal states such as nervousness of a subject on the basis of information easily obtainable without imposing loads or the like on the subject. <P>SOLUTION: When a person is nervous, his or her face is kept upright, whereas when she/he is relaxed, his or her face tends to have a tilt to the right and left. So, a tilt of the face obtained from a photographed image of the face when the nervousness is known is associated with the nervousness and basic data are created. Afterwards, the nervousness in correspondence to the tilt of the face is estimated by comparison between the tilt of the face detected by the photographed image and the basic data. For example, by a tilt in a vertical direction of a vertical bisector of a line segment connecting right and left edges of the mouth on the face image, a tilt of the face is obtained. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an information processing apparatus, an information processing method, and a computer program for evaluating an internal state of a subject, and in particular, an internal such as a degree of tension of a subject based on information easily obtained without imposing a burden on the subject. The present invention relates to an information processing apparatus, an information processing method, and a computer program for evaluating a state.

  More specifically, the present invention relates to an information processing apparatus and information processing method for managing audio-video information obtained by photographing a subject based on the degree of tension of the subject, and a computer program. In addition, information processing that manages and classifies conference information, such as presentation materials used, based on the degree of tension obtained based on the photographed image of at least one conference participant. The present invention relates to an apparatus, an information processing method, and a computer program.

  Human body identification / detection technology has been developed as a security system such as personal authentication or as a man-machine interface that does not place a burden on the user.

  For identification / authentication of a person, a method using biological information such as a face, a voiceprint, an iris, a retinal capillary, a fingerprint, a finger of a hand, and a back is generally used. However, other than the face and the voiceprint, it is troublesome for the user because an action in which a person is conscious of authentication, for example, an action of holding an eye or the back of the hand over the identification device is necessary. In addition, in the case of a robot aiming at symbiosis with humans, there is a problem in that a user's action conscious of such authentication at home or office is unnatural, and a plurality of people cannot be identified simultaneously. For this reason, it is considered that a method for performing face identification using visual data is promising.

  In the face recognition process, face area extraction (or face detection) for identifying the position of a user's face or recognizing the face direction from an image frame such as a still image or a moving image, and image information of the extracted face area are used. The process is roughly divided into two processes: face identification for identifying or recognizing a person's face. Face recognition methods include algorithms that use the relative positional relationship of each face element, algorithms that combine geometrically invariant features and statistical features, and focus on local features and their composition. (For example, see Non-Patent Document 1). Initially, a recognition technique using a profile was also studied, but at present, the recognition technique of the front image is the center.

  Furthermore, as an expanded technique for face recognition, a method of evaluating facial expressions included in a captured image and using it for image classification and management can be considered. For example, the user's facial expression in the input captured image can be evaluated and scored, and the photographed person can print and enjoy a photographed image with a good facial expression (see, for example, Patent Document 1). .

  If the present inventors can detect not only the superficial state of the subject such as facial expressions but also the internal state such as the psychological state and emotion such as the degree of tension, it is thought that a more fulfilling service can be provided. To do. However, although the facial expression can be evaluated by image analysis of the subject, the actual situation is that the internal state has not been evaluated well. This is because human reactions and emotions always change, but the state does not always appear in words or gestures.

  For example, by detecting physiological information of the human body and calculating the degree of tension of the human body, and by performing drive control according to the calculation result, that is, the degree of tension of the human body, the human motion is read and the machine operation appropriately corresponds to the emotion (For example, refer to Patent Document 2).

  In competitions and sports, a biological signal detector that detects biological signals is attached to athletes and animals, and the obtained biological signals are distributed so that spectators can know the conditions and tension of athletes and animals in real time. Therefore, it is possible to realize a competition viewing system that enhances the amusement of sports and competitions (see, for example, Patent Document 3).

  However, since it is necessary to attach a detector to the subject in order to acquire a biological signal for evaluating the degree of tension, these systems are bothersome to the subject and lack in convenience. As biological signals, mainly the pulse, heart sound, breathing sound, amount of publication, amount of lactic acid in the body, muscle elasticity or relaxation, electroencephalogram, etc., it is necessary to attach a probe to the subject. (A brain wave is usually measured from a minute potential change through an electrode attached to the scalp. As a simple electroencephalogram measuring device, for example, the action potential of the brain is measured from two channels of the left and right frontal regions on the scalp. (For example, see Non-Patent Document 2).

JP 2004-46591 A JP 2002-23918 A JP 2004-192632 A Information Processing Society of Japan “New Edition Information Processing Handbook” (Ohm, November 25, 1995, p. 1193) Hito Sensing Co., Ltd., HSK Central Rhythm Monitor System

  An object of the present invention is to provide an excellent information processing apparatus, information processing method, and computer program capable of suitably evaluating a subject's internal state such as tension, emotion, and psychological state.

  A further object of the present invention is to provide an excellent information processing apparatus and information processing method capable of evaluating an internal state such as a degree of tension of a subject based on information obtained easily without imposing a burden on the subject, and a computer To provide a program.

  The present invention has been made in consideration of the above problems, and a first aspect of the present invention is an information processing apparatus that evaluates an internal state of a subject, and an image input unit that inputs an image of the subject. A face detecting means for detecting the face of the subject from the input image, an index detecting means for detecting a face inclination index from the detected face area, and a face inclination detecting means for determining a change in the face inclination or the face inclination. And evaluation means for evaluating the internal state of the subject based on the face inclination or the change pattern of the face inclination. The evaluation means evaluates, for example, the degree of tension as the internal state of the subject.

  Face recognition technology has been developed as a man-machine interface that does not burden the user. In addition, the facial expression of the subject can be evaluated from the face image and used for classification and organization of the photographed image.

  On the other hand, by detecting not only the surface state of the subject such as facial expressions, but also the internal state such as the psychological state and emotion such as the degree of tension, a service more adapted to the subject can be realized.

  However, conventionally, the internal state such as the tension level of the subject is not acquired from the face image, and the pulse, heart sound, breathing sound, published amount, lactic acid amount in the body, muscle elasticity or relaxation is not performed. The degree of tension is measured on the basis of a biological signal such as an electroencephalogram and an electroencephalogram, and there is a troublesome task of attaching a biological signal detector to the subject.

  For example, participants in the conference are nervous or relaxed according to their age, company history, position, and the like. It is also assumed that the degree of tension changes with time as the conference progresses.

  Here, the inventors often face upright in a scene where humans are assumed to be tense, while the face tilts to the left and right in scenes assumed to be relaxed (left and right) ) Was proactively derived.

  Therefore, in the present invention, the inclination of the face is detected in the photographed image, and the degree of tension is estimated from the inclination. In this case, since it is not necessary to attach a detection signal to the subject in order to acquire the degree of tension, it is not necessary to place a physical and psychological burden on the subject. For example, when using tension as an index for classifying and organizing images, such as indexing of captured moving images, the tension can be estimated using the recorded captured image itself. It becomes a simple system.

  In the tension level estimation process according to the present invention, for example, an image of a face when the tension level is known is captured, and the inclination of the face obtained from the captured image is associated with the tension level to create basic data. And a step of detecting the tilt of the face based on the photographed image and estimating the degree of tension corresponding to the tilt of the face by comparison with the basic data.

  Here, the inclination of the face is obtained by extracting the left and right mouth corners or left and right iris positions in the front face image and drawing a vertical bisector connecting the left and right mouth corners or the left and right irises. It can be determined relatively easily by the inclination of the vertical bisector with respect to the vertical direction.

  In the step of obtaining the inclination of the face from the image of the face when the degree of tension is known and creating the basic data, for example, obtaining a more accurate degree of tension using the biological signal obtained by the biological signal detector, The correct association between the degree of tension and the inclination of the face may be performed. Even in such a case, once the basic data is created, the inclination obtained from the captured image may be compared and checked on the basic data, the biological signal becomes unnecessary, and the tension can be easily estimated. it can.

  For example, it is possible to detect the inclination of the face of the subject in the captured video image and create metadata such as indexing of the video image based on the inclination itself or the degree of tension estimated from the inclination.

  Although there is a general tendency that the face is upright when the scene is tense and the face tilts left and right when the scene is relaxed, how much the face tilts (or shakes) when the scene is relaxed ), There are individual differences depending on age, company history, position, and other individual information. Therefore, this system is more suitable for a method of obtaining the autocorrelation between the inclination of the face and the degree of tension in the same subject rather than obtaining basic data from a plurality of subjects.

  Even if the person is the same person, the correspondence between the inclination of the face and the degree of tension varies significantly between being a listener during a meeting and speaking. Therefore, it is preferable to exclude the data obtained during the conversation.

  For example, in a system that handles conference information in which the content of a conference is recorded by video or audio, captured images of a plurality of conference participants are recorded, but the tension level of the plurality of participants can be estimated from each image. . In this case, the change in the degree of tension of each person may be handled individually, but the change in the degree of tension in the entire meeting can be represented by superimposing each person's tension degree by giving a weight to each person.

  Here, since the tendency of the face to tilt is different for each person, the weight may be normalized for each person, or may be normalized depending on whether the person is speaking. Of course, you may normalize dynamically.

  When the temporal transition of the tension level in the entire conference is obtained in this way, the conference information including video and audio can be indexed based on the tension level of the entire conference. For example, the representative frame can be extracted from the time when the degree of tension in the entire meeting increases. Further, the degree of tension can be classified according to the degree of inclination of the face. For example, positions can be ordered or grouped among participants.

  According to a second aspect of the present invention, there is provided a computer program written in a computer-readable format so that a process for evaluating the internal state of a subject is executed on a computer system. An image input step for inputting an image; a face detection step for detecting the face of the subject from the input image; an index detection step for detecting an index of face inclination from the detected face region; and A computer program comprising: a face inclination detection step for obtaining a change; and an evaluation step for evaluating a subject's internal state based on a face inclination or a face inclination change pattern.

  The computer program according to the second aspect of the present invention defines a computer program described in a computer-readable format so as to realize predetermined processing on a computer system. In other words, by installing the computer program according to the second aspect of the present invention in the computer system, a cooperative action is exhibited on the computer system, and the information processing according to the first aspect of the present invention is performed. The same effect as the apparatus can be obtained.

  Advantageous Effects of Invention According to the present invention, an excellent information processing apparatus, information processing method, and computer program capable of evaluating an internal state such as a degree of tension of a subject based on information easily obtained without imposing a burden on the subject Can be provided.

  Further, according to the present invention, there are provided an excellent information processing apparatus, information processing method, and computer program capable of managing video / audio information of a photographed person based on the degree of tension of the photographed person. be able to.

  In addition, according to the present invention, the conference information that records the content of the conference such as video and audio, as well as used presentation materials is indexed based on the degree of tension obtained from the photographed image of at least one conference participant, and the like. It is possible to provide an excellent information processing apparatus, information processing method, and computer program capable of managing and classifying information.

  Other objects, features, and advantages of the present invention will become apparent from more detailed description based on embodiments of the present invention described later and the accompanying drawings.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  The present invention relates to an information processing apparatus for evaluating an internal state such as a degree of tension of a subject based on information that is easily obtained without imposing a burden on the subject. The person's tension can be estimated.

  In the present invention, the basic principle is that the face is upright in a scene where a human is nervous, but the face is inclined to the left and right in a relaxed scene. The degree of tension is estimated by obtaining the inclination of the face based on the inclination of the perpendicular bisector of the line segment connecting the left and right mouth corners of the face image with respect to the vertical direction.

  According to the present invention, it is possible to perform video indexing and other processing over a long period of time using the degree of tension of the subject estimated based on shooting information such as video. For example, the representative frame can be extracted from the time when the degree of tension becomes high.

A. Device Configuration FIG. 1 schematically shows a hardware configuration of an information processing device used for implementing the present invention.

  The illustrated system is configured around a processor 10. The processor 10 executes various processes based on a program stored in a memory under a program execution environment provided by an operating system (OS). The processor 10 also controls various peripheral devices connected via the external bus interface 14 and the bus 30. The peripheral devices connected to the bus 30 are as follows.

  The memory 20 is composed of a semiconductor memory such as a DRAM (Dynamic RAM), for example, and is used for loading a program code executed in the processor 10 and temporarily storing work data of the execution program. The memory 20 constitutes a memory space of the processor 10.

  The display controller 21 generates a display image according to the drawing command sent from the processor 10 and sends it to the display device 22. The display device 22 connected to the display controller displays the image on the screen according to the display image information sent from the display controller 21.

  The input / output interface 23 is connected to a keyboard 24 and a mouse 25, and transfers input signals from the keyboard 24 and mouse 25 to the processor 10.

  The input / output interface 23 includes a general-purpose interface such as USB (Universal Serial Bus), for example, and various information devices can be connected to the computer system shown in the figure. For example, the camera 29 can be connected via the input / output interface 23 when inputting an image of a user or other subject.

  The network interface 26 is connected to an external network such as a LAN (Local Area Network) or the Internet, and controls data communication via the Internet. That is, the data sent from the processor 10 is transferred to another device on the Internet, and the data sent via the Internet is received and passed to the processor 10. For example, programs and data can be received from outside via a network.

  A hard disk drive (HDD: Hard Disk Drive) controller 27 is connected to a large-capacity external storage device 28 such as an HDD, and controls input / output of data to / from the HDD 28 to which the HDD controller 27 is connected. The HDD 28 stores an operating system (OS) program to be executed by the processor, an application program, a driver program, and data and contents to be referred to or reproduced by the program. In the present embodiment, each program is installed on the HDD 28 in an executable format.

  Although the information processing apparatus can be configured using a general computer system such as a personal computer, many electric circuits other than those shown in FIG. 1 are required. However, since these are well known to those skilled in the art and do not constitute the gist of the present invention, they are omitted in the present specification. Also, it should be noted that only a part of the connections between the hardware blocks in the figure is shown in order to avoid the confusion of the drawings.

On the information processing apparatus, various application processes can be executed by starting a desired application program. In particular, in the present embodiment, it is assumed that an application program having the following functions is installed.
(1) Processing of input image from camera (including moving image, still image, audio video) (2) Accumulation of input image (3) Reproduction of input image (4) From face image of subject to be included in input image Tension estimation (5) Indexing and other image management based on estimated tension

B. Tension level estimation process The present invention provides a tension level estimation method that does not place a burden on a subject, in which the level of tension of a subject is estimated based on imaging information. This process uses as a basic principle the tendency that the face is upright when a person is nervous, but the face is tilted to the left and right when the person is relaxed.

  As a specific method of measuring the tilt of the face based on the captured face image, the tension is obtained by determining the tilt of the face by the tilt of the vertical bisector connecting the left and right mouth corners of the face image with respect to the vertical direction. Estimate the degree. Of course, instead of the left and right mouth corners, a method for obtaining the face inclination using the vertical bisector of the line segment connecting the left and right irises with respect to the vertical direction, and other alternatives for obtaining the face inclination from the face image There are several techniques.

  In the tension level estimation process according to the present invention, for example, an image of a face when the tension level is known is captured, and the inclination of the face obtained from the captured image is associated with the tension level to create basic data. And a step of detecting the tilt of the face based on the photographed image and estimating the degree of tension corresponding to the tilt of the face by comparison with the basic data.

  FIG. 2 shows a processing procedure for estimating the degree of tension from the face image in the form of a flowchart.

  First, a moving image taken by the camera 29 is input (step S1). Then, the face area of the subject is detected from the moving image (step S2). The face area can be extracted as a face area by scanning an image frame and performing matching with a predetermined face template, for example.

  Next, the positions of the left and right mouth corners are extracted from the front face image (step S3). Then, a perpendicular bisector connecting the left and right mouth corners is drawn, and the inclination of the face is obtained from the inclination with respect to the vertical straight line (step S4).

  FIG. 3 shows a state in which the positions of the left and right mouth corners are detected from the face image, and the perpendicular bisector of the line segment connecting these positions is drawn. FIG. 4 shows how the face inclination θ is obtained from the inclination of the perpendicular bisector.

  Alternatively, as an alternative method for obtaining the tilt of the face, the left and right iris positions are extracted, the left and right mouth corners or the perpendicular bisector connecting the left and right irises are drawn, and the tilt with respect to the vertical straight line. From the above, the face inclination θ can be obtained. FIG. 5 shows a state in which the positions of the left and right irises are detected from the face image, and a perpendicular bisector is drawn between the connecting lines. FIG. 6 shows how the face inclination θ is obtained from the inclination of the perpendicular bisector.

  Next, pattern recognition of the temporal transition θ (t) of the face inclination is performed (step S5).

 Humans tend to stand upright in scenes that are assumed to be tense, while faces tend to tilt left and right (sway from side to side) in scenes that are assumed to be relaxed. For example, participants in the conference are nervous or relaxed according to their age, company history, position, and the like. It is also assumed that the degree of tension changes with time as the conference progresses. Therefore, it is possible to detect the inclination of the face in the captured image and estimate the degree of tension from the inclination pattern.

  In the tension estimation process using the face inclination pattern θ (t), an image of a face when the tension is known is photographed, and the face inclination obtained from the photographed image is associated with the tension. Thus, the method includes a step of creating basic data and a step of detecting the tilt of the face based on the photographed image and estimating the degree of tension corresponding to the tilt of the face by comparison with the basic data.

  At the stage of creating the basic data, the inclination of the face is obtained from the image of the face when the degree of tension is known, and the pulse, heart sound, breathing sound, publication amount, lactic acid amount in the body, muscle elasticity or relaxation, EEG Thus, it is possible to obtain a more accurate degree of tension using a biological signal obtained by a general biological signal detector, and to correctly associate the degree of tension and the inclination of the face.

  FIG. 7 exemplifies the correspondence relationship between the temporal transition θ (t) of the face inclination obtained from the face image and the pulse rate. When the sympathetic nerve, which is an autonomic nerve, increases, the degree of tension increases, and when the parasympathetic nerve increases, the degree of tension decreases. The method of detecting the degree of tension from physiological information such as the pulse rate is well known. Therefore, the basic data can be created by identifying the inclination of the face when the degree of tension is increasing or the pattern of the face swinging left and right from the physiological information of the subject.

  Although there is a general tendency that the face is upright in a tense scene and the face is tilted left and right in a relaxed scene, how much the face tilts (or shakes) in a relaxed scene There are individual differences depending on age, company history, position, and other individual information. For this reason, it is suitable for the method of taking the autocorrelation between the inclination of the face and the tension degree in the same subject, rather than obtaining basic data from a plurality of subjects.

  In addition, even if the person is the same person, the correspondence between the inclination of the face and the degree of tension changes significantly between being a listener and speaking during the meeting. Is preferably excluded.

  Once the basic data is created, a step of estimating the degree of tension corresponding to the inclination of the face can be executed by comparison with the basic data. The inclination obtained from the photographed image may be compared and verified on the basic data, the biological signal becomes unnecessary, and the degree of tension of the subject can be easily estimated. For example, based on the pattern of θ (t), the inclination θ of the face of the subject reaches a predetermined angle, the width Δθ that the face swings left or right, or the speed dθ / dt that swings left or right becomes a predetermined value. By comparing with the data, it can be determined whether or not the subject is relaxed.

C. Application to Conference Recording System As described above, the degree of tension of the subject can be estimated based on the inclination of the subject's face obtained from the photographing information.

  On the other hand, there are various media related to the conference such as video information such as images of the participants and audio such as the participants' remarks, as well as whiteboards, slides and other presentation materials used in the conference. Conference recording systems that store data in synchronization with the progress of the conference are becoming common.

  Since conferences are held almost every day in a plurality of conferences every hour, the accumulated conference information is enormous, and the number thereof increases day by day. In order to reuse conference data appropriately, such as looking back on decisions made at conferences, it is essential to quickly search for the scenes you want to see, and create an index for conference information that records audio-video information at conferences. Therefore, it is necessary to efficiently search for desired information.

  According to the present invention, it is possible to estimate the degree of tension of each conference participant from each image using video information as conference information. In this case, the change in the degree of tension of each person may be handled individually, but the change in the degree of tension in the entire meeting can be represented by superimposing each person's tension degree by giving a weight to each person.

  Here, since the tendency of the face to tilt is different for each person, the weight may be normalized for each person, or may be normalized depending on whether the person is speaking. Of course, you may normalize dynamically.

  FIG. 8 schematically shows a configuration of a conference recording apparatus according to an embodiment of the present invention. The illustrated conference recording apparatus includes a conference status recording unit 120, a participant information recording unit 130, and a data processing unit 140.

  The meeting status recording unit 120 records the state of the meeting 100 as audio video information. The participant information recording unit 130 performs processing and recording of information on conference participants. The data processing unit 140 executes processing for creating an index of the conference 100 based on the data recorded by the conference status recording unit 120 and the data recorded by the participant information recording unit 130.

  FIG. 9 illustrates an embodiment to be recorded by the conference status recording unit 120 in the conference 100.

  In the example shown in the figure, the conference 100 is operated by four conference participants A11, B12, C13, and D14, and a document 160 is presented using a PC (Personal Computer) 150. The conference status recording unit 120 records video information including audio information of the conference, and the participant information recording unit 130 records face inclination pattern information for each conference participant.

  FIG. 10 shows in more detail the internal configuration of the participant information recording unit 130 and the data processing unit 140 in the conference recording apparatus.

  The participant information recording unit 130 acquires video information of conference participants, estimates the tension of the conference participants, and records these as participant information for each conference participant. In the illustrated embodiment, the participant information recording unit 130 includes a moving image input unit 131, a face detection unit 132 that detects a face from a captured image, a mouth angle detection unit 133 that detects a mouth angle from a face area, and left and right mouth angles. Is provided with a face inclination detecting unit 134 that detects the inclination of the face based on the inclination of the perpendicular bisector connecting the line segments, and a recording unit 135 that records the face inclination pattern as participant information.

  In the data processing unit 140, the degree of tension of each conference participant is estimated based on the information of the conference participant recorded by the participant information recording unit 130, and further, the conference in the conference information is estimated based on the estimated degree of tension. A time point at which the participant is highly interested is determined, index information is generated using the information of the corresponding place in the audio-video information recorded by the conference information recording unit 120, and this is recorded.

In the participant information recording unit 130, a face inclination pattern θ _i (t), which is an index of the degree of tension for each conference participant i, is measured and recorded. Although the change in each person's tension may be handled individually, the tension determination unit 141 expresses the change in the tension in the entire meeting by adding each person's tension with a weight for each person. .

Here, since the tendency of the face to tilt is different for each person, the weight may be normalized for each person, or may be normalized depending on whether the person is speaking. Of course, you may normalize dynamically. The face inclination pattern θ _i (t), which is an index of the degree of tension of the conference participant i, can be normalized by the following equation.

  Then, the weight of each normalized conference participant is weighted for each person, and the tension is changed (averaged) by superimposing and averaging by the following formula, thereby changing the tension Θ (t) in the entire conference. Represents.

  When the temporal transition of the degree of tension in the entire meeting is obtained in this way, the index information generating unit 142 can index the meeting information including video and audio based on the degree of tension. For example, the representative frame can be extracted from the time when the degree of tension in the entire meeting increases. Further, the degree of tension can be classified according to the degree of inclination of the face. For example, positions can be ordered or grouped among participants.

  Then, the index information recording unit 143 records the generated index information.

In FIG. 11, the tension θ _i (t) for each conference participant i is superimposed to determine the change in tension Θ (t) in the entire conference, and the representative frame is extracted from the video information recording the conference. Is shown.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiment without departing from the gist of the present invention.

  In the present specification, an embodiment has been described in which a vertical bisector connecting a line segment between the left and right mouth corners is obtained in order to measure the inclination of the face, which is an index representing the degree of tension of the subject. Other methods for measuring the tilt of the face based on the above can be used as appropriate.

  Further, in the present specification, an embodiment has been described in which conference information is indexed based on the degree of tension of a plurality of participants in a conference recording system that handles conference information in which the content of a conference is recorded by video or audio. The present invention can also be applied to services using psychological factors such as the user's degree of tension and other applications.

  In short, the present invention has been disclosed in the form of exemplification, and the description of the present specification should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

FIG. 1 is a diagram schematically showing a hardware configuration of an information processing apparatus provided for implementing the present invention. FIG. 2 is a flowchart showing a processing procedure for estimating the degree of tension from the face image. FIG. 3 is a diagram showing a state in which the positions of the left and right mouth corners are detected from the face image, and the perpendicular bisector of the line segment connecting these positions is drawn. FIG. 4 is a diagram showing how the face inclination θ is obtained from the inclination of the perpendicular bisector obtained from FIG. FIG. 5 is a diagram showing a state in which the positions of the left and right irises are detected from the face image, and the perpendicular bisector of the line segment connecting them is drawn. FIG. 6 is a diagram showing how the face inclination θ is obtained from the inclination of the perpendicular bisector obtained from FIG. FIG. 7 is a diagram exemplifying a correspondence relationship between the temporal transition θ (t) of the inclination of the face obtained from the face image and the pulse rate. FIG. 8 is a diagram schematically showing a configuration of a conference recording apparatus according to an embodiment of the present invention. FIG. 9 is a diagram showing an embodiment to be recorded by the conference status recording unit 120 in the conference 100. FIG. 10 is a diagram showing in more detail the internal configuration of the participant information recording unit 130 and the data processing unit 140 in the conference recording apparatus. FIG. 11 shows a state in which the tension θ _i (t) for each conference participant i is superimposed to obtain the tension change Θ (t) in the entire meeting, and the representative frame is extracted from the video information recording the meeting. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Processor 20 ... Memory 21 ... Display controller 22 ... Display apparatus 23 ... Input / output interface 24 ... Keyboard 25 ... Mouse 26 ... Network interface 27 ... HDD controller 28 ... HDD
DESCRIPTION OF SYMBOLS 29 ... Camera 30 ... Bus 100 ... Meeting 120 ... Meeting status recording part 130 ... Participant information recording part 131 ... Moving image input part 132 ... Face detection part 133 ... Mouth angle detection part 134 ... Face inclination detection part 135 ... Recording part 140 Data processing unit 141 Tension determination unit 142 Index information generation unit 143 Index information recording unit

Claims (25)

An information processing apparatus for evaluating the internal state of a subject,
Image input means for inputting an image of the subject,
Face detection means for detecting the face of the subject from the input image;
Index detecting means for detecting an index of face inclination from the detected face area;
A face inclination detecting means for calculating a change in the face inclination or the face inclination;
Evaluation means for evaluating the internal state of the subject based on the face inclination or the change pattern of the face inclination;
An information processing apparatus comprising: The evaluation means evaluates the degree of tension as the internal state of the subject.
The information processing apparatus according to claim 1. The evaluation means evaluates that the subject's tension has increased as the face is upright.
The information processing apparatus according to claim 2. The evaluation means captures an image of a face when the degree of tension is known, and creates basic data by associating the face inclination obtained from the photographed image or a change pattern of the face inclination with the degree of tension. In addition, the inclination of the face is detected based on the photographed image, and the degree of tension corresponding to the inclination of the face is estimated based on the comparison with the basic data.
The information processing apparatus according to claim 2. The index detecting means detects left and right mouth corners from the face image as a face tilt index,
The face inclination detecting means detects the inclination of the perpendicular bisector as a face inclination;
The information processing apparatus according to claim 1. The index detecting means detects left and right irises as facial tilt indices from a face image,
The face inclination detecting means detects the inclination of the perpendicular bisector as a face inclination;
The information processing apparatus according to claim 1. In a conference consisting of a plurality of conference participants, the internal state of each conference participant is evaluated based on the captured images for each conference participant,
It further includes a meeting state evaluation means that represents a change in the internal state of the entire meeting by adding a weight for each person and superimposing the internal state of each person.
The information processing apparatus according to claim 1. The meeting state evaluation means normalizes a change in the inclination of the face for each meeting participant.
The information processing apparatus according to claim 7.   The information processing apparatus according to claim 7, wherein the conference state evaluation unit normalizes a change in inclination of the face depending on whether or not a speech is being made. The meeting state evaluation means dynamically normalizes a change in face inclination;
The information processing apparatus according to claim 7. Conference information input means for inputting conference information in which the status of the conference is recorded by video and audio;
Meeting information processing means for processing meeting information based on a change in the internal state of the entire meeting obtained by the meeting state evaluation means;
The information processing apparatus according to claim 7, further comprising: The meeting information processing means includes at least meeting information indexing based on an internal state of the entire meeting, extraction of a representative frame from a point when the tension of the entire meeting is increased, and classification of the degree of tension according to the degree of inclination of the face. Do one process,
The information processing apparatus according to claim 11. An information processing method for evaluating an internal state of a subject,
An image input step for inputting an image of the subject,
A face detection step for detecting the face of the subject from the input image;
An index detection step of detecting an index of face inclination from the detected face area;
A face inclination detection step for obtaining a face inclination or a change in face inclination;
An evaluation step for evaluating the internal state of the subject based on the face inclination or the change pattern of the face inclination;
An information processing method comprising: In the evaluation step, the degree of tension is evaluated as the internal state of the subject.
The information processing method according to claim 13. In the evaluation step, it is evaluated that the tension of the subject has increased as the face is upright.
The information processing method according to claim 14. In the evaluation step, a face image when the degree of tension is known is photographed, and basic data is created by associating the face inclination obtained from the photographed image or a change pattern of the face inclination with the degree of tension. In addition, the inclination of the face is detected based on the photographed image, and the degree of tension corresponding to the inclination of the face is estimated based on the comparison with the basic data.
The information processing method according to claim 14. In the index detection step, the left and right mouth corners are detected from the face image as an index of the tilt of the face,
In the face inclination detection step, the inclination of the perpendicular bisector is detected as a face inclination;
The information processing method according to claim 13. In the index detection step, the left and right irises are detected from the face image as a tilt index of the face,
In the face inclination detection step, the inclination of the perpendicular bisector is detected as a face inclination;
The information processing method according to claim 13. In a conference consisting of a plurality of conference participants, the internal state of each conference participant is evaluated based on the captured images for each conference participant,
A conference state evaluation step that represents a change in the internal state of the entire conference by superimposing the internal state of each person with a weight for each person is further provided.
The information processing method according to claim 13. In the meeting state evaluation step, the change in face inclination is normalized for each meeting participant.
The information processing method according to claim 19. In the conference state evaluation step, the change in the inclination of the face is normalized depending on whether or not the speech is being made.
The information processing method according to claim 19. In the meeting state evaluation step, a change in face inclination is dynamically normalized.
The information processing method according to claim 19. A conference information input step for inputting conference information in which the status of the conference is recorded by video and audio;
A conference information processing step for processing conference information based on a change in the internal status of the entire conference obtained in the conference status evaluation step;
The information processing method according to claim 19. In the conference information processing step, at least one of indexing of conference information based on the internal state of the entire conference, extraction of a representative frame from the time when the tension of the entire conference has increased, classification of the degree of tension based on the degree of inclination of the face Do one process,
The information processing method according to claim 23. A computer program written in a computer-readable format to execute a process for evaluating the internal state of a subject on a computer system,
An image input step for inputting an image of the subject,
A face detection step for detecting the face of the subject from the input image;
An index detection step of detecting an index of face inclination from the detected face area;
A face inclination detection step for determining a face inclination or a change in face inclination;
An evaluation step for evaluating the internal state of the subject based on the face inclination or the change pattern of the face inclination;
A computer program comprising:

Images