JP2001067482A - Human reaction recognizing device and computer- readable recording medium where program for the same is recorded - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a human reaction recognizing device which can accurately transmit the reactions of many people at a distant place to some speaker to the speaker. SOLUTION: The human reaction recognizing device 34 which recognizes the reaction of a person to displayed video includes a video acquisition part 100 which acquires a video sequence of face images of the person, a detection and separation subsystem 102 which classifies the acquired video sequence into a stable stationary unit and an active detection unit according to movement between video frames, and a decision subsystem 104 which recognizes the reaction of the person from a frame sequence classified as the stable stationary unit and a frame sequence classified as the active detection unit.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a communication system used for conferences, lectures, classes, etc., which are conducted by connecting a number of points by communication. The present invention relates to a system for investigating and assessing audience response so that the subject can be properly understood.

[0002]

BACKGROUND OF THE INVENTION Broadcasting is a very important means of transmitting certain information to a large audience. A typical example is an education system using broadcasting. In an educational system using broadcasting, a lecture given by a lecturer is broadcast, and the broadcasting is received at attending venues provided in various places and displayed using, for example, a television. A student can attend a lecture by watching this broadcast at the attendance site. It should be noted that the number of participants in each place is not necessarily plural, and may be one.

[0003] On the other hand, with the recent development of communication technology, particularly Internet technology, a system for giving a lecture using a method similar to such broadcasting is emerging. This is particularly effective in learning a foreign language, for example, because bidirectional communication between the instructor and the student is possible. As an existing system, there is a system disclosed in Japanese Patent Application Laid-Open No. H11-55643.

The system disclosed in Japanese Patent Application Laid-Open No. H11-55643 is intended to protect the privacy of a user on the premise that video (and audio) is bidirectionally communicated between a lecturer and a student. The video is transmitted only when the user has given permission.

[0005]

In a general lecture not using communication, a lecturer stands on a podium and an audience often sits in front of the podium. The lecturer gives a lecture from the podium. In this case, the lecturer can see the response of the audience, and can change the content of the lecture or change the order of the lecture on the way according to the result. For example, if the audience is bored, the instructor switches to topics that are not directly related to the lecture but can be of interest to the audience, and when the audience concentrates on the lecture again. The content of the discussion can be returned to the main subject.

[0006] The same is possible with a learning system using communication. For example, if there are about three students per instructor, an image of each student can be displayed in front of the instructor. The instructor can grasp the reaction of these students as in the case of a general lecture, and can appropriately adjust the content of the lecture according to the student's response.

[0007] However, it is difficult to do the same for a large audience using communications. This is because, when transmitting images via the Internet or other means, if the number of venues is large, the amount of communication will be enormous, and it will be virtually impossible to send video from each venue to the instructor. is there. Also, even if the video could be sent from each venue to the instructor, it is expected that the number of audiences will be much larger than in the case of general lectures or lectures,
In that case, it is extremely difficult to accurately communicate the response of the entire audience to the instructor.

[0008] These problems are not limited to the transmission of so-called one-way information such as lectures, lectures, speeches, etc., but also exist, for example, when discussions are conducted among a relatively large number of persons gathered at a plurality of places. I do. In such discussions, so-called virtual space technology can be used, but the technology mainly used in virtual space is the technology of how to convey the facial expressions of each attendee to other attendees. Can not be solved.
In the discussion, not only the speaker but also the person corresponding to the audience can be the speaker, so it is convenient if the responses of the attendees at other venues can be accurately grasped.

[0009] Furthermore, not only in the case of the discussion, but also in the case of lectures, lectures, speeches, etc., it may be useful for the audience to know the reaction of the audience at other venues.

SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an apparatus for recognizing a reaction of a person, which can accurately transmit the reaction of a number of persons in a remote place to a speaker to the speaker.

Another object of the present invention is to provide an apparatus for recognizing a reaction of a person, which can accurately transmit the reactions of a large number of persons at remote locations to each other.

[0012]

According to a first aspect of the present invention, there is provided a person reaction recognition apparatus for recognizing a person's reaction to a displayed image, comprising: Video acquisition means for acquiring a video sequence, first classification means for classifying the acquired video sequence into a stable still unit and an activity detection unit based on motion between video frames, and a stable still unit And a reaction recognizing means for recognizing a person's reaction from the frame sequence classified into the activity detection unit.

According to the present invention, the reaction of a person can be recognized based on the stable stationary unit and the activity detecting unit. By sending the information indicating the reaction to the source of the video and collecting statistics, the source of the video can grasp the reactions of many audiences existing at the communication destination.

According to a second aspect of the present invention, in addition to the configuration of the first aspect, the first classifying means may further include a first classifying unit configured to determine a difference between frames between adjacent frames. A second classification means for classifying the video sequence into stationary units and motion units, and a third classification for classifying still frames into stable stationary units and unstable still units based on their duration. Means,
And integrating means for integrating the continuous unstable stationary unit and the moving unit as an activity detecting unit.

According to a third aspect of the present invention, in addition to the configuration of the first aspect of the present invention, the reaction recognizing means is characterized in that the reaction recognizing means converts the feature of the face image of the person from a frame in the stable stationary unit. A first feature vector extracting means for extracting a vector, and a pre-trained first neural network which outputs, as an input, a feature vector output from the first feature vector extracting means, and posture information corresponding to the feature vector. A network, second feature vector extracting means for extracting a feature vector corresponding to the motion of the face image of the person from the information of the inter-frame difference in the activity detecting unit, and features output by the second feature vector extracting means A pre-trained second neural network that outputs gesture information corresponding to a feature vector with a vector as input, Including.

According to a fourth aspect of the present invention, in addition to the configuration of the third aspect of the present invention, the human reaction recognition apparatus further includes at least one of a first neural network and a second neural network. Are associated with a predetermined reaction category, and receive a feature vector output from the first or second feature vector extraction unit as an input and output a high degree of association with the predetermined reaction category. Includes one-to-one neural networks.

Since a one-to-one neural network is used, when a new reaction category is added as a recognition target, a one-to-one neural network corresponding to the reaction category may be added, and the function can be easily expanded.

According to a fifth aspect of the present invention, in addition to the configuration of the first aspect of the present invention, the apparatus for recognizing a person's reaction further comprises a video camera,
A video signal conversion unit for converting a video signal output from the video camera into a digital signal for each frame, and a video signal conversion unit for specifying a face area of a person in each frame based on a video sequence output by the video signal conversion unit. And a face area specifying unit.

According to a sixth aspect of the present invention, in addition to the configuration of the first aspect of the present invention, the face region specifying means includes a video signal converting means. A first method for specifying a face area of a person in each frame by a first method based on a video sequence to be output.
Means, a second means for specifying a face area of a person in each frame by a second method based on a video sequence output from the video signal converting means, and first and second means. And a face area integrating means for integrating the result of specifying the face area according to the above.

Since the face image area is determined by using a plurality of techniques, the face image area can be reliably determined.

According to a seventh aspect of the present invention, in addition to the configuration of the sixth aspect, the video sequence is an RGB color video sequence, and the first means is an RGB color video sequence. Means for specifying a face area based on the similarity between a color distribution in an image obtained by converting the sequence into the rg color space and a predetermined color distribution pattern is included.

According to an eighth aspect of the present invention, in addition to the configuration of the sixth or seventh aspect, the video sequence is an RGB color video sequence. Includes means for specifying a face area based on the similarity between a color distribution in an image obtained by converting an RGB color image sequence into an NCb-NCr color space and a predetermined color distribution pattern.

A computer-readable recording medium according to a ninth aspect of the present invention is a computer-readable recording medium storing a program for operating a computer as a reaction recognition device for recognizing a person's reaction to a displayed image. A possible recording medium, this program
A first classification program portion for classifying a video sequence obtained for a face image of a person into a stable stationary unit and an activity detecting unit based on motion between video frames; A reaction recognition program part for recognizing a human reaction from the frame sequence and the frame sequence classified into the activity detection unit.

According to a tenth aspect of the present invention, in the computer readable recording medium according to the ninth aspect of the present invention, in addition to the configuration of the ninth aspect, the first classification program portion is configured to calculate a difference between frames between adjacent frames. A second unit for classifying the video sequence into stationary units and moving units based on
And a third classifier for classifying the still frames into stable and unstable stationary units based on their duration, and a series of unstable stationary and moving units. An integrated program part for integration as a detection unit.

According to the eleventh aspect of the present invention, in the computer-readable recording medium according to the tenth aspect of the present invention, in addition to the configuration of the tenth aspect, the reaction recognition program portion includes a face image of a person from a frame in the stable stationary unit. A first feature vector extraction program part for extracting the feature vector of the first and a feature vector output from the first feature vector extraction program part are input, and posture information corresponding to the feature vector is output. A first neural network program portion, a second feature vector extraction program portion for extracting a feature vector corresponding to a motion of a face image of a person from information of an inter-frame difference in the activity detection unit, and a second feature Using the feature vector output from the vector extraction program as an input, And outputs the gesture information, and a previously learned second neural network program portion.

According to a twelfth aspect of the present invention, a computer readable recording medium according to the eleventh aspect further comprises a first neural network program part and a second neural network program part. At least one of them is associated with a predetermined reaction category, and receives a feature vector output from the first or second feature vector extraction program portion as an input and outputs a high degree of association with the predetermined reaction category. ,
It includes a plurality of one-to-one neural network program portions.

Since a one-to-one neural network program portion is used, when a new reaction category is added as a recognition target, a one-to-one neural network program portion corresponding to the reaction category may be added, and the function is easily expanded. it can.

According to a thirteenth aspect of the present invention, in a computer readable recording medium according to the ninth aspect of the present invention, in addition to the configuration of the ninth aspect, the program further comprises: A face area specifying program portion for specifying a person's face region in each frame and providing the same to the first classification program portion is included.

According to a fourteenth aspect of the present invention, in addition to the configuration of the ninth to thirteenth aspect, the computer readable recording medium according to the ninth aspect further comprises: Based on the first method for identifying the face area of the person in each frame by the first method.
Based on the program part and the video sequence,
A second program part for specifying the face area of a person in each frame by the method of the first embodiment, and a face for specifying the face area by integrating the result of specifying the face area by the first program part and the second program part And the integrated program part of the domain.

Since the face image area is determined by using a plurality of methods, the face image area can be reliably determined.

According to a fifteenth aspect of the present invention, in addition to the configuration of the fourteenth aspect, the video sequence is an RGB color video sequence, and the first program portion is an RGB color video sequence. A program portion for specifying a face area based on a similarity between a color distribution in a video obtained by converting a color video sequence into an rg color space and a predetermined color distribution pattern is included.

A computer-readable recording medium according to the invention of claim 16 is the computer-readable recording medium of claim 14 or 1.
In addition to the configuration of the invention described in Item 5, the video sequence is RG
The first program portion is a B color video sequence, and specifies a face region based on a similarity between a color distribution in a video obtained by converting an RGB color video sequence into an NCb-NCr color space and a predetermined color distribution pattern. Including the program part.

[0033]

DESCRIPTION OF THE PREFERRED EMBODIMENTS [Equations used in the description] In the following description of the embodiments, the following equations will be used, and reference will be made as needed using the equation numbers of the respective equations.

[0034]

(Equation 1)

[0035]

(Equation 2)

[0036]

(Equation 3)

[0037]

(Equation 4)

[First Embodiment] [Entire System Configuration] Hereinafter, a remote broadcasting system according to a first embodiment of the present invention will be described. The system described below is a system in which a video of a lecturer who gives a lecture at one place is broadcast at a plurality of lecture halls, and the response of the audience at each lecture hall is presented to the lecturer. In the following description, audio is not described for simplicity, but audio is indispensable in lectures, lectures, discussions, etc. It is clear that the configuration exists. Also, the configuration for realizing them will be apparent to those skilled in the art.

Referring to FIG. 1, this remote broadcasting system 2
0 indicates a plurality of lecture hall systems 34 for generating data indicating the reactions, which are provided at a plurality of remote points and summarize and abstract the response of the audience at each point, and the state of the lecture conducted by the instructor. A lecturer system 32 for broadcasting to the plurality of lecture halls via the Internet, a plurality of lecture hall systems 34 and a lecturer system 3
And a tallying center 30 for synthesizing the response of the audience in front of each venue system 34 and transmitting it to the instructor system 32. The instructor system 32
The broadcast of the lecture from the to the venue system 34 is not limited to the Internet, but may be a broadcast using a normal radio wave such as a satellite broadcast or a terrestrial wave. [Hardware Configuration] This remote broadcasting system is realized by software executed on a computer such as a personal computer or a workstation.
FIG. 2 shows an external view of a computer used to realize the remote broadcasting system. The counting center 30, the instructor system 32, and the attendance hall system 34 have almost the same hardware configuration, and are the same as those shown in FIG. Hereinafter, the counting center 30, the instructor system 32, and the attendance hall system 34 are represented as a system 40 as a representative, and the system 40 will be described.

Referring to FIG. 2, a computer main body 60 constituting the system 40 includes a CD-ROM (Compact Diode).
sc Read-Only Memory) drive 70 and FD (Flex)
ibleDisk) Computer body 6 with drive 72
0, a display 62 as a display device connected to the computer main body 60, and a computer main body 60
, A keyboard 66 and a mouse 68 as input devices, a video camera 50 connected to the computer main body 60 for capturing an image of a person (audience), and an instructor transmitted from the instructor system 32. A large-screen display device 86 for displaying video. In the apparatus according to the present embodiment, a video camera including a CCD (solid-state imaging device) is used as the video camera 50, and the video camera 50 is placed in front of the audience and the audience responds to a lecture displayed on the large-screen display device 86. Is extracted. In addition, for the sake of simplicity, the following description will be made assuming that the number of audiences to be photographed is one.

FIG. 3 shows the configuration of the system 40 in block diagram form. As shown in FIG.
In addition to the CD-ROM drive 70 and the FD drive 72, the computer main body 60 constituting the CPU 0 includes a CPU (Central Processing Unit) connected to a bus 92, respectively.
t) 76, ROM (Read Only Memory) 78, RAM
(Random Access Memory) 80 and hard disk 74
And digital data (compressed and compressed) obtained through the bus 92 and transmitted from the instructor system 32 shown in FIG. And a video output circuit 90 for decompressing and converting it into a video signal.
A CD-ROM 82 is mounted on the CD-ROM drive 70. An FD 84 is mounted on the FD drive 72.

As described above, the main part of the remote broadcasting system is realized by computer hardware and software executed by the CPU 76. Generally, such software is a CD-ROM 82, FD8
4 and stored in a storage medium such as the CD-ROM drive 70 or the FD drive 72 and read from the storage medium and temporarily stored in the hard disk 74. Alternatively, if the device is connected to a network, the server on the network temporarily
4 once. Then, the data is further read from the hard disk 74 to the computer
Is executed by Note that, when connected to a network, the RA
The program may be directly loaded into the M80 and executed.

The hardware itself and the operation principle of the computer shown in FIGS. 2 and 3 are general. Therefore, the most essential part of the present invention is the CD-R
The software is stored in a storage medium such as the OM 82, the FD 84, and the computer main body 60.

As a recent general tendency, there is a method in which various program modules are prepared as a part of the operating system of a computer, and the application program calls these modules in a predetermined arrangement when necessary and proceeds with the processing. General. In such a case, the software itself for realizing the remote broadcasting system does not include such a module, and the functions of the components of the remote broadcasting system are realized only when the computer cooperates with the operating system. However, as long as a general platform is used, it is not necessary to distribute software including such modules, and the software itself that does not include those modules and a recording medium on which the software is recorded (and the software is distributed on a network) Data signal in such a case) can be considered to constitute an embodiment. [Student venue system 34] With reference to FIG.
A / D (analog / digital) conversion of a signal received from the video camera 50 for each frame, and a video acquisition unit 100 for storing the video signal. A detection / separation subsystem 102 for separating a frame into a “stable stationary unit” and an “activity detection unit” to be described later, and respective frames belonging to the stable stationary unit and the activity detection unit separated by the detection / separation subsystem 102 A judgment subsystem 104 for judging an audience response at the venue based on the image of the venue, classifying the reactions into a plurality of categories and outputting category information, and each category output from the judgment subsystem 104. Information for a given period of time, and the totaled result is a score indicating the response of the audience at this venue to the lecture. , Which transmits the output of the counting unit 106 to the counting center 30 via the Internet, and starts receiving video data of the lecture from the instructor system 32. Transmission / reception unit 1 for performing communication control in the system
08, an image processing unit 110 for performing necessary processing such as data decompression and data interpolation on the video data of the lecture received by the transmission / reception unit 108, converting the video data into a video signal, and outputting the video signal. The large-screen display device 86 for displaying the video signal from the unit 110 is included.

The configuration of the main functional blocks of the attendance hall system 34 will be described in detail below. [Detection / Separation Subsystem 102] Referring to FIG. 5, detection / separation subsystem 102 is used to detect a face area of a person who is an audience from the image of each frame provided from video acquisition section 100. For the face area detection unit 120 and the face area detected by the face area detection unit 120, paying attention to the movement of the image included in the input video, the video is referred to as a “stationary unit” and a “motion unit” to be described later. Video stream unit separation unit 122 for classifying
Focusing on the duration of a unit classified as a stationary unit by the video stream unit separation unit 122, a stationary unit that continues for a certain threshold or more is set as a “stable stationary unit”, The static unit classification unit 124 for classifying the unit into the “unstable stationary unit”, the unit classified as the motion unit by the video stream unit separation unit 122, and the unstable stationary unit by the stationary unit classification unit 124 And a unit integrating unit 126 for integrating the obtained unit with the above-mentioned “activity detecting unit”.

The stationary unit, the motion unit, the stable stationary unit, the unstable stationary unit and the activity detecting unit will be described in more detail later with reference to FIGS. [Face area detection unit 120] Many techniques have been proposed as techniques for detecting a face area of a person from an image including a complicated background. These are roughly classified into a first method for detecting a face region based on facial features represented by facial parts such as eyes, a nose, and a mouth, and a method based on a difference between a person's face color and a background color. And the second method of detecting a face area.
Either method may be used as long as it can detect a face area. In this embodiment, the second method is used as described below.

Also, in the system of the present embodiment, in order to reliably detect a face area, in particular, a method using an rgb color space and a method using an NCb-NCr color space are combined.
The face area is detected by integrating the results of both methods. Both of these approaches take advantage of the fact that the color distribution of a human face color can be modeled with a 2D Gaussian distribution, regardless of the color of the human face and the color of the illumination. When calculating the color distribution, one uses r, g, and b, and the other uses NCb-N
These methods differ only in that Cr is used.

Referring to FIG. 6, face area detecting section 120 includes:
An rgb color space for performing a color space conversion to an (r, g, b) component by performing the conversion represented by equations (1) to (3) on the input video stream 130 composed of RGB components Conversion process 132 and RGB color space conversion process 13
2 to perform similarity mapping between the color distribution of the input image and the color distribution prepared in advance using the r and g components of the color components converted to the (r, g, b) components by 2. Rg space similarity calculation processing 134.
Note that the sum of r, g, and b is always "1" as can be seen from equations (1) to (3). That is, any one of r, g, and b is represented by the other two components. Therefore, the color distribution can be expressed using only r and g.

Here, the color distribution of the human face color is 2D.
It can be expressed as Gaussian distribution _{G 1 (m 1, V 1} 2).
Here, each variable is a value represented by Expressions (4) to (7). The overlined variables in the equation represent the Gaussian means of each variable. V ₁ represents a covariance matrix of a 2D Gaussian distribution. N represents the total number of pixels in the face area.

The face area detecting section 120 further outputs rg
b color space conversion processing 132 and rg space similarity calculation processing 1
And (NCb, NCr) color space conversion processing 136 and NCb-NCr space similarity calculation processing 138 for performing the same processing as in the NCb-NCr color space. The values of the Y, Cb, and Cr color components for calculating NCb and NCr are given by equations (8) to (1).
0). Further, these are expressed by formula (11).
By normalizing as shown in (12), NCb and NCr color components are obtained.

The color distribution expressed using the NCb and NCr color components can be expressed as a 2D Gaussian distribution G2 (m ₂ , V ₂ ² ). Here, each variable is expressed by Expression (13)
Represents the value represented by (16). Here, the meaning of the overline is the same as the meaning in the expressions (4) to (7). V ₂ is a covariance matrix of a 2D Gaussian distribution.

The face area detection unit 120 further performs the similarity mapping calculated by the rg space similarity calculation processing 134 and the NC
b-NCr spatial similarity calculation processing 138: a similarity calculation integration processing 140 for integrating the similarity mapping calculated by the processing 138 to generate a new similarity mapping; A post-process 142 for expressing a portion corresponding to the face region as an ellipse or a rectangular region and outputting information indicating its position in accordance with the final similarity mapping obtained by the calculation integration process 140 .

Rg space similarity calculation processing 134 and NCb-N
Equation (17) was used as the similarity mapping in the Cr space similarity calculation processing 138. In equation (17), k
= 1, r, g space, k = 2, NCr-NC
This corresponds to the case of the b space. In equation (17), I (x, y) represents the intensity of each color component at point (x, y), m represents the average vector representation of the average of the color component values, and V represents the covariance matrix. . Equation (17) (k = 1, 2) is calculated for each pixel, and a certain threshold value is provided for the result. A pixel having a value smaller than the threshold value is set as a face area candidate.

The integration in the integration processing 140 of the similarity calculation is performed as follows. That is, the similarity mapping expression d obtained in the rg space similarity calculation processing 134
Equation (18) is calculated for _k (i, j) (k = 1, 2). Based on the result, it is determined whether each pixel is a candidate belonging to the face area according to Expression (19). Equation (1
8) In (19), μ _k and σ _k ² are respectively d
The mean and variance at _k (i, j) (k = 1,2). W ₁ and w ₂ are predetermined weights, respectively, and D 1
Is a threshold.

The post-processing 142 includes two processes. As a result of the integration processing 140 of the similarity calculation, for each pixel of the video,
Each pixel is assigned a value of 1 or 0 depending on whether the pixel is considered to belong to the face area. Therefore, an elliptical or rectangular area representing a face area is obtained by the following processing. In the following description, it is assumed that a face area is represented as a circle for simplification.

First, noise removal is performed. Here, the candidate portion of the face area is enlarged and its surrounding contour is obtained. Next, the circle area is applied to the face area by the following algorithm for the face area candidate part thus obtained.

First, the center (Cx, Cy) of the area considered to be flesh color is obtained by the equations (20) and (21). Where J
(x, y) is a binary image in which 1 is assigned to the face area candidate pixel and 0 is assigned to other pixels by the above processing. Next, the radius R of the region considered to be skin color is calculated by the equation (22). The center thus obtained is (Cx, C
y) A circle having a radius R is defined as a face area. Expression (20)-
(22) is merely an example, and any of these formulas can be selected according to the application and the design concept.

FIG. 7 shows the processing steps of the face area detecting section 120 and the resulting image. FIG. 7A shows an original image. The image obtained as a result of the rg space similarity calculation processing 134 is shown in FIG. 7B in the NCb-NCr space similarity calculation processing 138.
The images obtained as a result are shown in FIG. FIG. 7D shows a result of the integration process 140 of the similarity calculation and a circle (ellipse) indicating the face region obtained by the process performed by the post-process 142 on the result. FIG. 7E combines the circle (ellipse) thus obtained with the original image of FIG. 7A.

Once the face area has been determined in this way, tracking of this face area may be performed in the following. When the tracking of the face area fails, the face area can be determined by performing the above-described processing again.

Next, the video stream unit separation unit 12
2. The contents of the processing of the stationary unit classification unit 124 and the unit integration unit 126 will be described with reference to FIGS. First, the video stream unit separation unit 122
Is obtained by taking the difference between the input frames.
Are obtained. When the value of the difference between frames is obtained, the units are separated as follows.

[0061] In all the frames between frames N ₁ ~ frame _{N 2 (N 1 <N 2} ), larger than the threshold value (typically 0) where that inter-frame difference, frame N ₁
If the inter-frame difference is less than or equal to this threshold value in −1 and frame N ₂ +1, frames N _{1 to} N ₂ are called motion units. Frame N ₁ to frame N ₂ (N ₁ <
N ₂ ), if the inter-frame difference is less than or equal to this threshold in frames N ₁ -1 and N ₂ +1, then the frame N the ₁ to N ₂ is referred to as a stationary unit. Although a small value should be selected as the threshold value, it is not limited to 0.

The video stream unit separation unit 122
The inter-frame differences 150 shown in FIG. 8 are converted into the stationary units 160, 164, 168, 17 according to the above-described definition.
2 and 176, and motion units 162, 16
6, 170, 174 and 178.

FIG. 10 shows the actual procedure of unit separation. Referring to FIG. 10, first, feature point extraction and tracking of images of two consecutive frames in the image stream are performed (step 220). A known algorithm can be used for feature point extraction and tracking at this time. For example, KLT (Kanade-Lucas-Tomas
i) An algorithm can be used.

Next, an inter-frame difference is calculated using the correspondence between the feature points extracted in step 220 (222). Unit separation is performed based on the inter-frame difference thus obtained (224).

Next, the stationary unit classifying section 124 classifies the stationary units 160, 164, 168, 172 and 176 into stable stationary units and unstable stationary units as follows. That is, the stationary unit classification unit 124
Classifies a stationary unit if the number of frames constituting the stationary unit is equal to or greater than a certain threshold value, and otherwise classifies the unit as an unstable stationary unit. For example, the inter-frame difference 15 shown in FIG.
0, the static unit 160 and the static unit 1
68 are respectively classified into stable stationary units. Stationary units 164, 172 and 176 are classified as unstable stationary units.

Here, the stable stationary unit is considered to be a period in which a person does not move in a certain pose since it is considered that the person has not moved for a certain length of time or more. This could mean, for example, that a person is concentrating on the lecture, or that he is bored with the lecture and looks elsewhere. On the other hand, the unstable stationary unit is considered to correspond to a temporary stationary state included in some movement because it is only a very short period sandwiched between the moving units.

The stationary unit classifying unit 124 determines the information relating to the stable stationary unit to the determination subsystem 104, and transmits the information relating to the unstable stationary unit to the unit integrating unit 126.
To give each.

The unit integrating unit 126 integrates the information on the motion unit given from the video stream unit separating unit 122 and the information on the unstable still unit given from the still unit classifying unit 124 to make an activity detecting unit. In other words, a frame in which some movement is detected by a person and a very short period of still frame sandwiched between those moving frames can cause nodding, shaking, turning up and down, tilting the head, falling asleep, etc. It is considered that it is possible to detect a response that the audience shows to the lecture, for example, so that it is classified as an activity detection unit. However, these activities do not necessarily indicate that they are focused on the lecture, but may also correspond to movements unrelated to the lecture.

Referring again to FIG. 4, the information on the stable stationary unit and the information on the activity detecting unit thus classified are given to the judgment subsystem 104, and the judgment subsystem 104 makes the judgment and the information of each video frame. The response of the audience is recognized and determined based on the image information.

Referring to FIG. 11, judgment subsystem 10
Reference numeral 4 includes a posture estimating unit 230 for estimating the posture of the audience by receiving the stationary unit information, and a gesture recognizing unit 232 for recognizing the gesture of the audience by receiving the activity detecting unit information. The posture estimating unit 230 outputs estimated posture information, and the gesture recognizing unit 232 outputs estimated gesture information.

Referring to FIG. 12, posture estimating section 230 includes:
Feature vector extraction processing unit 242 for extracting an image feature vector from stable still unit frame image 240
And learning in advance to output information (posture category name) indicating the posture of the audience corresponding to the stable still unit frame image 240 using the feature vector of the frame image from the feature vector extraction processing unit 242 as an input. And a posture determination unit 244 using a neural network. In the system of this embodiment, the posture determination unit 244 based on the neural network performs a plurality of one-to-one neural networks that have been learned in advance so that one neural network corresponds to one posture category to be detected. The output of these one-to-one neural nets 250 is examined for the same input feature vector as that of the neural network 250, and the one-to-one neural net 25 showing the highest output value is examined.
And a maximum value detection unit 252 for outputting a posture category name corresponding to 0.

As described above, if one one-to-one neural net 250 is provided for one posture category, it is possible to easily cope with an attempt to add a recognition function for a new posture category. is there. That is, in this case, the one-to-one neural network 250, which has been learned in advance to detect the new posture category, is replaced with the neural network-based posture determination unit 2.
44, and connect its output to the input to the maximum value detection unit 252. If the whole posture determination unit 244 using a neural network is a large neural network, it is necessary to relearn the entire neural network in order to add a function for detecting a new posture category. Therefore, in an actual application, it is practical to provide a plurality of one-to-one neural nets 250 like the attitude determination unit 244 using a neural network.

Referring to FIG. 13, gesture recognition unit 2
32 has a similar configuration. That is, the gesture recognizing unit 232 generates the activity detection unit frame image information 260.
A feature vector extraction process 262 for extracting a feature vector of an image from an image, and a gesture determining unit using a neural network for classifying audience gestures into categories using the feature vectors as input and outputting the information (gesture category name) 264.

The gesture determination unit 264 based on the neural network also includes a plurality of one-to-one neural nets 270 provided to correspond to one gesture category, and a plurality of one-to-one neural nets 2.
And a maximum value detection unit 272 for outputting a gesture category name corresponding to the one-to-one neural network 270 having the largest output.

In this embodiment, the feature vector extraction processing unit 242 extracts a feature vector from the image of the head region of the stable still image using the wavelet transform. For example, a low-pass filter H _i (Z) (i = 0, i) having a relationship represented by Expressions (23) to (25) with respect to an input image.
1) and high-pass filter _{G i (Z) (i =} 0,1)
Prepare The result of applying these in combination in the order of i = 0,1 to the original input image P, P ₁ by H ₀ H ₁ is, I ₁ by H ₀ G ₁ is, I by G ₀ H _1
It is assumed that I ₃ is obtained by G ₀ G ₁ and I ₂ respectively. As a result of further relative P ₁ similarly by applying the above-mentioned filter _{P 2, I 1 (P 1} ), I 2 (P 1), I
₃ (P ₃ ) is obtained, and similarly, P _N , I ₁ (P _N-1 ), I ₂
(P _N-1 ) and I ₃ (P _{N- 1} ) are obtained. All the values thus obtained are compared with a predetermined threshold value, and each value is set to either 0 or 1. Then, these 0 or 1 values are arranged in a predetermined order to form a feature vector.

On the other hand, the feature vector extraction by the feature vector extraction processing 262 is different from the case of the feature vector extraction processing unit 242. In this case, since the movement of the audience's head becomes a problem, the feature vector is extracted based on the difference between two consecutive frames. In this embodiment, angles θ _t , φ _t, and ρ _t representing the motion of the head of a person between two frames (frames t-1 and t) are obtained using a known method. theta _t is a rotation angle around the optical axis, phi _t is the x-axis of the coordinate axes fixed to the subject, the angle between the rotating axis Φ parallel to the image plane, [rho _t is the angle about the Φ axis, Shown respectively. This value is obtained for N consecutive frames.

Further, other information is added to make the process more robust. In the present embodiment, such information includes the coordinates of the center of the head movement, the direction of the head movement, the energy of the head movement in the x-axis direction and the y-axis direction, and the like. In the present embodiment, these values are extracted from predetermined N consecutive frames, and are combined with the above-described angle information of the N frames to form a feature vector.

Although the above N has been discussed as being a fixed numerical value, for example, the number of frames included in the activity detection unit may be larger or smaller than N. This is dealt with as follows. First, a desirable value of N is determined. Next, N is compared with the number of frames in the activity detection unit. If the number of frames is larger than N, a feature vector is extracted using only the first N frames. If the number of frames is less than N, the feature vector is extracted on the assumption that the missing portion is the same as the last frame. Even if such a correction is made, it is possible to make an accurate determination by appropriately learning the one-to-one neural network 270.

Next, the configuration of the counting unit 106 shown in FIG. 4 will be described. Since the counting unit 106 is substantially realized by software, a configuration of processing performed by the software will be described below with reference to FIG. In this processing, first, a necessary storage area is secured and initialized (290). Next, it waits for a certain sampling time to elapse (292). This process may be a process of waiting for an event that occurs when there is an output from the determination subsystem 104. When the sampling time elapses (or when there is an output from the determination subsystem 104), the posture and the gesture based on the posture information (posture category name) and the gesture information (gesture category name) output from the determination subsystem 104 at that time. Evaluate the category score. This evaluation is performed as follows in the present embodiment. Note that the posture category names output by the maximum value detection unit 252 include “Frontal-view (front)”, “Right (right)”, “Left (left)”, “Up (up)”, and “D”.
There are five types of “own (lower)”, and the gesture category names output by the maximum value detection unit 272 are “nod (nod)”, “shake head (shake your head)”, “Look right (look right)” “Look” It is assumed that there are six types: left (look at the left), "Look down (look at the bottom)", and "Look Up (look at the top)".

First, the following evaluation is performed on the activity detection unit. (1) If the gesture category of this unit is “nod” or “shakehead”, the score of this unit is set to 1.

(2) The posture category name of the immediately preceding stable stationary unit is “Left”, the gesture category of this activity detecting unit is “Look right”, and the posture category name of the stable stationary unit immediately after this unit Is “Frotal-view”, the activity detection unit score is set to “1”.

(3) The posture category name of the immediately preceding stable stationary unit is “Right”, the gesture category of this activity detecting unit is “Look left”, and the posture category name of the stable stationary unit immediately after this unit Is “Frotal-view”, the activity detection unit score is set to “1”.

(4) The posture category name of the immediately preceding stable stationary unit is “Up”, the gesture category of this activity detecting unit is “Look down”, and the posture category name of the stable stationary unit immediately after this unit Is “Frotal-view”, the activity detection unit score is set to “1”.

(5) The posture category name of the immediately preceding stable stationary unit is “Down”, the gesture category of this activity detecting unit is “Look up”, and the posture category name of the stable stationary unit immediately after this unit Is “Frotal-view”, the activity detection unit score is set to “1”.

(6) In all other cases, the score is “1”. The score of the stable stationary unit is evaluated based on the following criteria.

(1) If the posture category name of this unit is “Frontal-view”, the score is “1”.

(2) In other cases, the score is “0”. The scores thus obtained are accumulated for a predetermined time (steps 296 and 298). Every time a predetermined time elapses, the accumulation result is calculated by equation (26) and output as a reaction score Sf (step 300). In equation (26), T represents the accumulation time, S _u represents the score of the u-th unit, and t _u represents the duration of the u-th unit. This score evaluation method is an example, and any evaluation method can be adopted depending on the application.

Subsequently, the work area for score calculation is cleared (step 302), and the control returns to step 292.
Hereinafter, the same processing is repeated.

The above-mentioned final score S _f is transmitted to the transmitting / receiving section 108.
(See FIG. 4).

The above is an outline of the configuration and operation of the lecture hall system 34. [Aggregation Center 30] Referring to FIG. 15, the aggregation center 30 transmits and receives a score transmitted from the attendance hall system 34 and a transmission / reception unit 320 for communicating with the attendance hall system 34 and the instructor system 32. The result totaling unit 322 for totaling over all attendance hall systems 34, the system setting unit 324 for the user to operate for setting the totaling center 30, the totaling method, and the like, and the system setting unit 324 are set. The system management unit 326 controls the counting method in the result counting unit 322 according to the conditions, and periodically transmits the counting result to the instructor system 32 via the transmission / reception unit 320.
And a system management unit 326 for displaying the operation status of the tallying center 30, the tallying status of the result tallying unit 322, and the like.
And a display device 328 used by the device.

FIG. 15 shows the simplest configuration. However, based on the description of the output of the attendance hall system 34 as described above, those skilled in the art can use a general computer to calculate the totaling center. 30 could easily be created. In this embodiment, the counting center 30 is provided separately from the lecturer system 32 or the lecture hall system 34. However, of course, the present invention is not limited to such a configuration. For example, the tallying center 30 may be implemented on the same computer as any one of the lecturer system 32 or the lecture hall system 34. [Instructor System 32] Referring to FIG. 16, instructor system 32 converts video signal output from video camera 50 into a digital video signal, and compresses the video signal. Image compression unit 34 for
0, a transmitting / receiving unit for transmitting the compressed image output from the image compressing unit 340 to each attendance hall system 34 and the counting center 30, and receiving the counting result of the score indicating the audience's response from the counting center 30. 342 and information indicating the reaction, such as totaling for display, totaling, classification, ordering, etc. for the audience venue system 34 with respect to the total result of the audience response score received from the totaling center 30 by the transmitting / receiving unit 342. A result processing unit 344 for executing information processing to be performed on a display condition, a display condition setting unit 346 for setting a format in which a result output from the result processing unit 344 is displayed, and a display condition Setting unit 34
6. The result processing unit 34 according to the conditions set by
4 for generating a video signal that expresses the audience's response in an easy-to-understand manner based on the output of FIG. 4 and a video signal output by the video generation unit 348.
The above-described large screen display device 86 is included.

As a display format of the reaction, for example, the response may be simply converted into a percentage and displayed as a number so that if the score becomes “1” in all attendance hall systems 34, the response becomes 100%. Good. In this embodiment, the response of the audience can be displayed as a number. Therefore, any display format can be realized based on these numbers. For example, it is conceivable to provide a display section for each attendance hall system 34 and display the audience response in color for each attendance hall, or to display it in a graph format such as a bar graph or a pie chart. In addition, the images of a plurality of persons (for example, about 10 to 30 persons) representing the audience are synthesized, and the postures of the images of the persons are adjusted so that the reactions of the persons correspond exactly to the ratios of the reactions of the entire audience. , Movement, expression, etc. may be controlled. [Operation of System] This system having the above-described configuration operates as follows. The instructor system 32 establishes a connection with the counting center 30 in advance. The attendance hall system 34 establishes an appropriate connection with the aggregation center 30 on the network so that the system can receive the video from the instructor system 32 and transmit information to the aggregation center 30. Keep it.

In the instructor system 32, a video signal (including an audio signal) obtained by imaging the instructor is compressed by the image compressing section 340 and transmitted to the tallying center 30. Counting Center 3
No. 0 transmits this image to the attended venue system 34 connected thereto.

[0094] The attendance hall system 34 expands the video of the lecturer transmitted from the counting center 30 and displays the video on the large screen display device 86. The attendance hall system 34 also images the audience using the video camera 50,
This is converted into a digital signal for each frame. Further, these frames are classified by the detection and separation subsystem 102 into stable stationary units and activity detection units. Then, based on the video signals of the frames belonging to those units, the determination subsystem 104 estimates the audience response and outputs it as a score. The totalizing section 106 totals the scores at predetermined time intervals and transmits the final score to the totaling center 30 via the transmitting / receiving section 108.

In the tallying center 30, the expression (26) is applied to all of the plurality of attending venue systems 34 connected.
Is performed, and the result is transmitted to the instructor system 32.

[0096] Instructor system 32 receives the tallying result from tallying center 30 and presents to the instructor an image showing the audience's response to the instructor in accordance with a preset expression format so as to express the audience's response.

The instructor looks at the reaction of the audience presented in this manner, changes the topic, takes some action to get the attention of the audience, for example, asks a question, or performs an appropriate action such as asking a question. If the response is satisfactory, the lecturer can continue the lecture or take appropriate action. [Embodiment 2] The system of Embodiment 1 described above
It was a one-way system of information from the lecturer to the audience at each classroom. However, the present invention can be applied not only to such a system but also to a system for transmitting and receiving information in two directions, for example, an electronic conference system using a virtual space. In this case, the contents of the virtual space are transmitted to all the lecture hall systems except the aggregation center in the system based on the same video transmission device as the lecturer system 32 and the response from the other lecture hall systems. What is necessary is just to add a controllable function. In this case, what is equivalent to the instructor system 32 is unnecessary,
A system can be constructed with the counting center 30 and a plurality of attendance hall systems.

FIG. 17 shows such a lecture hall system 3
FIG. Referring to FIG. 17, this attendance hall system 360 includes a video camera 50, a video acquisition unit 100, and the like, similarly to the attendance hall system shown in FIG. 4.
Detection / separation subsystem 102, judgment subsystem 10
4. A totalizing unit 106 is provided.
A transmission / reception unit 370 for transmitting the result of the tallying from the tallying unit 106 to the tallying center 30 to receive a response from the tallying center 30 at another attendance hall system, and other courses obtained from the transmitting / receiving unit 370. The result processing unit 344 (see FIG. 16) responds to the audience reaction in the venue system.
A result processing unit 372 for performing the same processing as described above, and a virtual space for updating information about the virtual space and the person in the virtual space based on the result processed by the result processing unit 372 so as to reflect the aggregation result. Space maintenance part 374
A video generation unit 376 that generates a video of the environment and the person in the virtual space according to the virtual space information maintained by the virtual space maintenance unit 374; and displays the video of the virtual space generated by the video generation unit 376. And a large screen display device 86 for performing the operations. In FIG. 17, blocks having the same functions as the respective blocks shown in FIG. 4 are denoted by the same reference numerals. Their configurations are the same. Therefore, detailed description thereof will not be repeated here.

In the system according to the second embodiment, a plurality of persons in various places are not in a one-to-many format.
The effect is that discussion can be performed while grasping the reaction of each other with each other.

The embodiments disclosed this time are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

[Brief description of the drawings]

FIG. 1 is a diagram showing an overall configuration of a remote broadcasting system according to a first embodiment of the present invention.

FIG. 2 is an external view of a computer for realizing a tallying center 30, a lecturer system 32, and a lecture hall system 34, which constitute the system according to the first exemplary embodiment of the present invention.

FIG. 3 is a block diagram showing a hardware configuration of the computer according to FIG. 2;

FIG. 4 is a block diagram of a lecture hall system 34 of the system according to the first embodiment of the present invention.

FIG. 5 is a functional block diagram of the detection / separation subsystem 102.

FIG. 6 is a functional block diagram of the face area detection unit 120.

FIG. 7 is a diagram illustrating an example of a result of a process performed by the face area detection unit 120;

FIG. 8 is a diagram for explaining a method of classifying a video sequence into units.

FIG. 9 is a diagram for explaining a method of reconfiguring a unit into a stable stationary unit and an activity detecting unit.

FIG. 10 is a diagram showing a flow of a process of classifying a video sequence into units.

FIG. 11 is a functional block diagram of a determination subsystem 104.

FIG. 12 is a functional block diagram of a posture estimating unit 230.

13 is a functional block diagram of a gesture recognition unit 232. FIG.

FIG. 14 is a flowchart illustrating a processing configuration of a counting unit.

FIG. 15 is a functional block diagram showing the configuration of the counting center 30.

FIG. 16 is a functional block diagram showing the configuration of the instructor system 32.

FIG. 17 is a functional block diagram of a lecture hall system 360 according to the second embodiment of the present invention;

[Explanation of symbols]

Reference Signs List 20 remote broadcasting system, 30 counting center, 32 lecturer system, 34 lecture hall system, 50 camera, 100 video acquisition unit, 102 detection / separation subsystem, 104 judgment subsystem, 106 counting unit, 1
Reference Signs List 20 face area detection unit, 122 video stream unit separation unit, 124 still unit classification unit, 126 unit integration unit, 230 posture estimation unit, 232 gesture recognition unit, 242, 262 feature vector extraction processing unit, 2
44 posture determination unit, 264 gesture determination unit.

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] August 18, 2000 (2000.8.1)
8)

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] 0050

[Correction method] Change

[Correction contents]

The face area detecting section 120 further outputs rg
b color space conversion processing 132 and rg space similarity calculation processing 1
34 including (NCb, NCr) color space conversion process 13 6 Contact and NCb-NCr space similarity calculation process 13 8 for performing at NCb-NCr color space processing similar to. The values of the Y, Cb, and Cr color components for calculating NCb and NCr are given by equations (8) to (1).
0). Further, these are expressed by formula (11).
By normalizing as shown in (12), NCb and NCr color components are obtained.

[Procedure amendment 2]

[Document name to be amended] Statement

[Correction target item name] 0052

[Correction method] Change

[Correction contents]

The face area detection unit 120 further performs the similarity mapping calculated by the rg space similarity calculation processing 134 and the NC
and b-NCr space similarity calculation process 13 8 integration process 140 of calculation of similarity to produce a new similarity mapping integrates calculated a similarity mapping, post-processing on the image, and similar According to the final similarity mapping obtained by the integration processing 140 of degree calculation, a post-processing 142 for expressing a portion corresponding to the face area as an ellipse or a rectangular area and outputting information indicating its position is included. Including.

[Procedure amendment 3]

[Document name to be amended] Statement

[Correction target item name] 0062

[Correction method] Change

[Correction contents]

The video stream unit separation unit 122
The inter-frame difference 150 shown in FIG. 8 is converted into the stationary units 160, 164, 168, 17 according to the above-described definition.
2 and 176, and motion units 162, 16
6, 170, 174 and 178.

[Procedure amendment 4]

[Document name to be amended] Statement

[Correction target item name] 0071

[Correction method] Change

[Correction contents]

Referring to FIG. 12, posture estimating section 230 includes:
Feature vector extraction processing unit 242 for extracting an image feature vector from stable still unit frame image 240
And learning in advance to output information (posture category name) indicating the posture of the audience corresponding to the stable still unit frame image 240 using the feature vector of the frame image from the feature vector extraction processing unit 242 as an input. And a posture determination unit 244 using a neural network. In the system of this embodiment, as the posture determination unit 24 4 by the neural network one neural net corresponding to one posture categories to be detected, a plurality of pair have already done previously learned 1 New over <br/> and Rarunetto 250, the same check the output of the one-to-one New over Rarunetto 25 0 plurality of these for the input feature vector, 1 pair exhibited the highest output value New over Rarunetto 2
And a maximum value detection unit 252 for outputting a posture category name corresponding to 50.

[Procedure amendment 5]

[Document name to be amended] Statement

[Correction target item name] 0072

[Correction method] Change

[Correction contents]

[0072] The effect of this way, if you provided one of the one-to-one New over Rarunetto 250 for one of the attitude category, easily can respond when you try to add a recognition function for a new attitude category There is.
That is, if so, adds a one to one New over <br/> Rarunetto 25 0 previously learning has been carried out to detect the new posture categories posture determination unit 244 according to the neural network, maximize its output What is necessary is just to connect to the input to the value detection part 252. If the whole posture determination unit 244 using a neural network is a large neural network, it is necessary to relearn the entire neural network in order to add a function for detecting a new posture category. Therefore, it is practical to be provided a plurality of one-to-one two <br/>-menu Rarunetto 250 as posture determination unit 244 according to the neural network in practical applications.

[Procedure amendment 6]

[Document name to be amended] Statement

[Correction target item name] 0073

[Correction method] Change

[Correction contents]

Referring to FIG. 13, gesture recognition unit 2
32 has a similar configuration. That gesture recognition unit 232, the activity detection unit frame image information 26 0
And the extraction process 26 2 feature vectors to extract the feature vectors of pressurized et al images, gestures audience as input the feature vector classified into categories gesture by the neural network for outputting the information (gesture category name) And a determination unit 264.

[Procedure amendment 7]

[Document name to be amended] Statement

[Correction target item name] 0090

[Correction method] Change

[Correction contents]

The above is an outline of the configuration and operation of the lecture hall system 34. [Aggregation Center 30] Referring to FIG. 15, the aggregation center 30 transmits and receives a score transmitted from the attendance hall system 34 and a transmission / reception unit 320 for communicating with the attendance hall system 34 and the instructor system 32. The result totaling unit 322 for totaling over all attendance hall systems 34, the system setting unit 324 for the user to operate for setting the totaling center 30, the totaling method, and the like, and the system setting unit 324 are set. controls aggregation method in result counting unit 322 in accordance with conditions, total result the system manager 326 for periodically transmitting the instructor system 32 via the transceiver 32 0
And a system management unit 326 for displaying the operation status of the tallying center 30, the tallying status of the result tallying unit 322, and the like.
And a display device 32 8 are used.

[Procedure amendment 8]

[Document name to be amended] Statement

[Correction target item name] 0091

[Correction method] Change

[Correction contents]

FIG. 15 shows the simplest configuration. However, based on the description of the output of the attendance hall system 34 as described above, those skilled in the art can use a general computer to calculate the totaling center. 30 could easily be created. In this embodiment, the counting center 30 is provided separately from the lecturer system 32 or the lecture hall system 34. However, of course, the present invention is not limited to such a configuration. For example, the tallying center 30 may be implemented on the same computer as any one of the lecturer system 32 or the lecture hall system 34. [Instructor System 32] Referring to FIG. 16, instructor system 32 converts video signal output from video camera 50 and video camera 50 into a digital video signal, and compresses the video signal. Image compression unit 340 for performing
And a transmission / reception unit 342 for transmitting the compressed image output from the image compression unit 340 to each attendance hall system 34 and the tallying center 30, and receiving from the tallying center 30 the tallying result of the score indicating the audience response. With respect to the tallying result of the audience response score received from the tallying center 30 by the transmission / reception unit 342, the information indicating the reaction such as tallying for display, tallying, classification, and ordering for each attendance hall system 34 is displayed. A result processing unit 344 for executing information processing to be performed on the display condition, a display condition setting unit 346 for setting a format in which a result output from the result processing unit 344 is displayed, and a display condition setting Part 34
6. The result processing unit 34 according to the conditions set by
4 and a large-screen display device for displaying a video signal output from the video generation unit 348 for generating a video signal that clearly expresses the audience's response based on the output of the display unit 4. 86.

[Procedure amendment 9]

[Document name to be amended] Statement

[Correction target item name]

[Correction method] Change

[Correction contents]

[0094] The attendance hall system 34 expands the video of the lecturer transmitted from the counting center 30 and displays the video on the large screen display device 86. Together the student venue system 3 4 captures an audience using a video camera 50, converts it into a digital signal for each frame. further,
These frames are classified by the detection and separation subsystem 102 into stable stationary units and activity detection units.
Then, based on the video signals of the frames belonging to those units, the determination subsystem 104 estimates the audience response and outputs it as a score. Counting unit 106 transmits the final score counting center 30 through the transceiver 10 8 by aggregating the scores for each predetermined time.

[Procedure amendment 10]

[Document name to be amended] Statement

[Correction target item name] 0097

[Correction method] Change

[Correction contents]

The instructor looks at the reaction of the audience presented in this manner, changes the topic, takes some action to get the attention of the audience, for example, asks a question, or performs an appropriate action such as asking a question. If the response is satisfactory, the lecturer can continue the lecture or take appropriate action. [Embodiment 2] The system of Embodiment 1 described above
It was a one-way system of information from the lecturer to the audience at each classroom. However, the present invention can be applied not only to such a system but also to a system for transmitting and receiving information in two directions, for example, an electronic conference system using a virtual space. In this case, the contents of the virtual space are transmitted to all the lecture hall systems except the aggregation center in the system based on the same video transmission device as the lecturer system 32 and the response from the other lecture hall systems. What is necessary is just to add a controllable function. In this case, what is equivalent to the instructor system 32 is unnecessary,
A system can be constructed with the counting center 30 and a plurality of attendance hall systems.

[Procedure amendment 11]

[Document name to be amended] Statement

[Correction target item name] 0098

[Correction method] Change

[Correction contents]

FIG. 17 shows such a lecture hall system 3
FIG. Referring to FIG. 17, this attendance hall system 360 includes a video camera 50, a video acquisition unit 100, and the like, similarly to the attendance hall system shown in FIG. 4.
Detection / separation subsystem 102, judgment subsystem 10
4, includes a counting unit 106, in addition to the Re its, totaling unit 106
And a transmission / reception unit 370 for transmitting the result of the tallying to the tallying center 30 and receiving a response from the tallying center 30 at another attendance hall system.
A result processing unit 372 and a result processing unit 372 for performing the same processing as the result processing unit 344 (see FIG. 16) with respect to the audience reaction in the other attendance hall system obtained from
Virtual space maintaining unit 374 for updating information about the virtual space and the person in the virtual space so as to reflect the totaling result based on the result processed by the virtual space maintaining unit 374, and the virtual space maintained by the virtual space maintaining unit 374 an image generating unit 37 6 to generate an image of the environment and the person in the virtual space according to the information, a large-screen display device 86 for displaying the image of the virtual space generated by the image generation unit 376
And In FIG. 17, blocks having the same functions as the respective blocks shown in FIG. 4 are denoted by the same reference numerals. Their configurations are the same. Therefore, detailed description thereof will not be repeated here.

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 7/15 630 G06F 15/70 465A (72) Inventor Shinjiro Kawato Ojiya, Sanikacho, Soraku-gun, Kyoto Pref. No. 5 Hiratani A / T Intelligent Motion Imaging Communications Laboratory (72) Inventor Atsushi Atsushi Atsushi Oya, Seika-cho, Kyoto Prefecture Term (reference) 2C028 AA12 BA02 BB04 BC05 BD02 CA12 DA06 5B057 AA20 BA02 CA01 CA08 CA12 CA16 DA12 DB02 DB06 DB09 DC08 DC25 DC32 DC40 5C064 AA02 AB04 AC04 AC13 AC18 AD08 AD14 5L096 DA02 FA19 GA08 GA41 HA02 HA04 HA11 JA04 9 EE05 FF02 GG05 HH06 HH20 HH30 HH31

Claims (16)

[Claims] 1. A reaction recognition device for recognizing a response of a person to a displayed video, comprising: a video obtaining unit for obtaining a video sequence of a face image of a person; First classifying means for classifying into a stable stationary unit and an activity detecting unit based on motion between frames; and a human response from the frame sequence classified into the stable stationary unit and the frame sequence classified into the activity detecting unit Reaction recognition means for recognizing
Human reaction recognition device. 2. The method according to claim 1, wherein the first classifying unit is configured to classify the video sequence into a still unit and a moving unit based on an inter-frame difference between adjacent frames. Third classification means for classifying the unstable stationary unit and the unstable stationary unit based on the duration thereof, and integrating the continuous unstable stationary unit and the motion unit as the activity detecting unit. 2. The apparatus for recognizing a person's reaction according to claim 1, further comprising: integrating means for performing the following. 3. The reaction recognition unit includes: a first feature vector extraction unit configured to extract a feature vector of a human face image from a frame in the stable stationary unit; and an output of the first feature vector extraction unit. A first neural network, which has been trained in advance and outputs posture information corresponding to the feature vector as an input of a feature vector to be input, and a motion of a person's face image from information of a difference between frames in the activity detection unit. A second feature vector extracting unit for extracting a corresponding feature vector, and a gesture vector corresponding to the feature vector, which receives a feature vector output from the second feature vector extracting unit and outputs gesture information corresponding to the feature vector. 2. The apparatus for recognizing a person's reaction according to claim 1, further comprising: a second neural network. 4. At least one of the first neural network and the second neural network is associated with a predetermined reaction category, and is output from the first or second feature vector extracting means. 4. The apparatus according to claim 3, further comprising a plurality of one-to-one neural networks that receive a feature vector as input and output a degree of relevance to the predetermined reaction category. 5. 5. A video camera, a video camera, a video signal converter for converting a video signal output from the video camera into a digital signal for each frame, and a video sequence output from the video signal converter. 5. The apparatus according to claim 1, further comprising: a face area specifying unit configured to specify a face area of the person in each frame based on the information. 6. A first means for specifying a face area of a person in each frame by a first method based on a video sequence output from the video signal converting means, A second unit for specifying a face area of a person in each frame by a second method based on a video sequence output by the video signal conversion unit; and a first unit and a second unit. The apparatus for recognizing a person's reaction according to any one of claims 1 to 5, further comprising means for integrating a face area by integrating the result of specifying the face area. 7. The video sequence is an RGB video sequence, and the first means is based on a similarity between a color distribution in a video obtained by converting the RGB color video sequence into an rg color space and a predetermined color distribution pattern. 7. The apparatus for recognizing a person's reaction according to claim 6, further comprising means for specifying a face area by using the apparatus. 8. The image sequence is an RGB color image sequence, and the first unit is configured to perform similarity between a color distribution in an image obtained by converting the RGB color image sequence into an NCb-NCr color space and a predetermined color distribution pattern. 8. The apparatus according to claim 6, further comprising means for specifying a face area based on the degree. 9. A computer-readable recording medium storing a program for causing a computer to operate as a reaction recognition device for recognizing a reaction of a person to a displayed image, wherein the program comprises: A first classification program portion for classifying a video sequence acquired for an image into a stable still unit and an activity detection unit based on motion between video frames; and a frame sequence classified as a stable still unit; A computer-readable recording medium comprising: a response recognition program portion for recognizing a human response from a frame sequence classified by the activity detection unit. 10. The first classification program portion, the second classification program portion for classifying a video sequence into a still unit and a motion unit based on an inter-frame difference between adjacent frames; A third classification program part for classifying a frame into the stable stationary unit and the unstable stationary unit based on the duration thereof; and a continuous unstable stationary unit and the motion unit as the activity detecting unit. 10. The computer-readable recording medium according to claim 9, comprising an integrated program part for integrating. 11. The reaction recognition program portion includes: a first feature vector extraction program portion for extracting a feature vector of a human face image from a frame in the stable still unit; and the first feature vector extraction program. A first neural network program portion that has been trained in advance, and outputs a posture information corresponding to the feature vector, using a feature vector output from the portion as an input, and a human neural network based on information of a difference between frames in the activity detection unit. A second feature vector extraction program portion for extracting a feature vector corresponding to the movement of the face image; and a feature information output from the second feature vector extraction program portion as input, gesture information corresponding to the feature vector Output a second neural network that has been learned in advance And a network program portion, the computer readable recording medium of claim 10. 12. At least one of the first neural network program portion and the second neural network program portion is each associated with a predetermined reaction category, and
12. The program according to claim 11, further comprising a plurality of one-to-one neural network program parts, which output a high degree of relevance to the predetermined reaction category by using as input a feature vector output from the feature vector extraction program part. Computer readable recording medium. 13. The program according to claim 13, further comprising a face area specifying program part for specifying a face area of a person in each frame based on the video sequence and providing the face area to the first classification program part. 9 to Claim 12
A computer-readable recording medium according to any one of the above. 14. A face area identification program portion, comprising: a first program portion for identifying a face area of a person in each frame by a first method based on the video sequence; and A second program portion for specifying a face area of a person in each frame by a second method, and a result of specifying the face region by the first program portion and the second program portion. 14. The computer-readable recording medium according to claim 9, further comprising a face area integration program part for specifying a face area. 15. The image sequence is an RGB color image sequence, and the first program portion is a similarity between a color distribution in an image obtained by converting the RGB color image sequence into an rg color space and a predetermined color distribution pattern. 15. The computer-readable recording medium according to claim 14, further comprising a program part for specifying a face area based on the computer program. 16. The video sequence is an RGB color video sequence, and the first program portion includes a color distribution in a video obtained by converting the RGB color video sequence into an NCb-NCr color space and a predetermined color distribution pattern. 16. The computer-readable recording medium according to claim 14, further comprising a program part for specifying a face area based on the degree of similarity.