JP7130290B2 - information extractor - Google Patents

Description

The present disclosure relates to an information extraction device.

In recent years, distribution services for receiving distribution of various contents are becoming popular.

Patent Literature 1 discloses a technique that enables a process of detecting desired music content by inputting a hum of the music to be searched even when the user does not know the title of the music content that the user wishes to distribute. It is

JP-A-2002-55994

By the way, in the technology described in Patent Document 1, since the content to be distributed is limited to music content, in order to perform processing for detecting search targets for all other video content, a huge amount of arithmetic processing by a computer is required. necessary.

Accordingly, the present disclosure has been made in view of such circumstances, and has an object to provide an information extraction device capable of reducing the load associated with arithmetic processing.

The main invention of the present invention for solving the above problems is characterized by comprising an extraction unit for extracting a specific frame group from a moving image composed of a plurality of frames according to a predetermined specific condition instructed from the outside. do.

According to the present disclosure, it is possible to reduce the load associated with arithmetic processing.

1 is a conceptual diagram showing a configuration example of an at-home tutoring system 1 according to a first embodiment of the present disclosure; FIG. 1 is a diagram showing a hardware configuration example of a computer that realizes a classroom video distribution device 10 according to the first embodiment of the present disclosure; FIG. 1 is a diagram showing a hardware configuration example of a computer that implements a student terminal 20 according to the first embodiment of the present disclosure; FIG. 1 is a diagram showing a software configuration example of a classroom video distribution device 10 according to a first embodiment of the present disclosure; FIG. 4 is a flowchart for explaining the flow of processing of the classroom video distribution method according to the first embodiment of the present disclosure; FIG. 10 is a diagram showing a software configuration example of a classroom video distribution device 10 according to a second embodiment of the present disclosure; FIG. FIG. 11 is a flow chart for explaining the flow of processing of a classroom video distribution method according to a second embodiment of the present disclosure; FIG. FIG. 11 is a diagram showing a software configuration example of a classroom video distribution device 10 according to a third embodiment of the present disclosure; FIG. FIG. 11 is a flow chart for explaining the flow of processing of a classroom video distribution method according to a third embodiment of the present disclosure; FIG. FIG. 11 is a diagram showing a software configuration example of a classroom video distribution device 10 according to a fourth embodiment of the present disclosure; FIG. FIG. 16 is a flow chart for explaining the flow of processing of a classroom video distribution method according to a fourth embodiment of the present disclosure; FIG.

The contents of the embodiments of the present disclosure are listed and described. The present disclosure has the following configurations.
[Item 1]
an acquisition unit that acquires a moving image composed of a plurality of frames;
a storage unit that stores specific conditions for specifying predetermined data included in the moving image;
an extraction unit that extracts a plurality of specific frame groups from the moving image according to the specific condition;
a connection unit that connects the extracted specific frame groups;
an output unit that outputs digest information containing a plurality of concatenated frame groups,
Information extractor.
[Item 2]
The information extraction device according to item 1,
further comprising a waveform registration unit for pre-registering predetermined waveform data,
The specific condition is whether or not the waveform data of the sound contained in the moving image matches the registered waveform data,
When both waveform data match, the extraction unit extracts a frame group corresponding to the matched waveform from the moving image as the specific frame group.
Information extractor.
[Item 3]
The information extraction device according to item 2,
further comprising a conversion unit that converts the sound contained in the video into text information by speech recognition;
wherein the conversion unit converts the sound corresponding to an auxiliary frame group including the specific frame group and a predetermined number of frames before and after the specific frame group;
Information extractor.
[Item 4]
The information extraction device according to item 2 or item 3,
Conversation information and non-conversation information are mixed in the information indicated by the surrounding sounds including the subject.
Information extractor.
[Item 5]
The information extraction device according to item 4,
The conversation information includes at least one of words indicating positive emotions and words indicating negative emotions.
Information extractor.
[Item 6]
The information extraction device according to item 4 or item 5,
The non-conversation information includes information indicating at least one of tongue clicks, sighs, and backtracking.
Information extractor.
[Item 7]
The information extraction device according to item 1,
further comprising a face information registration unit for pre-registering a predetermined face evaluation value related to facial expressions,
The specific condition is whether or not the face evaluation value calculated from the expression of the face included in the moving image matches the registered face evaluation value,
When the two face evaluation values match, the extracting unit extracts the frame group corresponding to the matching face evaluation value from the moving image as the specific frame group.
Information extractor.
[Item 8]
The information extraction device according to item 7,
The face evaluation value includes an evaluation value that evaluates the degree of happiness, boredom, or tension of the person.
Information extractor.
[Item 9]
The information extraction device according to item 7 or item 8,
The face evaluation value includes an evaluation value obtained by evaluating the facial expression of the person, the direction of the line of sight of the person, and the direction of the face of the person.
Information extraction device [Item 10]
The information extraction device according to item 1,
further comprising a motion information registration unit that pre-registers a predetermined motion evaluation value related to the motion of the person;
The specific condition is whether or not a motion evaluation value calculated from a person included in the moving image matches the registered motion evaluation value,
When both motion evaluation values match, the extracting unit extracts a group of frames corresponding to the matching motion evaluation value from the moving image as the specific frame group.
Information extractor.
[Item 11]
11. The information extraction device according to item 10,
The motion evaluation value includes an evaluation value obtained by evaluating at least one motion of the person's gestures, gestures, gestures, and body language.
Information extractor.

[Item 12]
The information extraction device according to item 1,
a biometric information registration unit for pre-registering a biometric evaluation value related to predetermined biometric information;
The specific condition is whether or not a biometric evaluation value that can be calculated from a person included in the moving image matches the registered biometric evaluation value,
When both biometric evaluation values match, the extracting unit extracts a frame group corresponding to the matching biometric evaluation value from the moving image as the specific frame group.
Information extractor.
[Item 13]
13. The information extraction device according to item 12,
The biological evaluation value includes at least one of the person's blood pressure, pulse, and pulse pressure,
Information extractor.
[Item 14]
The information extraction device according to any one of items 1 to 13,
a frame addition unit that adds additional frames that are chronologically successive to the specific frame group, to the specific frame group;
Information extractor.
[Item 15]
identification means for identifying at least face images or voices included in the digest information extracted by the information extraction device according to any one of items 1 to 14 for each predetermined frame unit;
further comprising evaluation means for calculating an evaluation value for the identified face image;
Video meeting evaluation terminal.
[Item 16]
16. A video meeting evaluation terminal according to item 15,
The video meeting evaluation terminal provides graphical information of the evaluation value over time.
Video meeting evaluation terminal.
[Item 17]
A video meeting evaluation terminal according to item 15 or item 16,
The video meeting evaluation terminal calculates a plurality of evaluation values obtained by evaluating the face image from a plurality of different viewpoints.
Video meeting evaluation terminal.
[Item 18]
A video meeting evaluation terminal according to any one of items 15 to 17,
The video meeting evaluation terminal calculates the evaluation value together with the audio included in the moving image.
Video meeting evaluation terminal.
[Item 19]
The video meeting evaluation terminal according to any one of items 15 to 18,
The video meeting evaluation terminal calculates the evaluation value together with an object other than the face image included in the moving image.
Video meeting evaluation terminal.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

In this disclosure, a cram school/educational institution is connected to a student/student's home via a digital communication line, and each student/student can watch a lecture video being given at the cram school/educational institution while staying at home. , an example of applying the information extraction device to a home tutoring system in which one can take classes at cram schools and educational institutions will be described.

<First embodiment>
FIG. 1 is a conceptual diagram showing a configuration example of an at-home tutoring system 1 according to the first embodiment of the present disclosure. As shown in the figure, in this home tutoring system 1, a classroom video distribution device 10 is provided in the classroom of a lecturer T who conducts remote classes, and a group of students (students A, B, Student terminals 20A, 20B, and 20C associated with C) are communicably connected via a network NW. In the following description, the student terminals 20A, 20B, and 20C are simply referred to as the student terminals 20 when there is no need to distinguish them. Similarly, students A, B, and C are simply abbreviated as students when there is no need to distinguish between them.

The classroom video distribution device 10 is an example of the information extraction device described in the claims. In addition, this configuration is an example, and a configuration may have other configurations or may include other configurations. Although three students A, B, and C are shown here, the number of students may be increased according to the number of students who can be instructed by the lecturer at the same time or the number of connection lines of the network NW.

In the present embodiment, "lecturer" is a concept including professor, teacher, and teacher. “Classroom” means cram schools, cultural centers, educational institutions (for example, elementary, secondary, and higher educational institutions, high schools, technical colleges, vocational schools, junior colleges, four-year universities, graduate schools, etc., according to the Ministry of Education, Culture, Sports, Science and Technology). registered school). "Student" is a concept that includes students, students, and auditors.

In this embodiment, the network NW is assumed to be the Internet. The network NW is constructed by, for example, a public telephone line network, a mobile phone line network, a wireless communication network, Ethernet (registered trademark), or the like.

<Hardware configuration>
FIG. 2 is a diagram showing a hardware configuration example of a computer that implements the classroom video distribution apparatus 10 according to this embodiment. The computer includes at least a communication unit 11, an imaging unit 12, a sound pickup unit 13, a monitor 14, a memory 15, a storage 16, an input/output unit 17, a control unit 18, and the like. These are electrically connected to each other through a bus 19 .

The communication unit 11 connects the classroom video distribution device 10 to the network NW. The communication unit 11 is, for example, a wired LAN (Local Area Network), wireless LAN, Wi-Fi (Wireless Fidelity, registered trademark), infrared communication, Bluetooth (registered trademark), short-distance or non-contact communication, or the like. Communicate with the device directly or through a network access point.

The imaging unit 12 has a function of electronic imaging using an imaging device such as CMOS or CCD. The image capturing unit 12 captures an image of the lecturer T, who gives a lecture to the students, as a subject to obtain a lecturer video. The image capturing unit 12 may be configured to capture images written on the blackboard or whiteboard used by the lecturer T when the lecture progresses, but an independent camera may be provided for the blackboard or whiteboard.

The sound pickup unit 13 picks up surrounding sounds including the lecturer T. The sound pickup unit 13 includes a microphone or the like for acquiring surrounding sounds including the lecturer T's voice. Furthermore, the sound pickup unit 13 can perform appropriate processing such as converting the acquired sound into an electric signal.

The monitor 14 can display the student video transmitted from the student terminal 20 and the lecturer video acquired by the imaging unit 12 in a viewable state. Of course, the monitor 14 may display only the student image alone, or may display only the lecturer image alone.

The memory 15 includes a main memory composed of a volatile memory device such as a DRAM (Dynamic Random Access Memory), and an auxiliary memory composed of a non-volatile memory device such as a flash memory or a HDD (Hard Disc Drive). . The memory 15 is used as a work area or the like for the control unit 18, and stores a BIOS (Basic Input/Output System) executed when the classroom video distribution apparatus 10 is started, various setting information, and the like.

The storage 16 stores various programs such as application programs. A database storing data used for each process may be constructed in the storage 16 .

The input/output unit 17 is, for example, information input equipment such as a keyboard, mouse, and touch panel.

The control unit 18 is an arithmetic unit that controls the overall operation of the classroom video distribution apparatus 10, controls transmission and reception of data between elements, executes applications, and performs information processing necessary for authentication processing. For example, the control unit 18 is a processor such as a CPU (Central Processing Unit), and executes programs and the like stored in the storage 16 and developed in the memory 15 to carry out each information process.

A bus 19 is commonly connected to each of the elements and transmits, for example, address signals, data signals and various control signals.

FIG. 3 is a diagram showing a hardware configuration example of a computer that realizes the student terminal 20 according to this embodiment. The computer includes at least a communication unit 21, an imaging unit 22, a sound pickup unit 23, a monitor 24, a memory 25, a storage 26, an input/output unit 27, a control unit 28, and the like. These are electrically connected to each other through a bus 29 . A hardware configuration such as a computer that realizes the student terminal 20 according to the present embodiment is the same as the hardware configuration example of the classroom video distribution apparatus 10 shown in FIG. 2, so only differences will be described.

The communication unit 21 connects the student terminal 20 to the network NW.

The imaging unit 22 captures an image of a student taking a lecture as a subject to acquire a student video.

The sound pickup unit 23 picks up surrounding sounds including the student. In order to transmit and receive audio data, the student terminal 20 may be provided with a headphone with a microphone, or a microphone and a speaker built into the terminal may be used.

The monitor 24 can display the lecturer video transmitted from the classroom video distribution device 10 and the student video acquired by the imaging unit 22 in a viewable state. Of course, the monitor 24 may display only the lecturer image alone, or may display only the student image alone.

The control unit 28 is an arithmetic unit that controls the overall operation of the student terminal 20, controls transmission and reception of data between elements, executes applications, and performs information processing necessary for authentication processing.

<Software configuration>
FIG. 4 is a diagram showing a software configuration example of the classroom video distribution apparatus 10 according to this embodiment. Classroom video distribution apparatus 10 includes extractor 101 , waveform register 102 , converter 103 , display 104 , frame extractor 105 , and generator 106 .

The extraction unit 101, the waveform registration unit 102, the conversion unit 103, the display unit 104, the frame extraction unit 105, and the generation unit 106 are configured so that the control unit 18 reads the program stored in the storage 16 to the memory 15. It can be realized by executing

The extraction unit 101 appropriately selects and synthesizes the lecturer video acquired by the imaging unit 12 and the student video acquired by the imaging unit 22, and the sound is collected by the sound collection unit 13 or the sound collection unit 23. A classroom video is generated by combining the sounds. The lecturer video or student video here is an example of a plurality of frames described in the claims. Also, the classroom video is an example of the moving image described in the claims. Classroom videos are text data, numerical data, graphic data, image data, video data, audio data, etc., or a combination of these. Refers to items that can be exchanged, including items similar to these.

The extraction unit 101 has a function of extracting a specific frame group from the classroom video according to a predetermined specific condition instructed from the outside. For example, the instruction from the outside may be an instruction for the items of the classroom video to be synthesized, the layout of the images, the area occupied by the images, and the like. The instruction from the outside is preferably, for example, a simple operation that allows the lecturer T to edit the video composition during the lecture.

The waveform registration unit 102 has a function of pre-registering predetermined waveform data. The predetermined specific condition is, for example, whether or not the waveform data of the sound picked up by the sound pickup unit 13 or the sound pickup unit 23 matches the waveform data registered in the waveform registration unit 102. Of course, other conditions are possible. The waveform data registered in the waveform registration unit 102 in this embodiment can be used to evaluate the overall atmosphere of the lecture.

When a predetermined specific condition is satisfied, for example, when both waveform data match, the extraction unit 101 extracts a frame group corresponding to the matched waveform as a specific frame group from the classroom video.

The conversion unit 103 has a function of converting the sound collected by the sound collection unit 13 into text by speech recognition. This text is any string of characters. When the speech recognition is successful, the conversion unit 103 outputs the speech recognition result including the generated text. The speech recognition result may include success information indicating that the speech recognition was successful.

The display unit 104 has a function of displaying the text converted by the conversion unit 103 on the monitor 14 or the monitor 24 . Conversational information and non-conversational information are mixed in the information indicated by the surrounding sounds including the lecturer T or the students A, B, and C. The conversation information includes, for example, at least one of words indicating positive emotions and words indicating negative emotions.

Examples of words that express positive emotions include words used by instructors to praise, support, and encourage students: "You did a great job," "You did a great job," "You're amazing," "You're amazing," You can say "ne", "wonderful", and so on.

On the other hand, as an example of words that show negative emotions, the lecturer puts down, criticizes, or slanders the student, such as "You can't do that," "You can't do it," "What are you doing?" "Idiot", etc. can be mentioned.

Non-conversation information is text information other than conversation information. The non-conversational information includes information indicating at least one of tongue clicking, sighing, and backtracking. These non-conversational information can serve as criteria for judging the emotions of the students attending the lecture. Students' emotions can be classified into, for example, "happiness", "boredom", and "tension".

The frame clipping unit 105 has a function of clipping at least a group of frames that are consecutive before and after the specific frame group extracted by the extraction unit 101 . The frame extraction unit 105 can extract a group of consecutive frames using any applicable frame rate for obtaining context information indicating in what context each word is used, for example. . The context information here is, for example, information indicating a character string in an arbitrary range before and after a word, a co-occurrence relationship between words, and the like.

The generation unit 106 has a function of connecting a preceding frame group and subsequent frames that are continuous before and after a specific frame group extracted by the extraction unit 101 to generate a digest video.

Next, the operation of the at-home tutoring system 1 configured as described above will be described. FIG. 5 is a flow chart explaining the flow of processing of the classroom video distribution method according to the first embodiment of the present disclosure.

Here, an example will be described in which an operator or the like who manages a classroom checks whether or not a lecturer has made inappropriate or problematic remarks to students. Specifically, a scene in which a digest video is generated using waveform data representing "stupid" as a word indicating a negative emotion in the voice of a pre-registered lecturer will be described as an example.

First, when the lecture starts, the students A, B, and C connect the student terminals 20A, 20B, and 20C to the home tutoring system 1 via the network NW, and wait for the instructor T to start the lecture. . Lecturer T looks at monitor 14 provided in classroom video distribution apparatus 10 and determines whether or not students A, B, and C are ready to attend, and if they are ready to attend, starts the lecture. That is, the imaging operation by the imaging unit 12 and the imaging unit 22 is started, and the sound pickup operation by the sound pickup unit 13 and the sound pickup unit 23 is started (step S100).

Then, the extracting unit 101 selects and synthesizes the lecturer video acquired by the imaging unit 12 and the student video acquired by the imaging unit 22 as appropriate, and collects The produced sounds are combined to generate a classroom image, and the generated classroom image is distributed to the student terminals 20A, 20B, and 20C via the network NW (step S102).

Next, when the lecturer utters "stupid", the sound is picked up by the sound pickup unit 13, and the classroom video including the sound data is extracted by the extraction unit 101 as a specific frame group (step S104).

The frame cutout unit 105 cuts out a group of frames that are continuous at time intervals of, for example, about 10 to 20 seconds before and after the classroom video (step S106).

Then, the generation unit 106 generates a digest video by connecting these frame groups, and performs a predetermined acoustic analysis on the lecturer's voice data based on the digest video (step S108).

According to this acoustic analysis, it is possible to grasp in what context the word "stupid" was used.

For example, if the instructor uses the word "stupid" in the context of "a person who says such stupid things is useless" (to a third party other than the student), the instructor will You can understand that I didn't say that I was stupid.

Also, for example, when the teacher uses the word "stupid" in the context of "because I'm stupid", it can be understood that the lecturer calls himself stupid.

Thus, the administrator or the like who manages the classroom can obtain the word "stupid" and the context before and after it as text sentences, for example, in the form of a report, so that the lecturer cannot make inappropriate or problematic remarks to the students. You can easily check if it is not

In other words, by matching using pre-registered waveform data, it is possible to reduce the load associated with arithmetic processing, and at the same time reduce the likelihood that the lecturer will make inappropriate or problematic remarks in the classroom video. It is possible to draw out the scene with pinpoint. For example, if the number of times the lecturer utters words indicating negative emotions reaches a relatively large predetermined number of times (for example, 10 times) for the duration of the lecture (for example, 90 minutes), the human nature of the lecturer is determined. It is also possible to Of course, the number of times the instructor utters words that indicate positive emotions can also be used as a material for judging the instructor's humanity. Furthermore, the number of tongue clicks, sighs, and backtracking in response to utterances from the lecturer, not only from the instructor's side, but also from the student's side, is a material for judging what kind of feelings the student has. can be.

<Second embodiment>
The home tutoring system according to the second embodiment will be described below with reference to FIGS. 6 and 7. FIG. In this embodiment, the same reference numerals are assigned to the same elements as those described in the above-described first embodiment, and detailed description thereof will be omitted.

In the at-home tutoring system according to the first embodiment, as already described, a specific frame group is extracted from the classroom video by matching using pre-registered waveform data. In the at-home tutoring system according to the embodiment, a process of extracting a specific frame group from the classroom video is performed based on the evaluation value regarding the facial expression of the person.

<Software configuration>
FIG. 6 is a diagram showing a software configuration example of the classroom video distribution apparatus 10 according to this embodiment. Classroom video distribution apparatus 10 includes extraction unit 101 , frame extraction unit 105 , generation unit 106 , face detection unit 107 , calculation unit 108 , and face information registration unit 109 .

The face detection unit 107 has a function of detecting a person's face for each of a plurality of frames forming the classroom video stored in the memory 15, for example. For example, the face detection unit 107 extracts feature points from the classroom video, and detects the face area of the lecturer T or each of the students A, B, and C, the size of the face area, the face area, and the like.

Feature points include, for example, end points of the eyebrows, eyes, nose, and lips, face contour points, vertex of the head, lower end points of the chin, and the like. Then, the face detection unit 107 identifies the position information of the face area. For example, the face detection unit 107 calculates the X and Y coordinates of the pixels included in the face area, with the horizontal direction of the image as the X axis and the vertical direction as the Y axis. Furthermore, the face detection unit 107 can determine the expression, age, etc. of the detected lecturer or each student by arithmetic processing using the feature points described above.

The calculation unit 108 has a function of calculating an evaluation value regarding the face of the lecturer T or each of the students A, B, and C for each of a plurality of frames forming the classroom video. Each evaluation value calculated by the calculation unit 108 includes predetermined evaluation values (1) to (6) shown below. These evaluation values (1) to (6) are registered in the face information registration unit 109 in advance.

(1) The degree-of-smile calculation unit 108 calculates the degree of smile as an evaluation value for each face detected by the face detection unit 107, for example, using a known technique such as pattern matching. In this embodiment, as an example of the degree, the degree of smile is shown in four stages up to "0: no smile", "1: smile", "2: normal laugh", and "3: big laugh".

(2) The line-of-sight direction calculation unit 108 uses a known technique to calculate the line-of-sight direction as an evaluation value for each face detected by the face detection unit 107 . In the present embodiment, as an example, the line-of-sight direction is indicated by four types: "0: front of line of sight", "1: left and right direction of line of sight", "2: right of line of sight", and "3: undetectable line of sight".

(3) The face orientation calculation unit 108 uses a known technique to calculate the orientation of each face detected by the face detection unit 107 as an evaluation value. In the present embodiment, as an example, the face orientation is indicated by four types: "0: facing front", "1: facing left", "2: facing right", and "3: undetectable".

(4) The face area calculation unit 108 calculates the area of the face portion as an evaluation value for each face detected by the face detection unit 107 .

(5) The age calculator 108 uses a known technique to calculate the age of each face detected by the face detector 107 as an evaluation value.

(6) The blink degree calculation unit 108 calculates the blink degree as an evaluation value for each face detected by the face detection unit 107 using a known technique. In this embodiment, as an example, the degree of blinking can be set in four types: "0: no blinking", "1: partially blinking", "2: both eyes blinking", and "3: blinking undetectable". show.

These evaluation values (1) to (6) can serve as judgment criteria for judging the emotions of the students attending the lecture. Students' emotions can be classified into, for example, "happiness", "boredom", and "tension".

When a predetermined specific condition is satisfied, for example, when the evaluation value calculated by the calculation unit 108 and the evaluation value registered in advance in the face information registration unit 109 match, the extraction unit 101 determines whether the match is detected. A frame group corresponding to the obtained evaluation value is extracted from the classroom video as a specific frame group.

Next, the operation of the at-home tutoring system configured as described above will be described. FIG. 7 is a flowchart for explaining the flow of processing of the classroom video distribution method according to the second embodiment of the present disclosure.

In this case, the administrator who manages the classroom responds to the request from the guardian who entrusts the child of student A to a cram school, etc., and wants to understand the learning attitude and learning situation of student A. will be described as an example.

First, when the lecture starts, the students A, B, and C connect the student terminals 20A, 20B, and 20C to the home tutoring system via the network NW, and wait for the instructor T to start the lecture. Lecturer T looks at monitor 14 provided in classroom video distribution apparatus 10 and determines whether or not students A, B, and C are ready to attend, and if they are ready to attend, starts the lecture. That is, the imaging operation by the imaging unit 12 and the imaging unit 22 is started, and the sound pickup operation by the sound pickup unit 13 and the sound pickup unit 23 is started (step S200).

Then, the extracting unit 101 selects and synthesizes the lecturer video acquired by the imaging unit 12 and the student video acquired by the imaging unit 22 as appropriate, and collects The produced sounds are combined to generate a classroom image, and the generated classroom image is distributed to the student terminals 20A, 20B, and 20C via the network NW (step S202).

Next, the face detection unit 107 detects the face of student A in the classroom video (step S204). A specific method for face detection is the same as a known technique, so description thereof will be omitted.

Then, the calculation unit 108 calculates the orientation of the face of student A detected by the face detection unit 107 as an evaluation value using a known technique (step S206). Subsequently, the extracting unit 101 determines whether the evaluation value calculated by the calculating unit 108 is one of the evaluation values “1: face direction left” and “2: face direction right” registered in advance in the face information registration unit 109. If any match is found, the frame group corresponding to the matching evaluation value is extracted from the classroom video as a specific frame group (step S208).

In other words, when student A's face is facing leftward or rightward instead of front, it can be inferred that student A's concentration on the lecture is declining.

The frame extracting unit 105 extracts a group of frames that are continuous at time intervals of about 10 to 20 seconds, for example, before and after the classroom video in which student A's face is not facing the front (step S210).

Then, the generation unit 106 generates a digest video by connecting these frame groups, and performs predetermined acoustic analysis on the speech data of the lecturer based on the digest video (step S212). According to such acoustic analysis, it is possible to manage only what words the lecturer T is uttering when the student A is not facing the front using a flag. As a result, it is possible for the administrator of the classroom to collect words that may be the cause of student A's decreased concentration on the lecture, and requests from parents, etc. to understand student A's learning attitude and learning situation. can be efficiently responded to.

Of course, the degree of concentration of the student A on the lecture can be determined not only by the orientation of the student A's face, but also by the orientation of the student A's line of sight. That is, when student A's line of sight is directed to the left or right rather than to the front, it can be inferred that student A's degree of concentration on the lecture is declining.

The following usage examples are conceivable for the at-home tutoring system according to the present embodiment.

Concretely, only the scenes in which the student A felt positive emotions such as satisfaction, happiness, and fulfillment are collected and edited to generate a digest movie for the parent or the like of the student A.

As an example of specific processing that can generate such a digest video, first, the calculation unit 108 uses a known technique for the face of student A detected by the face detection unit 107, and calculates the degree of smile as an evaluation value. can be calculated as Next, when the degree calculated by the calculation unit 108 matches one of the degrees “2: normal laughter” or “3: loud laughter” pre-registered in the face information registration unit 109, the extraction unit 101 A frame group corresponding to the matched evaluation value can be extracted from the classroom video as a specific frame group.

Furthermore, the frame extracting unit 105 sets a time interval of, for example, about 10 to 20 seconds before and after the classroom video when the degree of smile of student A is either "2: normal laughter" or "3: loud laughter." A series of frames can be cut out with . Finally, the generation unit 106 generates a digest video by connecting these frame groups as a digest video for the parent or the like of student A. FIG.

<Third Embodiment>
A home tutoring system according to the third embodiment will be described below with reference to FIGS. 8 and 9. FIG. In this embodiment, the same reference numerals are assigned to the same elements as those described in the above-described first embodiment, and detailed description thereof will be omitted.

In the at-home tutoring system according to the first embodiment, as already described, a specific frame group is extracted from the classroom video by matching using pre-registered waveform data. In the at-home tutoring system according to the embodiment, processing for extracting a specific frame group from the classroom video is performed based on the motion information about the motion of the person.

<Software configuration>
FIG. 8 is a diagram showing a software configuration example of the classroom video distribution apparatus 10 according to this embodiment. The classroom video distribution device 10 includes an extraction unit 101 , a frame extraction unit 105 , a generation unit 106 , a specification unit 110 and an action information registration unit 111 .

The specifying unit 110 has, for example, a function of specifying motion information related to a person's motion for each of a plurality of frames forming the classroom video stored in the memory 15 . This motion information is, for example, information in which the motion of a person is captured as a series of a plurality of postures, and may include information on each joint forming the skeleton of the human body corresponding to various postures. The motion information includes, for example, at least one of a person's gestures, gestures, gestures, and body language.

The specifying unit 110 obtains the coordinates of each joint forming the skeleton of the human body from a plurality of frames forming the classroom video, for example, by pattern matching using human body patterns. A specific method of obtaining the coordinates is the same as a known technique, so the explanation is omitted. Then, the coordinates of each joint represented by this coordinate system can be, for example, skeleton information for one frame. Furthermore, skeleton information for a plurality of frames can serve as predetermined motion information.

Such predetermined motion information is registered in the motion information registration unit 111 in advance. That is, the motion information registration unit 111 stores in advance motion information corresponding to various postures, for example, by machine learning using known artificial intelligence technology. For example, in the present embodiment, a gesture corresponding to the movement of a student's hand waving is used as a gesture pattern for a student who is unsatisfied with the content of the lecture or who finds it difficult to ask the lecturer for supplementary explanations. It is assumed that learning has been completed.

If a predetermined specific condition is satisfied, for example, if the motion information specified by the specifying unit 110 and the motion information registered in advance in the motion information registration unit 111 match, the extraction unit 101 determines whether the match is detected. A frame group corresponding to the motion information is extracted from the classroom video as a specific frame group.

Next, the operation of the at-home tutoring system configured as described above will be described. FIG. 9 is a flow chart explaining the flow of processing of the classroom video distribution method according to the third embodiment of the present disclosure.

In this example, a scene in which a student A, who is not satisfied with the content of the lecture, asks the lecturer for supplementary explanation, is picked up from the classroom video.

First, when the lecture starts, the students A, B, and C connect the student terminals 20A, 20B, and 20C to the home tutoring system via the network NW, and wait for the instructor T to start the lecture. Lecturer T looks at monitor 14 provided in classroom video distribution apparatus 10 and determines whether or not students A, B, and C are ready to attend, and if they are ready to attend, starts the lecture. That is, the imaging operation by the imaging unit 12 and the imaging unit 22 is started, and the sound pickup operation by the sound pickup unit 13 and the sound pickup unit 23 is started (step S300).

Then, the extracting unit 101 selects and synthesizes the lecturer video acquired by the imaging unit 12 and the student video acquired by the imaging unit 22 as appropriate, and collects The produced sounds are combined to generate a classroom image, and the generated classroom image is distributed to the student terminals 20A, 20B, and 20C via the network NW (step S302).

Next, the specifying unit 110 obtains the coordinates of each joint forming the skeleton of the student A from a plurality of frames forming the classroom video by pattern matching using the human body pattern. Further, the identifying unit 110 identifies a plurality of frames of skeletal information as student A's gesture based on the coordinates of each joint (step S304).

Subsequently, when the gesture identified by the identifying unit 110 matches a gesture pattern (a gesture pattern in which student A asks the lecturer for supplementary explanation) that has been machine-learned in the motion information registration unit 111, the extracting unit 101 Next, the frame group corresponding to the matching gesture is extracted from the classroom video as a specific frame group (step S306).

The frame extracting unit 105 extracts a group of frames that are continuous at time intervals of about 10 to 20 seconds, for example, before and after the classroom video requesting supplementary explanation from the lecturer A (step S308).

Then, the generation unit 106 generates a digest video by connecting these frame groups (step S310).

Thus, since the gesture pattern for student A to request a supplementary explanation from the lecturer has already been registered, if a situation similar to the above flow occurs in the future, the gesture pattern held in the motion information registration unit 111 will be used. By picking up a specific group of frames from the classroom video through such collation, it is possible to generate a similar digest video without increasing the load associated with arithmetic processing.

<Fourth Embodiment>
The home tutoring system according to the fourth embodiment will be described below with reference to FIGS. 10 and 11. FIG. In this embodiment, the same reference numerals are assigned to the same elements as those described in the above-described first embodiment, and detailed description thereof will be omitted.

In the at-home tutoring system according to the first embodiment, as already described, a specific frame group is extracted from the classroom video by matching using pre-registered waveform data. In the at-home tutoring system according to the embodiment, processing for extracting a specific frame group from the classroom video is performed based on the biometric information of the person.

<Software configuration>
FIG. 10 is a diagram showing a software configuration example of the classroom video distribution apparatus 10 according to this embodiment. The classroom video distribution apparatus 10 includes an extraction unit 101 , a frame extraction unit 105 , a generation unit 106 , a biometric information detection unit 112 , and a biometric information registration unit 113 .

The biometric information detection unit 112 has a function of detecting biometric information of a person for each of a plurality of frames forming the classroom video stored in the memory 15, for example. A person's biological information includes at least one of a person's blood pressure, pulse, and pulse pressure. These predetermined biological information are obtained by extracting the face area of the lecturer or student reflected in each frame by a general face detection technique or the like, and then dividing it into a plurality of areas along the blood flow direction. It can be obtained based on time-series changes of color images showing blood flow.

Such predetermined biometric information is registered in the biometric information registration unit 113 in advance. That is, the biometric information registration unit 113 stores in advance biometric information used for detecting a mental state such as the presence or absence of tension, and a physical state such as the physical condition, for example, by machine learning using a known artificial intelligence technology. there is For example, in this embodiment, student A is unconsciously controlled by changes in facial expression, pupil dilation, pulse rate (pulse rate), facial flushing, sweating, etc. in milliseconds. It is assumed that the biometric information that can read the emotion that is present has already been learned.

When a predetermined specific condition is satisfied, for example, when the biometric information detected by the biometric information detection unit 112 and the biometric information pre-registered in the biometric information registration unit 113 match, the extraction unit 101 A frame group corresponding to the matching biometric information is extracted from the classroom video as a specific frame group.

Next, the operation of the at-home tutoring system configured as described above will be described. FIG. 11 is a flow chart explaining the flow of processing of the classroom video distribution method according to the fourth embodiment of the present disclosure.

Based on the knowledge that human beings have a genetic bias to maintain physical safety, and that things that are unfamiliar or difficult to understand momentarily feel abnormal, A scene in which A is in a tense state is picked up from a classroom video as an example for explanation.

First, when the lecture starts, the students A, B, and C connect the student terminals 20A, 20B, and 20C to the home tutoring system via the network NW, and wait for the instructor T to start the lecture. Lecturer T looks at monitor 14 provided in classroom video distribution apparatus 10 and determines whether or not students A, B, and C are ready to attend, and if they are ready to attend, starts the lecture. That is, the imaging operation by the imaging unit 12 and the imaging unit 22 is started, and the sound pickup operation by the sound pickup unit 13 and the sound pickup unit 23 is started (step S400).

Then, the extracting unit 101 selects and synthesizes the lecturer video acquired by the imaging unit 12 and the student video acquired by the imaging unit 22 as appropriate, and collects The produced sounds are combined to generate a classroom image, and the generated classroom image is distributed to the student terminals 20A, 20B, and 20C via the network NW (step S402).

Next, the biometric information detection unit 112 detects the pulse rate of student A for each of a plurality of frames forming the classroom video using a known technique (step S404).

Next, when the pulse rate detected by the biometric information detection unit 112 matches the pulse rate that has been machine-learned in the biometric information registration unit 113 (the pulse rate at which student A is in a tense state), the extraction unit 101 , the frame group corresponding to the matched pulse rate is extracted from the classroom video as a specific frame group (step S406).

The frame extracting unit 105 extracts a group of frames that are continuous at time intervals of about 10 to 20 seconds, for example, before and after the classroom video in which student A is in a tense state (step S408).

Then, the generation unit 106 generates a digest video by connecting these frame groups (step S410).

Thus, since the pulse rate of student A who is in a tense state has already been registered, if there is a situation similar to the above flow in the future, by matching according to the pulse rate held in the biometric information registration unit 113 , by picking up a specific group of frames from the classroom video, it is possible to generate a similar digest video without increasing the load associated with arithmetic processing.

Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modifications or modifications within the scope of the technical idea described in the claims. are naturally within the technical scope of the present disclosure.

In each of the above-described embodiments, an example in which the information extracting device is applied to the home tutoring system that supports remote classes has been described. However, it is not limited to this. For example, the information extraction device is applied to a system that supports various events such as conferences, lectures, etc. where the start and end times are decided in advance and are mainly held in specific spaces such as classrooms and conference rooms. may

The devices described herein may be implemented as a single device, or may be implemented by a plurality of devices (eg, cloud servers) partially or wholly connected via a network, or the like. For example, the storage 16 or the control unit 18 of the classroom video distribution device 10 may be realized by different servers connected to each other via a network.

The sequence of operations performed by the apparatus described herein may be implemented using software, hardware, or a combination of software and hardware. It is possible to prepare a computer program for realizing each function of the classroom video distribution apparatus 10 according to the present embodiment and to implement it in a PC or the like. A computer-readable recording medium storing such a computer program can also be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Also, the above computer program may be distributed, for example, via a network without using a recording medium.

Also, the processes described with reference to the flowcharts in this specification do not necessarily have to be performed in the illustrated order. Some processing steps may be performed in parallel. Also, additional processing steps may be employed, and some processing steps may be omitted.

Also, the effects described herein are merely illustrative or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.

1 home tutoring system 10 classroom video distribution device (information extraction device)
101 extraction unit 102 waveform registration unit 103 conversion unit 104 display unit 105 frame extraction unit 106 generation unit 107 face detection unit 108 calculation unit 109 face information registration unit 110 identification unit 111 motion information registration unit 112 biometric information detection unit 113 biometric information registration unit network

Claims (15)

an acquisition unit that acquires moving images obtained by imaging each of the lecturer who conducts the remote class and the student who receives the instruction at home;
a storage unit that stores specific conditions for specifying predetermined data included in the moving image;
an extraction unit that extracts a plurality of specific frame groups from the moving image according to the specific condition,
The extraction unit detects the face of the student, calculates an evaluation value based on the specific condition regarding the student in the moving image, and selects only a specific scene regarding the student from the moving image based on the evaluation value. collect and edit to extract a specific group of frames,
An information extraction device,
further comprising a face information registration unit for pre-registering a predetermined face evaluation value relating to facial expressions of the student,
The specific condition is whether or not the face evaluation value calculated from the expression of the student's face included in the moving image matches the registered face evaluation value,
The extracting unit, when the two face evaluation values match, extracts a frame group corresponding to the matching face evaluation value from the moving image as the specific frame group,
The face evaluation value includes an evaluation value that evaluates the degree of happiness, boredom, or tension of the student .
Information extractor. The information extraction device according to claim 1,
further comprising a waveform registration unit for pre-registering predetermined waveform data,
The specific condition is whether or not the waveform data of the sound contained in the moving image matches the registered waveform data,
When both waveform data match, the extraction unit extracts a frame group corresponding to the matched waveform from the moving image as the specific frame group.
Information extractor. The information extraction device according to claim 2,
further comprising a conversion unit that converts the sound contained in the video into text information by speech recognition;
wherein the conversion unit converts the sound corresponding to an auxiliary frame group including the specific frame group and a predetermined number of frames before and after the specific frame group;
Information extractor. The information extraction device according to claim 2 or 3,
Conversational information and non-conversational information are mixed in the sound,
Information extractor. The information extraction device according to claim 4,
The conversation information includes at least one of words indicating positive emotions and words indicating negative emotions.
Information extractor. The information extraction device according to claim 4 or claim 5,
The non-conversation information includes information indicating at least one of tongue clicks, sighs, and backtracking.
Information extractor. The information extraction device according to claim 1,
The face evaluation value includes an evaluation value obtained by evaluating the student 's facial expression, the student 's gaze direction, and the student 's face direction.
information extractor The information extraction device according to claim 1,
further comprising a motion information registration unit for pre-registering a predetermined motion evaluation value relating to the motion of the student ;
The specific condition is whether or not the motion evaluation value calculated from the student included in the moving image matches the registered motion evaluation value,
When both motion evaluation values match, the extracting unit extracts a group of frames corresponding to the matching motion evaluation value from the moving image as the specific frame group.
Information extractor. The information extraction device according to claim 8,
The action evaluation value includes an evaluation value obtained by evaluating at least one action of the student 's gestures, gestures, gestures, and body language.
Information extractor. The information extraction device according to claim 1,
a biometric information registration unit for pre-registering a biometric evaluation value related to predetermined biometric information;
The specific condition is whether or not the biometric evaluation value that can be calculated from the student included in the moving image matches the registered biometric evaluation value,
When both biometric evaluation values match, the extracting unit extracts a frame group corresponding to the matching biometric evaluation value from the moving image as the specific frame group.
Information extractor. The information extraction device according to claim 10,
The biological evaluation value includes at least one of the student 's blood pressure, pulse, and pulse pressure,
Information extractor. The information extraction device according to any one of claims 1 to 11,
a frame addition unit that adds additional frames that are chronologically successive to the specific frame group to the calculation target of the evaluation value,
Information extractor. The information extraction device according to claim 1,
facial expressions of said student Calculate multiple evaluation values evaluated from multiple different viewpoints,
Information extractor. The information extraction device according to claim 1,
calculating the evaluation value together with the audio included in the moving image;
Information extractor. The information extraction device according to any one of claims 1 to 14,
Further comprising means for generating digest information by linking the extracted specific frame groups.
Information extractor.

Images