JP2022075661A - Information extraction apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information extraction apparatus capable of reducing the load due to arithmetic processing.

SOLUTION: In a home individual tutoring system, a classroom video distribution apparatus 10, which is an information extraction apparatus, includes: an extraction unit which extracts a specific frame group in accordance with a predetermined specific condition designated from the outside, from a video image formed of a plurality of frames; an imaging unit which images a predetermined subject and acquires the video image; a sound collection unit which collects ambient sound including the subject; and a waveform registration unit which registers a predetermined waveform in advance. When waveform data of the sound collected by the sound collection unit coincides with the waveform data registered on the waveform registration unit, a frame group corresponding to the above waveform is extracted, as a specific frame group, from the video image.

SELECTED DRAWING: Figure 4

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present disclosure relates to an information extraction device.

In recent years, distribution services that receive distribution of various contents are becoming widespread.

Patent Document 1 discloses a technique that enables a process of detecting a desired music content by inputting a humming song of the music to be searched even when the user does not know the song title or the like of the music content desired to be distributed. Has been done.

Japanese Patent Application Laid-Open No. 2002-55994

By the way, in the technique described in Patent Document 1, since the content to be distributed is limited to music content, a huge amount of arithmetic processing by a computer is required to perform a process of detecting a search target for all other video contents. You will need it.

Therefore, the present disclosure has been made in view of such a situation, and one object of the present invention is to provide an information extraction device capable of reducing the load associated with arithmetic processing.

The main invention of the present invention for solving the above-mentioned problems is characterized by comprising an extraction unit for extracting a specific frame group from a moving image composed of a plurality of frames according to a predetermined specific condition instructed from the outside. do.

According to the present disclosure, the load associated with the arithmetic processing can be reduced.

It is a conceptual diagram which shows the structural example of the home tutoring system 1 which concerns on 1st Embodiment of this disclosure. It is a figure which shows the hardware configuration example of the computer which realizes the classroom video distribution apparatus 10 which concerns on 1st Embodiment of this disclosure. It is a figure which shows the hardware configuration example of the computer which realizes the student terminal 20 which concerns on 1st Embodiment of this disclosure. It is a figure which shows the software configuration example of the classroom video distribution apparatus 10 which concerns on 1st Embodiment of this disclosure. It is a flowchart explaining the process flow of the classroom image distribution method by 1st Embodiment of this disclosure. It is a figure which shows the software configuration example of the classroom video distribution apparatus 10 which concerns on the 2nd Embodiment of this disclosure. It is a flowchart explaining the process flow of the classroom image distribution method by the 2nd Embodiment of this disclosure. It is a figure which shows the software configuration example of the classroom video distribution apparatus 10 which concerns on 3rd Embodiment of this disclosure. It is a flowchart explaining the process flow of the classroom image distribution method by the 3rd Embodiment of this disclosure. It is a figure which shows the software configuration example of the classroom video distribution apparatus 10 which concerns on 4th Embodiment of this disclosure. It is a flowchart explaining the process flow of the classroom image distribution method by 4th Embodiment of this disclosure.

The contents of the embodiments of the present disclosure will be listed and described. The present disclosure comprises the following configurations.
[Item 1]
An acquisition unit that acquires a moving image composed of multiple frames,
A storage unit that stores specific conditions for specifying predetermined data contained in the moving image, and a storage unit.
An extraction unit that extracts a plurality of specific frame groups from the moving image according to the specific conditions, and an extraction unit.
A connecting portion that connects the extracted specific frame groups to each other,
It includes an output unit that outputs digest information including a plurality of concatenated frame groups.
Information extraction device.
[Item 2]
The information extraction device according to item 1.
Further equipped with a waveform registration unit for pre-registering predetermined waveform data,
The specific condition is whether or not the waveform data of the sound included in the moving image and the registered waveform data match.
When both waveform data match, the extraction unit extracts a frame group corresponding to the matched waveform from the moving image as the specific frame group.
Information extraction device.
[Item 3]
The information extraction device according to item 2.
It also has a conversion unit that converts the sound contained in the video into text information by voice recognition.
The conversion unit converts the sound corresponding to the auxiliary frame group including the specific frame group and a predetermined number of frames before and after the specific frame group.
Information extraction device.
[Item 4]
The information extraction device according to item 2 or item 3.
Conversational information and non-conversational information are mixed in the information indicated by the surrounding sounds including the subject.
Information extraction device.
[Item 5]
The information extraction device according to item 4.
The conversation information includes at least one of a word indicating a positive emotion and a word indicating a negative emotion.
Information extraction device.
[Item 6]
The information extraction device according to item 4 or item 5.
The non-conversational information includes information indicating at least one of a tongue-and-groove, a sigh, and an aizuchi.
Information extraction device.
[Item 7]
The information extraction device according to item 1.
Further equipped with a face information registration unit that registers a predetermined facial evaluation value regarding facial expressions in advance.
The specific condition is whether or not the face evaluation value calculated from the facial expression included in the moving image matches the registered face evaluation value.
When both face evaluation values match, the extraction unit extracts a frame group corresponding to the matched face evaluation values from the moving image as the specific frame group.
Information extraction device.
[Item 8]
The information extraction device according to item 7.
The face evaluation value includes an evaluation value for evaluating the degree of happiness, boredom, or tension of the person.
Information extraction device.
[Item 9]
The information extraction device according to item 7 or item 8.
The face evaluation value includes an evaluation value that evaluates the facial expression of the person, the direction of the line of sight of the person, and the direction of the face of the person.
Information extraction device [Item 10]
The information extraction device according to item 1.
Further equipped with an operation information registration unit for pre-registering a predetermined motion evaluation value related to the motion of a person.
The specific condition is whether or not the motion evaluation value calculated from the person included in the moving image and the registered motion evaluation value match.
When both motion evaluation values match, the extraction unit extracts a frame group corresponding to the matched motion evaluation values from the moving image as the specific frame group.
Information extraction device.
[Item 11]
The information extraction device according to item 10.
The motion evaluation value includes an evaluation value that evaluates at least one motion of the person's gesture, hand gesture, gesture, or body language.
Information extraction device.

[Item 12]
The information extraction device according to item 1.
It is equipped with a biometric information registration unit that pre-registers biometric evaluation values related to predetermined biometric information.
The specific condition is whether or not the biometric evaluation value that can be calculated from the person included in the moving image and the registered biometric evaluation value match.
When both bio-evaluation values match, the extraction unit extracts a frame group corresponding to the matched bio-evaluation values from the moving image as the specific frame group.
Information extraction device.
[Item 13]
The information extraction device according to item 12.
The biometric evaluation value includes at least one of the blood pressure, pulse, and pulse pressure of the person.
Information extraction device.
[Item 14]
The information extraction device according to any one of items 1 to 13.
A frame addition unit for adding additional frames that are continuous in time series with the specific frame group is provided for the specific frame group.
Information extraction device.
[Item 15]
An identification means for identifying at least a face image or a sound included in the digest information extracted by the information extraction device according to any one of items 1 to 14 for each predetermined frame unit.
Further provided with an evaluation means for calculating an evaluation value for the identified face image.
Video meeting evaluation terminal.
[Item 16]
The video meeting evaluation terminal according to item 15,
The video meeting evaluation terminal provides graph information in chronological order of the evaluation values.
Video meeting evaluation terminal.
[Item 17]
The video meeting evaluation terminal according to item 15 or item 16.
The video meeting evaluation terminal calculates a plurality of evaluation values obtained by evaluating the face image from a plurality of different viewpoints.
Video meeting evaluation terminal.
[Item 18]
The video meeting evaluation terminal according to any one of items 15 to 17.
The video meeting evaluation terminal calculates the evaluation value together with the sound included in the moving image.
Video meeting evaluation terminal.
[Item 19]
The video meeting evaluation terminal according to any one of items 15 to 18.
The video meeting evaluation terminal calculates the evaluation value together with an object other than the face image contained in the moving image.
Video meeting evaluation terminal.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

In this disclosure, the learning school / educational institution is connected to the student / student's home via a digital communication line, and each student / student can watch the lecture video given at the learning school / educational institution while staying at home. , An example of applying an information extraction device to a home-based tutoring system that allows students to take classes at a cram school / educational institution will be described.

<First Embodiment>
FIG. 1 is a conceptual diagram showing a configuration example of a home-based individual guidance system 1 according to the first embodiment of the present disclosure. As shown in the figure, in this home-based individual guidance system 1, the classroom video distribution device 10 provided on the classroom side of the instructor T who conducts distance lessons and the student group (students A, B, who receive guidance at each home). The student terminals 20A, 20B, and 20C related to C) are connected to each other so as to be communicable via the network NW. In the following, when it is not necessary to distinguish and explain the student terminals 20A, 20B, and 20C, they are simply abbreviated as the student terminals 20. Similarly, when it is not necessary to distinguish and explain students A, B, and C, they are simply abbreviated as students.

The classroom video distribution device 10 is an example of the information extraction device described in the claims. It should be noted that this configuration is an example, and a certain configuration may have another configuration or may include another configuration. Although the case of three students A, B, and C is shown here, the number may be further increased depending on the number of people that the instructor can teach at the same time or the number of connected lines of the network NW.

In this embodiment, the "lecturer" is a concept including a professor, a teacher, and a teacher. "Classroom" refers to the Ministry of Education, Culture, Sports, Science and Technology, including cram schools, culture centers, educational institutions (for example, primary, secondary and higher education institutions, high schools, technical colleges, vocational schools, junior colleges, four-year universities, graduate schools, etc.) It is a concept that includes registered schools). "Student" is a concept that includes students, students, and auditors.

In this embodiment, the network NW assumes the Internet. The network NW is constructed by, for example, a public telephone line network, a mobile phone line network, a wireless communication network, Ethernet (registered trademark), and the like.

<Hardware configuration>
FIG. 2 is a diagram showing a hardware configuration example of a computer that realizes the classroom video distribution device 10 according to the present embodiment. The computer includes at least a communication unit 11, an image pickup unit 12, a sound collection unit 13, a monitor 14, a memory 15, a storage 16, an input / output unit 17, a control unit 18, and the like. These are electrically connected to each other through the bus 19.

The communication unit 11 connects the classroom video distribution device 10 to the network NW. The communication unit 11 may use, for example, a wired LAN (Local Area Network), wireless LAN, Wi-Fi (Wireless Fidelity, registered trademark), infrared communication, Bluetooth (registered trademark), short-range or non-contact communication, or the like. Communicate directly with the device or via a network access point.

The image pickup unit 12 has a function of electronically photographing using an image pickup element such as CMOS or CCD. The image pickup unit 12 takes an image of the instructor T who gives a lecture to the students as a subject, and acquires the instructor image. The image pickup unit 12 may be configured to be able to capture an image described on the blackboard or whiteboard used by the instructor T when proceeding with the lecture, but an independent camera may be provided for the blackboard or whiteboard.

The sound collecting unit 13 collects ambient sounds including the instructor T. The sound collecting unit 13 is provided with a microphone or the like for acquiring ambient sounds including the voice of the instructor T. Further, the sound collecting unit 13 can perform appropriate processing such as converting the acquired sound into an electric signal.

The monitor 14 can display the student image transmitted from the student terminal 20 and the instructor image acquired by the imaging unit 12 in a listable state. Of course, the monitor 14 may display only the student video alone, or may display only the lecturer video alone.

The memory 15 includes a main storage configured by a volatile storage device such as a DRAM (Dynamic Random Access Memory) and an auxiliary storage configured by a non-volatile storage device such as a flash memory or an HDD (Hard Disk Drive). .. The memory 15 is used as a work area or the like of the control unit 18, and also stores a BIOS (Basic Input / Output System) executed when the classroom video distribution device 10 is started, various setting information, and the like.

The storage 16 stores various programs such as application programs. A database storing data used for each process may be built in the storage 16.

The input / output unit 17 is an information input device such as a keyboard, a mouse, and a touch panel.

The control unit 18 is an arithmetic unit that controls the operation of the entire classroom video distribution device 10, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the control unit 18 is a processor such as a CPU (Central Processing Unit), and executes each information processing by executing a program or the like stored in the storage 16 and expanded in the memory 15.

The bus 19 is commonly connected to each of the above elements and transmits, for example, an address signal, a data signal, and various control signals.

FIG. 3 is a diagram showing a hardware configuration example of a computer that realizes the student terminal 20 according to the present embodiment. The computer includes at least a communication unit 21, an image pickup unit 22, a sound collection unit 23, a monitor 24, a memory 25, a storage 26, an input / output unit 27, a control unit 28, and the like. These are electrically connected to each other through the bus 29. Since the hardware configuration of the computer or the like that realizes the student terminal 20 according to the present embodiment is the same as the hardware configuration example of the classroom video distribution device 10 shown in FIG. 2, only the differences will be described.

The communication unit 21 connects the student terminal 20 to the network NW.

The image pickup unit 22 takes an image of the student taking the lecture as a subject and acquires the student image.

The sound collecting unit 23 collects the surrounding sounds including the students. In order to send and receive audio data, the student terminal 20 may be provided with headphones with a microphone, but may also use a microphone and a speaker built in the terminal.

The monitor 24 can display the instructor video transmitted from the classroom video distribution device 10 and the student video acquired by the imaging unit 22 in a listable state. Of course, the monitor 24 may display only the instructor image alone, or may display only the student image alone.

The control unit 28 is an arithmetic unit that controls the operation of the entire student terminal 20, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing.

<Software configuration>
FIG. 4 is a diagram showing a software configuration example of the classroom video distribution device 10 according to the present embodiment. The classroom video distribution device 10 includes an extraction unit 101, a waveform registration unit 102, a conversion unit 103, a display unit 104, a frame cutting unit 105, and a generation unit 106.

The extraction unit 101, the waveform registration unit 102, the conversion unit 103, the display unit 104, the frame cutting unit 105, and the generation unit 106 read the program stored in the storage 16 by the control unit 18 into the memory 15. It can be realized by executing.

The extraction unit 101 appropriately selects and synthesizes the instructor image acquired by the imaging unit 12 and the student image acquired by the imaging unit 22, and also collects sound by the sound collecting unit 13 or the sound collecting unit 23. Generate a classroom video by combining the sounds. The instructor video or student video here is an example of a plurality of frames described in the claims. In addition, the classroom video is an example of a moving image described in the claims. Classroom video is text data, numerical data, graphic data, image data, video data, audio data, etc., or a combination thereof, and is a target for storage, editing, searching, etc., as an individual unit between systems or users. Exchangeable items, including items similar to these.

The extraction unit 101 has a function of extracting a specific frame group from the classroom image according to a predetermined specific condition instructed from the outside. For example, the instruction from the outside may indicate the item of the classroom image to be synthesized, the arrangement of the image, the occupied area of the image, and the like. It is preferable that the instruction from the outside is, for example, a simple operation in which the lecturer T himself can edit the video composition in the middle of the lecture.

The waveform registration unit 102 has a function of pre-registering predetermined waveform data. The predetermined specific condition is, for example, whether or not the waveform data of the sound collected by the sound collecting unit 13 or the sound collecting unit 23 and the waveform data registered in the waveform registration unit 102 match. Well, of course, other conditions may be used. The waveform data registered in the waveform registration unit 102 in the present embodiment can be used to evaluate the atmosphere of the entire place in the lecture.

When a predetermined specific condition is satisfied, for example, when both waveform data match, the extraction unit 101 extracts a frame group corresponding to the matched waveform as a specific frame group from the classroom image.

The conversion unit 103 has a function of converting the sound collected by the sound collecting unit 13 into text by voice recognition. This text is any character string. If the voice recognition is successful, the conversion unit 103 outputs the voice recognition result including the generated text. The speech recognition result may include success information indicating that the speech recognition was successful.

The display unit 104 has a function of displaying the text converted by the conversion unit 103 on the monitor 14 or the monitor 24. Conversational information and non-conversational information are mixed in the information indicated by the surrounding sounds including the instructor T or the students A, B, and C. Conversational information includes, for example, at least one of a word indicating positive emotions and a word indicating negative emotions.

As an example of a word that shows positive emotions, the instructor praises, cheers, and encourages the students, "You did your best," "You worked hard," "Wow," "Great," and "Great." You can mention "ne", "great" and so on.

On the other hand, as an example of a word that shows negative emotions, the instructor may despise, blame, or despise the student, saying, "That's no good," "You're no good," "What are you doing?" "Zo", "stupid" and so on.

Non-conversational information is textual information other than conversational information. Non-conversational information includes information indicating at least one of tongue-and-groove, sighing, and aizuchi. These non-conversational information can be a criterion for judging the emotions of the students who take the lecture. The student's emotions can be classified into, for example, "happiness", "boring", and "tension".

The frame cutting unit 105 has a function of cutting out a frame group that is continuous back and forth at least in time series with respect to a specific frame group extracted by the extraction unit 101. The frame cutting unit 105 can cut out a group of consecutive frames before and after using, for example, an arbitrary frame rate applicable for acquiring context information indicating in what context each word was used. .. The context information here is, for example, information indicating an arbitrary range of character strings before and after a word, a co-occurrence relationship between words, and the like.

The generation unit 106 has a function of connecting a preceding frame group and a succeeding frame that are continuous before and after the specific frame group extracted by the extraction unit 101 to generate a digest moving image.

Next, the operation of the home-based individual guidance system 1 configured in this way will be described. FIG. 5 is a flowchart illustrating a processing flow of the classroom video distribution method according to the first embodiment of the present disclosure.

Here, an example will be given in which the operator of the classroom checks whether the instructor has made inappropriate or problematic remarks to the students. Specifically, a scene in which a digest moving image is generated using waveform data representing "stupid" as a word indicating negative emotions in the voice of a pre-registered instructor will be described as an example.

First, at the time when the lecture starts, each student A, B, C connects the student terminals 20A, 20B, 20C to the home-based individual guidance system 1 via the network NW, and waits for the lecture start of the lecturer T. .. The instructor T looks at the monitor 14 provided in the classroom video distribution device 10 to determine whether or not each student A, B, and C is ready to attend, and if the student is ready to attend, the lecturer T starts the lecture. That is, the image pickup operation by the image pickup unit 12 and the image pickup unit 22 is started, and the sound collection operation by the sound collection unit 13 and the sound collection unit 23 is started (step S100).

Then, the extraction unit 101 appropriately selects and synthesizes the instructor image acquired by the imaging unit 12 and the student image acquired by the imaging unit 22, and collects the sound collecting unit 13 or the sound collecting unit 23. A classroom video is generated by combining the sounds, and the generated classroom video is distributed to the student terminals 20A, 20B, and 20C via the network NW (step S102).

Next, when the instructor utters "stupid", the voice is picked up by the sound collecting unit 13, and the classroom image including the voice data is extracted by the extracting unit 101 as a specific frame group (step S104).

The frame cutting unit 105 cuts out a group of frames connected at time intervals of, for example, about 10 seconds to 20 seconds before and after the classroom image (step S106).

Then, the generation unit 106 generates a digest moving image in which these frame groups are connected, and performs a predetermined acoustic analysis on the voice data of the instructor based on the digest moving image (step S108).

This acoustic analysis gives us an idea of the context in which the word "stupid" was used.

For example, if the instructor uses the word "stupid" in the context of "a person who is saying such stupid things is not good" (to a third party other than the student), the instructor is a student. I can understand that I didn't call it stupid.

Also, for example, if the word "stupid" is used in the instructor in the context of "because I am stupid", it can be understood that the instructor is calling himself stupid.

Thus, the operator who operates the classroom can obtain the word "stupid" and the context before and after it as a text sentence, for example, in the form of a report, so that the instructor makes inappropriate or problematic statements to the students. You can easily check if it is not.

That is, it is characteristic that the instructor is likely to cause inappropriate remarks / problem remarks in the classroom video while reducing the load associated with the arithmetic processing by collation using the waveform data registered in advance. It is possible to pull out the scene pinpoint. For example, when the number of times a word indicating negative emotions is uttered by the instructor reaches a predetermined number of times (for example, 10 times) with respect to the lecture time (for example, 90 minutes), the human nature of the instructor is judged. It will also be possible to do. Of course, the number of times the instructor utters a word that shows positive emotions can also be a factor in judging the instructor's humanity. Furthermore, not only on the side of the instructor, but also on the side of the students, the number of tongue-and-grooves, the number of sighs, and the number of aizuchis for the utterances from the instructor are materials for judging what kind of feelings the students have. Can be.

<Second embodiment>
Hereinafter, the home-based individual guidance system according to the second embodiment will be described with reference to FIGS. 6 and 7. In this embodiment, the same elements as those described in the first embodiment described above are designated by the same reference numerals, and detailed description thereof will be omitted.

In the home-based individual guidance system according to the first embodiment described above, as described above, a specific frame group is extracted from the classroom video by collation using the waveform data registered in advance, but the second In the home-based individual guidance system according to the embodiment, a process of extracting a specific frame group from the classroom video is performed based on the evaluation value regarding the facial expression of the person.

<Software configuration>
FIG. 6 is a diagram showing a software configuration example of the classroom video distribution device 10 according to the present embodiment. The classroom video distribution device 10 includes an extraction unit 101, a frame cutting unit 105, a generation unit 106, a face detection unit 107, a calculation unit 108, and a face information registration unit 109.

The face detection unit 107 has, for example, a function of detecting the face of a person for each of a plurality of frames constituting the classroom image stored in the memory 15. For example, the face detection unit 107 extracts feature points from the classroom image and detects the face area of the instructor T or each student A, B, C, the size of the face area, the face area, and the like.

The characteristic points include, for example, the end points of the eyebrows, eyes, nose, and lips, the contour points of the face, the apex of the head, and the lower end points of the chin. Then, the face detection unit 107 specifies the position information of the face region. For example, the face detection unit 107 calculates the X-coordinates and Y-coordinates of the pixels included in the face region with the horizontal direction of the image as the X-axis and the vertical direction as the Y-axis. Further, the face detection unit 107 can determine the facial expression, age, etc. of the detected instructor or each student by the arithmetic processing using the above-mentioned feature points.

The calculation unit 108 has a function of calculating evaluation values regarding the faces of the instructor T or each student A, B, and C for each of the plurality of frames constituting the classroom image. Each evaluation value calculated by the calculation unit 108 includes predetermined evaluation values (1) to (6) shown below. These evaluation values (1) to (6) are registered in advance in the face information registration unit 109.

(1) The smile degree calculation unit 108 calculates the degree of smile as an evaluation value for each of the faces detected by the face detection unit 107 by using a known technique such as pattern matching. In the present embodiment, as an example of the degree, the degree of smile is shown in four stages of "0: no smile", "1: smile", "2: normal laugh", and "3: big laugh".

(2) The line-of-sight direction calculation unit 108 calculates the line-of-sight direction as an evaluation value for each of the faces detected by the face detection unit 107 using a known technique. In the present embodiment, as an example, the direction of the line of sight is shown by four types of "0: front of the line of sight", "1: left and right direction of the line of sight", "2: right direction of the line of sight", and "3: line of sight cannot be detected".

(3) The face orientation calculation unit 108 calculates the face orientation as an evaluation value for each of the faces detected by the face detection unit 107 using a known technique. In the present embodiment, as an example, the orientation of the face is indicated by four types of "0: front facing the face", "1: left facing the face", "2: right facing the face", and "3: undetectable".

(4) The face area calculation unit 108 calculates the area of the face portion as an evaluation value for each of the faces detected by the face detection unit 107.

(5) The age calculation unit 108 calculates the age of the person as an evaluation value for each of the faces detected by the face detection unit 107 by using a known technique.

(6) The eye-blurring degree calculation unit 108 calculates the eye-blurring degree as an evaluation value for each of the faces detected by the face detection unit 107 by using a known technique. In this embodiment, as an example, the degree of eye contact is determined by four types: "0: no eye contact", "1: partly eye contact", "2: both eye eye contact", and "3: eye contact not detected". show.

These evaluation values (1) to (6) can be used as criteria for judging the emotions of the students who take the lecture. The student's emotions can be classified into, for example, "happiness", "boring", and "tension".

The extraction unit 101 matches when a predetermined specific condition is satisfied, for example, when the evaluation value calculated by the calculation unit 108 and the evaluation value registered in advance in the face information registration unit 109 match. The frame group corresponding to the evaluated evaluation value is extracted from the classroom image as a specific frame group.

Next, the operation of the home-based tutoring system configured in this way will be described. FIG. 7 is a flowchart illustrating a processing flow of the classroom video distribution method according to the second embodiment of the present disclosure.

Here, the operator who manages the classroom responds to the request from the guardian who entrusts the child who is student A to a cram school etc. and wants to grasp the learning attitude and learning situation of student A. Will be described as an example.

First, at the time when the lecture starts, each student A, B, C connects the student terminals 20A, 20B, 20C to the home tutoring system via the network NW, and waits for the lecture start of the lecturer T. The instructor T looks at the monitor 14 provided in the classroom video distribution device 10 to determine whether or not each student A, B, and C is ready to attend, and if the student is ready to attend, the lecturer T starts the lecture. That is, the image pickup operation by the image pickup unit 12 and the image pickup unit 22 is started, and the sound collection operation by the sound collection unit 13 and the sound collection unit 23 is started (step S200).

Then, the extraction unit 101 appropriately selects and synthesizes the instructor image acquired by the imaging unit 12 and the student image acquired by the imaging unit 22, and collects the sound collecting unit 13 or the sound collecting unit 23. A classroom video is generated by combining the sounds, and the generated classroom video is distributed to the student terminals 20A, 20B, and 20C via the network NW (step S202).

Next, the face detection unit 107 detects the face of the student A in the classroom image (step S204). Since the specific method of face detection is the same as the known technique, the description thereof will be omitted.

Then, the calculation unit 108 calculates the face of the student A detected by the face detection unit 107 with the orientation of the face as an evaluation value using a known technique (step S206). Subsequently, in the extraction unit 101, the evaluation values calculated by the calculation unit 108 are the evaluation values "1: face facing left" and "2: face facing right" registered in advance in the face information registration unit 109. When any of them match, the frame group corresponding to the matched evaluation value is extracted from the classroom video as a specific frame group (step S208).

That is, when the face of the student A is facing left or right instead of the front, it can be inferred that the degree of concentration of the student A on the lecture is reduced.

The frame cutting unit 105 cuts out a group of frames connected at time intervals of, for example, about 10 seconds to 20 seconds before and after the classroom image in which the face of the student A is not facing the front (step S210).

Then, the generation unit 106 generates a digest moving image in which these frame groups are connected, and performs a predetermined acoustic analysis on the voice data of the instructor based on the digest moving image (step S212). According to such an acoustic analysis, when the face of the student A is not facing the front, only what the word spoken by the instructor T can be managed by the flag. As a result, the managers of the classrooms can collect words that can be a factor in reducing the concentration of student A's lectures, and parents and others want to understand the learning attitude and learning situation of student A. Can also be responded to efficiently.

Of course, the determination of the degree of concentration of the student A on the lecture can be made not only by the direction of the student A's face but also by the direction of the student A's line of sight. That is, when the line of sight of the student A is directed to the left or right instead of the front, it can be inferred that the degree of concentration of the student A on the lecture is reduced.

The following usage examples can be further considered for the home-based individual guidance system according to this embodiment.

Specifically, a video edited by collecting only scenes in which student A has positive emotions such as satisfaction, happiness, and fulfillment can be generated as a digest video for the parents of student A.

As an example of a specific process capable of generating such a digest moving image, first, the calculation unit 108 evaluates the degree of smile on the face of the student A detected by the face detection unit 107 by using a known technique. Can be calculated as. Subsequently, the extraction unit 101 determines that the degree calculated by the calculation unit 108 matches any of the degrees "2: normal laughter" and "3: big laughter" registered in advance in the face information registration unit 109. The frame group corresponding to the matching evaluation value can be extracted from the classroom video as a specific frame group.

Further, the frame cutting section 105 has a time interval of, for example, about 10 to 20 seconds before and after the classroom image when the degree of smile of student A is either "2: normal laughter" or "3: big laughter". You can cut out a series of frames with. Finally, the generation unit 106 generates a digest video in which these frame groups are connected as a digest video for the guardian or the like of the student A.

<Third embodiment>
Hereinafter, the home-based individual guidance system according to the third embodiment will be described with reference to FIGS. 8 and 9. In this embodiment, the same elements as those described in the first embodiment described above are designated by the same reference numerals, and detailed description thereof will be omitted.

In the home-based individual guidance system according to the first embodiment described above, as described above, a specific frame group is extracted from the classroom video by collation using the waveform data registered in advance, but the third In the home-based individual guidance system according to the embodiment, a process of extracting a specific frame group from the classroom video is performed based on the motion information related to the motion of the person.

<Software configuration>
FIG. 8 is a diagram showing a software configuration example of the classroom video distribution device 10 according to the present embodiment. The classroom video distribution device 10 includes an extraction unit 101, a frame cutting unit 105, a generation unit 106, a specific unit 110, and an operation information registration unit 111.

The specifying unit 110 has, for example, a function of specifying motion information regarding the motion of a person for each of a plurality of frames constituting a classroom image stored in the memory 15. This motion information is, for example, information that captures the motion of a person as a series of a plurality of postures, and may include information on each joint forming the skeleton of the human body corresponding to various postures. The motion information includes, for example, at least one of a person's gesture, hand gesture, gesture, and body language.

For example, the specific unit 110 obtains the coordinates of each joint forming the skeleton of the human body from a plurality of frames constituting the classroom image by pattern matching using the human body pattern. Since the specific method for acquiring coordinates is the same as the known technique, the description thereof will be omitted. Then, the coordinates of each joint represented by this coordinate system can be, for example, skeleton information for one frame. Further, the skeleton information for a plurality of frames can be predetermined operation information.

Such predetermined operation information is registered in advance in the operation information registration unit 111. That is, the motion information registration unit 111 stores motion information corresponding to various postures in advance by machine learning using, for example, a known artificial intelligence technique. For example, in the present embodiment, the gesture corresponding to the movement of the student waving is a machine as a gesture pattern in which the student asks the instructor for a supplementary explanation because the content of the lecture is not convincing or there is a place where the student gets caught. It is assumed that it has been learned.

The extraction unit 101 matches when a predetermined specific condition is satisfied, for example, when the operation information specified by the specific unit 110 and the operation information registered in advance in the operation information registration unit 111 match. The frame group corresponding to the motion information is extracted from the classroom image as a specific frame group.

Next, the operation of the home-based tutoring system configured in this way will be described. FIG. 9 is a flowchart illustrating a processing flow of the classroom video distribution method according to the third embodiment of the present disclosure.

Here, a scene in which a student A who is not satisfied with the content of the lecture picks up a scene including a situation in which a supplementary explanation is requested from the instructor from the classroom video will be described as an example.

First, at the time when the lecture starts, each student A, B, C connects the student terminals 20A, 20B, 20C to the home tutoring system via the network NW, and waits for the lecture start of the lecturer T. The instructor T looks at the monitor 14 provided in the classroom video distribution device 10 to determine whether or not each student A, B, and C is ready to attend, and if the student is ready to attend, the lecturer T starts the lecture. That is, the image pickup operation by the image pickup unit 12 and the image pickup unit 22 is started, and the sound collection operation by the sound collection unit 13 and the sound collection unit 23 is started (step S300).

Then, the extraction unit 101 appropriately selects and synthesizes the instructor image acquired by the imaging unit 12 and the student image acquired by the imaging unit 22, and collects the sound collecting unit 13 or the sound collecting unit 23. A classroom video is generated by combining the sounds, and the generated classroom video is distributed to the student terminals 20A, 20B, and 20C via the network NW (step S302).

Next, the specific unit 110 obtains the coordinates of each joint forming the skeleton of the student A from a plurality of frames constituting the classroom image by pattern matching using the human body pattern. Further, the specifying unit 110 specifies the skeleton information for a plurality of frames as the gesture of the student A based on the coordinates of each joint (step S304).

Subsequently, in the extraction unit 101, when the gesture specified by the specific unit 110 matches the machine-learned gesture pattern (gesture pattern in which the student A asks the instructor for supplementary explanation) in the operation information registration unit 111. In addition, the frame group corresponding to the matching gesture is extracted from the classroom video as a specific frame group (step S306).

The frame cutting unit 105 cuts out a group of frames connected at time intervals of, for example, about 10 seconds to 20 seconds before and after the classroom video requesting the student A instructor for supplementary explanation (step S308).

Then, the generation unit 106 generates a digest moving image in which these frame groups are connected (step S310).

Thus, since the gesture pattern in which student A asks the instructor for supplementary explanation has already been registered, if there is a situation similar to the above flow in the future, the gesture pattern held in the operation information registration unit 111 will be used. By picking up a specific frame group from the classroom video by the matching, it is possible to generate the same digest video without increasing the load associated with the arithmetic processing.

<Fourth Embodiment>
Hereinafter, the home-based individual guidance system according to the fourth embodiment will be described with reference to FIGS. 10 and 11. In this embodiment, the same elements as those described in the first embodiment described above are designated by the same reference numerals, and detailed description thereof will be omitted.

In the home-based tutoring system according to the first embodiment described above, as described above, a specific frame group is extracted from the classroom video by collation using the waveform data registered in advance. In the home-based individual guidance system according to the embodiment, a process of extracting a specific frame group from the classroom video is performed based on the biological information of the person.

<Software configuration>
FIG. 10 is a diagram showing a software configuration example of the classroom video distribution device 10 according to the present embodiment. The classroom video distribution device 10 includes an extraction unit 101, a frame cutting unit 105, a generation unit 106, a biological information detection unit 112, and a biological information registration unit 113.

The biological information detection unit 112 has, for example, a function of detecting the biological information of a person for each of a plurality of frames constituting the classroom image stored in the memory 15. The biological information of a person includes at least one of the person's blood pressure, pulse, and pulse pressure. These predetermined biometric information is obtained by extracting the face area of the instructor or student reflected in each frame by a general face detection technique or the like, and then dividing the face area into a plurality of areas along the blood flow direction in each area. It can be acquired based on the time-series change of the color image showing the blood flow.

Such predetermined biometric information is registered in advance in the biometric information registration unit 113. That is, the biological information registration unit 113 previously stores biological information used for detecting a mental state such as the presence or absence of tension and a physical state such as good or bad physical condition by machine learning using, for example, a known artificial intelligence technique. There is. For example, in the present embodiment, the student A is unconsciously controlled by the change of facial expression in milliseconds, the opening of the pupil, the speed of the pulse (pulse rate), the flushing of the face, the degree of sweating, and the like. It is assumed that the biometric information that can read the emotions that are present has been learned.

The extraction unit 101 satisfies a predetermined specific condition, for example, when the biological information detected by the biological information detection unit 112 and the biological information registered in advance in the biological information registration unit 113 match. The frame group corresponding to the matched biological information is extracted from the classroom image as a specific frame group.

Next, the operation of the home-based tutoring system configured in this way will be described. FIG. 11 is a flowchart illustrating a processing flow of the classroom video distribution method according to the fourth embodiment of the present disclosure.

Here, students are based on the knowledge that human beings have a genetic bias to maintain physical safety, and that they momentarily feel abnormalities for things that are unfamiliar or difficult to understand. An example of a scene in which A is in a tense state is picked up from a classroom image.

First, at the time when the lecture starts, each student A, B, C connects the student terminals 20A, 20B, 20C to the home tutoring system via the network NW, and waits for the lecture start of the lecturer T. The instructor T looks at the monitor 14 provided in the classroom video distribution device 10 to determine whether or not each student A, B, and C is ready to attend, and if the student is ready to attend, the lecturer T starts the lecture. That is, the image pickup operation by the image pickup unit 12 and the image pickup unit 22 is started, and the sound collection operation by the sound collection unit 13 and the sound collection unit 23 is started (step S400).

Then, the extraction unit 101 appropriately selects and synthesizes the instructor image acquired by the imaging unit 12 and the student image acquired by the imaging unit 22, and collects the sound collecting unit 13 or the sound collecting unit 23. A classroom video is generated by combining the sounds, and the generated classroom video is distributed to the student terminals 20A, 20B, and 20C via the network NW (step S402).

Next, the biological information detection unit 112 detects the pulse rate of the student A for each of the plurality of frames constituting the classroom image by using a known technique (step S404).

Subsequently, in the extraction unit 101, when the pulse rate detected by the biological information detection unit 112 matches the pulse rate that has been machine-learned in the biological information registration unit 113 (the pulse rate in which the student A is in a tense state). , The frame group corresponding to the matched pulse rate is extracted from the classroom video as a specific frame group (step S406).

The frame cutting unit 105 cuts out a group of frames connected at time intervals of, for example, about 10 seconds to 20 seconds before and after the classroom image in which the student A is in a tense state (step S408).

Then, the generation unit 106 generates a digest moving image in which these frame groups are connected (step S410).

Thus, since the pulse rate in which the student A is in a tense state has already been registered, if there is a situation similar to the above flow in the future, the collation according to the pulse rate held in the biometric information registration unit 113 will be performed. By picking up a specific frame group from the classroom video, it is possible to generate a similar digest video without increasing the load associated with the arithmetic processing.

Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that anyone with ordinary knowledge in the technical field of the present disclosure may come up with various modifications or modifications within the scope of the technical ideas set forth in the claims. Is, of course, understood to belong to the technical scope of the present disclosure.

In each of the above-described embodiments, an example of applying an information extraction device to a home-based tutoring system that supports distance learning has been described. However, it is not limited to this. For example, the information extraction device is applied to a system that supports various events held in a specific space such as a classroom or a conference room, where the start time and end time are predetermined, such as a conference or a lecture. You may.

The device described in the present specification may be realized as a single device, or may be realized by a plurality of devices (for example, a cloud server) which are partially or wholly connected by a network. For example, the storage 16 or the control unit 18 of the classroom video distribution device 10 may be realized by different servers connected to each other by a network.

The series of processes by the apparatus described herein may be implemented using software, hardware, or any combination of software and hardware. It is possible to create a computer program for realizing each function of the classroom video distribution device 10 according to the present embodiment and implement it on a PC or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed, for example, via a network without using a recording medium.

Further, the processes described with reference to the flowcharts in the present specification may not necessarily be executed in the order shown in the drawings. Some processing steps may be performed in parallel. Further, additional processing steps may be adopted, and some processing steps may be omitted.

In addition, the effects described herein are merely explanatory or exemplary and are not limited. That is, the techniques according to the present disclosure may have other effects apparent to those skilled in the art from the description herein, in addition to or in place of the above effects.

1 Home-based individual guidance system 10 Classroom video distribution device (information extraction device)
101 Extraction unit 102 Waveform registration unit 103 Conversion unit 104 Display unit 105 Frame cutting unit 106 Generation unit 107 Face detection unit 108 Calculation unit 109 Face information registration unit 110 Specific unit 111 Operation information registration unit 112 Biological information detection unit 113 Biological information registration unit NW network

Claims (19)

An acquisition unit that acquires a moving image composed of multiple frames,
A storage unit that stores specific conditions for specifying predetermined data contained in the moving image, and a storage unit.
An extraction unit that extracts a plurality of specific frame groups from the moving image according to the specific conditions, and an extraction unit.
A connecting portion that connects the extracted specific frame groups to each other,
It includes an output unit that outputs digest information including a plurality of concatenated frame groups.
Information extraction device. The information extraction device according to claim 1.
Further equipped with a waveform registration unit for pre-registering predetermined waveform data,
The specific condition is whether or not the waveform data of the sound included in the moving image and the registered waveform data match.
When both waveform data match, the extraction unit extracts a frame group corresponding to the matched waveform from the moving image as the specific frame group.
Information extraction device. The information extraction device according to claim 2.
It also has a conversion unit that converts the sound contained in the video into text information by voice recognition.
The conversion unit converts the sound corresponding to the auxiliary frame group including the specific frame group and a predetermined number of frames before and after the specific frame group.
Information extraction device. The information extraction device according to claim 2 or 3.
Conversational information and non-conversational information are mixed in the information indicated by the surrounding sounds including the subject.
Information extraction device. The information extraction device according to claim 4.
The conversation information includes at least one of a word indicating a positive emotion and a word indicating a negative emotion.
Information extraction device. The information extraction device according to claim 4 or 5.
The non-conversational information includes information indicating at least one of a tongue-and-groove, a sigh, and an aizuchi.
Information extraction device. The information extraction device according to claim 1.
Further equipped with a face information registration unit that registers a predetermined facial evaluation value regarding facial expressions in advance.
The specific condition is whether or not the face evaluation value calculated from the facial expression included in the moving image matches the registered face evaluation value.
When both face evaluation values match, the extraction unit extracts a frame group corresponding to the matched face evaluation values from the moving image as the specific frame group.
Information extraction device. The information extraction device according to claim 7.
The face evaluation value includes an evaluation value for evaluating the degree of happiness, boredom, or tension of the person.
Information extraction device. The information extraction device according to claim 7 or 8.
The face evaluation value includes an evaluation value that evaluates the facial expression of the person, the direction of the line of sight of the person, and the direction of the face of the person.
Information extraction device The information extraction device according to claim 1.
Further equipped with an operation information registration unit for pre-registering a predetermined motion evaluation value related to the motion of a person.
The specific condition is whether or not the motion evaluation value calculated from the person included in the moving image and the registered motion evaluation value match.
When both motion evaluation values match, the extraction unit extracts a frame group corresponding to the matched motion evaluation values from the moving image as the specific frame group.
Information extraction device. The information extraction device according to claim 10.
The motion evaluation value includes an evaluation value that evaluates at least one motion of the person's gesture, hand gesture, gesture, or body language.
Information extraction device.
The information extraction device according to claim 1.
It is equipped with a biometric information registration unit that pre-registers biometric evaluation values related to predetermined biometric information.
The specific condition is whether or not the biometric evaluation value that can be calculated from the person included in the moving image and the registered biometric evaluation value match.
When both bio-evaluation values match, the extraction unit extracts a frame group corresponding to the matched bio-evaluation values from the moving image as the specific frame group.
Information extraction device. The information extraction device according to claim 12.
The biometric evaluation value includes at least one of the blood pressure, pulse, and pulse pressure of the person.
Information extraction device. The information extraction device according to any one of claims 1 to 13.
A frame addition unit for adding additional frames that are continuous in time series with the specific frame group is provided for the specific frame group.
Information extraction device. An identification means for identifying at least a face image or a sound included in the digest information extracted by the information extraction device according to any one of claims 1 to 14 for each predetermined frame unit.
Further provided with an evaluation means for calculating an evaluation value for the identified face image.
Video meeting evaluation terminal. The video meeting evaluation terminal according to claim 15.
The video meeting evaluation terminal provides graph information in chronological order of the evaluation values.
Video meeting evaluation terminal. The video meeting evaluation terminal according to claim 15 or 16.
The video meeting evaluation terminal calculates a plurality of evaluation values obtained by evaluating the face image from a plurality of different viewpoints.
Video meeting evaluation terminal. The video meeting evaluation terminal according to any one of claims 15 to 17.
The video meeting evaluation terminal calculates the evaluation value together with the sound included in the moving image.
Video meeting evaluation terminal. The video meeting evaluation terminal according to any one of claims 15 to 18.
The video meeting evaluation terminal calculates the evaluation value together with an object other than the face image contained in the moving image.
Video meeting evaluation terminal.

Images