WO2020082566A1 - Physiological sign recognition-based distance learning method, device, apparatus, and storage medium - Google Patents
Physiological sign recognition-based distance learning method, device, apparatus, and storage medium Download PDFInfo
- Publication number
- WO2020082566A1 WO2020082566A1 PCT/CN2018/123186 CN2018123186W WO2020082566A1 WO 2020082566 A1 WO2020082566 A1 WO 2020082566A1 CN 2018123186 W CN2018123186 W CN 2018123186W WO 2020082566 A1 WO2020082566 A1 WO 2020082566A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- teaching
- learner
- mixed reality
- facial
- facial feature
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/20—Education
- G06Q50/205—Education administration or guidance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
Definitions
- the present application relates to the field of biometrics technology, in particular to a remote teaching method, device, equipment and storage medium based on biometrics.
- distance teaching also known as online teaching
- online teaching has gradually entered daily life, becoming a common method for people to learn knowledge, effectively helping those who cannot participate in closed teaching environments such as schools due to various restrictions The learner who is learning.
- distance teaching brings convenience to people, it can enable learners to study according to their needs at any time and any place.
- the inventor realized that because the object of distance teaching is more generalized and not targeted, in the specific teaching process, learners cannot experience the authenticity of classroom teaching, nor can they participate in the teaching process, so the teaching effect Poor.
- the main purpose of the present application is to provide a distance learning method, device, equipment and storage medium based on biometrics, aiming to increase the participation of learners in distance learning.
- the present application provides a distance learning method based on biometrics, which includes the following steps:
- the facial expression determine whether the learner is in a preset inefficient learning state
- the present application also proposes a remote teaching device based on biometrics, the device includes:
- Play module used to receive learning instructions triggered by learners and play teaching streaming media
- a collection module configured to collect a video containing the learner's face during the playback of the teaching streaming media
- a determination module for determining the facial expression of the learner in the video
- the judgment module is used to judge whether the learner is in a preset low-efficiency learning state according to the facial expression
- An obtaining module configured to obtain the teaching picture and teaching voice currently played by the teaching streaming media when the learner is in a preset low-efficiency learning state
- the processing module is used for keyword extraction of the teaching speech, determining the object that needs to be mixed reality in the teaching picture according to the extracted keyword, and performing mixed reality processing on the object to obtain the learner's presence
- the mixed reality teaching picture in the context, so that the learner can interact with the objects in the mixed reality teaching picture.
- the present application also proposes a biometrics-based remote teaching device, the device includes: a memory, a processor, and a biometrics-based device stored on the memory and operable on the processor
- the readable instruction of the distance learning based on biometrics is configured to implement the steps of the method of distance learning based on biometrics as described above.
- the present application also proposes a storage medium on which readable instructions for distance learning based on biometrics are stored, and the readable instructions for distance learning based on biometrics are executed by the processor To realize the steps of the remote teaching method based on biometrics as described above.
- the remote teaching method, device, equipment and storage medium based on biometrics in this embodiment collect video containing the learner's face in real time while the learner is watching the teaching streaming media, and confirm the learner's Facial expressions, and then determine whether the learner is in the preset low-efficiency learning state by analyzing the facial expressions.
- FIG. 1 is a schematic structural diagram of a remote teaching device based on biometrics in a hardware operating environment according to an embodiment of the present application
- FIG. 2 is a schematic flowchart of a first embodiment of a remote teaching method based on biometrics identification
- FIG. 3 is a schematic flowchart of a second embodiment of a remote teaching method based on biometrics identification
- FIG. 4 is a structural block diagram of a first embodiment of a remote teaching device based on biometrics of the present application.
- FIG. 1 is a schematic structural diagram of a remote teaching device based on biometrics in a hardware operating environment according to an embodiment of the present application.
- the remote teaching device based on biometrics may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005.
- the communication bus 1002 is used to implement connection communication between these components.
- the user interface 1003 may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.
- the network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface).
- the memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory.
- RAM Random Access Memory
- NVM Non-Volatile Memory
- the memory 1005 may optionally be a storage device independent of the foregoing processor 1001.
- FIG. 1 does not constitute a limitation on the remote teaching device based on biometrics, and may include more or less components than the illustration, or a combination of certain components, or different Parts layout.
- the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and readable instructions for distance learning based on biometrics.
- the network interface 1004 is mainly used for data communication with the long-distance teaching platform, Internet platform, etc .; the user interface 1003 is mainly used for data interaction with the user; this application is based on biometrics
- the processor 1001 in the long-distance teaching device of the remote control device 1001, and the memory 1005 may be provided in a remote teaching device based on biometrics. It can read instructions and execute the remote teaching method based on biometrics provided by the embodiments of the present application.
- FIG. 2 is a schematic flowchart of a first embodiment of a distance learning method based on biometrics.
- the remote teaching method based on biometrics includes the following steps:
- Step S10 Receive a learning instruction triggered by a learner, play teaching streaming media, and collect a video containing the learner's face during the playing of the teaching streaming media.
- the execution subject in this example is a terminal device capable of playing teaching streaming media, such as a learner's personal computer, smart phone, tablet computer, etc., which will not be enumerated here, and there is no restriction on this.
- the terminal device playing the teaching streaming media is the above-mentioned terminal device, in order to ensure that the subsequent mixed reality teaching picture can be presented in front of the learner, the learner needs to wear 3D glasses while watching the teaching streaming media Switching equipment.
- the terminal device that plays the teaching streaming media can directly use the 3D player, so that the learner does not need to wear 3D glasses, and can simply and conveniently use the interactive pen to get the objects in the subsequent mixed reality teaching screen. Interact.
- step S10 is roughly as follows in practical applications:
- the processor in the terminal device can receive the learning instruction triggered by the learner, and then control the terminal device to play according to the learning instruction
- the teaching streaming media at the same time, during the playback process, the camera built in the terminal device or the camera in the room where the learner is located (the external camera and the terminal device establish a communication connection in advance), and the face containing the learner is collected in real time Video.
- Step S20 Determine the facial expression of the learner in the video.
- the operation of determining the facial expression of the learner in the video may be implemented by the following steps:
- the face image of the learner may be identified from the video according to the face detection model obtained in advance. Then, according to the facial feature detection model obtained in advance, the facial feature points of the learner are extracted from the facial image, such as the feature points of the eyes, eyebrows, mouth, jaw and other parts.
- the facial area of the learner's face is divided to obtain a facial feature area corresponding to each facial feature point.
- each facial feature point is located in a facial feature area.
- facial feature points of the same object are located in one facial feature area, for example, all facial feature points identifying the left eyebrow are located in the same facial feature area, and all facial feature points identifying the right eyebrow are located in the same facial feature area.
- the speed vector mentioned here is not only used to indicate the motion speed information of the corresponding facial feature point, but also used to indicate the motion direction information of the facial feature point.
- the method of determining the velocity vector of the facial feature points in each face area based on the optical flow method it may specifically be that by traversing each facial feature area, the facial feature points in the current facial feature area traversed are detected in the adjacent two The intensity of pixel changes between image frames; then, based on the intensity of pixel changes, the velocity vector of facial feature points in the current facial feature area is inferred.
- the spatial position coordinates of the above facial features need to be determined according to the face key point positioning technology, and then the offset amount is determined according to the change of the position coordinates. And through the corresponding sensing device, determine the current video intensity.
- the position coordinate of a certain facial feature point is P (x, y, t)
- the intensity is I (x, y, t)
- ⁇ x, ⁇ y, ⁇ t is moved between two frames.
- x is the abscissa
- y is the ordinate
- t is the optical value.
- I (x, y, t) I (x + ⁇ x, y + ⁇ y, t + ⁇ t);
- V x and V y are the components of x and y, respectively, the velocity or optical flow of I (x, y, t). Therefore, when the distance ⁇ t is between two frames, the optical value t of the above feature point is expressed as a two-dimensional velocity vector
- the characteristic point of the upper lip is marked upward
- the characteristic point of the lower lip and the characteristic point of the upper lip are moved upward, causing the upper lip to lift up, and the lower lip and the upper lip are tightly closed.
- the characteristic point of the inner corner moves toward the heart of the eyebrow, causing the inner corner of the eyebrows to wrinkle together and the eyebrows to be raised. It is generally considered that the learner's facial expression in the video is doubtful.
- Step S30 according to the facial expression, determine whether the learner is in a preset inefficient learning state.
- step S40 After judging that the user has entered an inefficient learning state, first pause the currently playing teaching streaming media, play a light and pleasant music for the learner, or tell a small joke to give the learner a short break. Then, the operations in and after step S40 are performed.
- Step S40 If the learner is in a preset low-efficiency learning state, acquire the teaching picture and teaching voice currently played by the teaching streaming media.
- the learner when it is determined that the learner is currently in a preset inefficient learning state, in order to mobilize the enthusiasm of the learner, so that the learner can recover from the inefficient learning state to the efficient learning state as soon as possible, the The teaching pictures and teaching voices played, so that the current moment can be determined more accurately, and the knowledge points that cause the learner to be confused, so that the mixed reality processing in the subsequent step S50 is specifically for the content that causes the current learner to be confused. In order to provide different learners with the teaching methods they need in the process of distance teaching.
- Step S50 Perform keyword extraction on the teaching voice, determine the object that needs to be mixed reality in the teaching picture according to the extracted keyword, and perform mixed reality processing on the object to obtain an immersive learner Of mixed reality teaching pictures to enable learners to interact with objects in the mixed reality teaching pictures.
- the learner Assuming that the learner's course is the content of contour terrain judgment in the geography course, the learner has extracted the keywords "mountain peak”, “basin depression”, “ridge ridge line”, “valley valley line”, “ “Saddle”, “steep cliff” and other contents are more confusing, and the pictures displayed in the teaching screen are also contour lines. In this state, the learner may not be able to imagine a specific three-dimensional picture in the brain.
- the "mountain peak” is determined as the contour map representing the mountain peak in the teaching picture.
- the learner can see the three-dimensional contour model, and during the viewing process, he can also make preset actions by holding the interactive pen, such as sliding to the left to rotate the contour model to the left. It is convenient for scholars to see the picture of the opposite, so as to better understand the knowledge points currently explained.
- the above-mentioned mixed reality (MR) processing operation on the object may be roughly performed by first digitizing the object to obtain an image matrix corresponding to the object; then, determining the image matrix and The similarity between the image feature matrices corresponding to various types of objects obtained by pre-training; then, according to the preset filtering rules, the image feature matrix whose similarity meets the filtering rules is selected; then, according to the preset mapping relationship table To obtain the rendering model corresponding to the selected image feature matrix and the corresponding introduction information, the mapping relationship table is the correspondence between each image feature matrix and the corresponding rendering model and the corresponding introduction information; then, from the teaching Extract image data in real time from streaming media to determine the real-time position and size of the object in the image data; Finally, according to the real-time position and size of the object in the image data, superimpose on the image data in real time The rendering model and the introduction information obtain the mixed reality teaching picture.
- the frame is the unit Extract image data from the teaching streaming media in real time, so that when determining the real-time position and size of the object in the image data, you can perform image data for each frame according to the feature information of the object Feature detection to determine the real-time position and size of the object in the image data.
- the image feature matrix and the mapping relationship table corresponding to various types of objects used in the above operation can be constructed in advance.
- the set of training sample images includes sample images corresponding to various types of objects and object categories corresponding to each sample image; taking each sample image and the object category corresponding to each sample image as input, for deep learning
- the model performs classification training to obtain image feature matrices corresponding to various types of objects; establishes a correspondence between image feature matrices corresponding to various types of objects, corresponding rendering models and corresponding introduction information, and generates the mapping relationship table.
- the remote teaching method based on biometrics collects video containing the learner's face in real time while the learner is watching the teaching streaming media, and indeed learns in the video with the help of biometric technology
- the facial expression of the learner determines whether the learner is in the preset low-efficiency learning state by analyzing the facial expression.
- a mixed reality teaching picture that can make the learner immersive can be obtained, so that the learner can interact with the objects in the mixed reality teaching picture while watching the teaching streaming media, thereby enhancing the participation of the learner, making Learners can better use distance teaching methods Line self-learning.
- FIG. 3 is a schematic flowchart of a second embodiment of a remote teaching method based on biometrics of the present application.
- the remote teaching method based on biometrics in this embodiment after step S50 may further include:
- Step S60 Find corresponding learning materials according to the extracted keywords, and push the found learning materials to the learner to assist the scholar to understand the knowledge points corresponding to the keywords.
- the above-mentioned operation of searching for corresponding learning materials may be based on the extracted keywords to search for corresponding learning materials in a learning case or a pre-stored learning case in the Internet or a terminal device used by the learner.
- the learning materials are pushed to the learner, which may specifically be sent to the mailbox set by the user, or displayed directly on the user interface of the current terminal device for convenience Learners view and learn.
- a feedback portal may be provided in the user interface of the terminal device to enable the learner to make the content for the teaching (for example, the teaching method of recording teaching streaming media, the teaching courses arranged, etc.) feedback information, so that after the user enters the feedback information and clicks the OK button on the interface, the feedback information is uploaded to the remote teaching service platform, so Not only can the teaching content be adaptively adjusted according to the content fed back by the learner, but also can be used as a reference to evaluate the teaching quality of the teacher based on the content fed back by the learner.
- the teacher can be refined to each different knowledge point, thereby enabling the remote teaching service platform
- the stored teaching streaming media is based on knowledge points and stored separately.
- the learner can first input the keywords of the content that he wants to learn, and then the terminal device sends these keywords to the distance teaching service platform, so that the distance teaching service platform according to the user Provide the keywords of the learning content, find the corresponding knowledge points and combine them to obtain the teaching streaming media that meets the user's requirements.
- the distance learning method based on biometrics provided in this embodiment, when it is determined that the learner is in a preset low-efficiency learning state, searches for the knowledge blind spots that the learner is currently facing based on the extracted keywords Relevant learning materials, and push the found learning materials to learners, realizing immediate prompts to learners, which can help learners eliminate knowledge blind spots as early as possible, and better assist learners to use distance teaching methods for learning.
- the computer-readable instructions may be stored in In a non-volatile computer-readable storage medium, the aforementioned non-volatile readable storage medium may be a read-only memory, a magnetic disk, or an optical disk.
- FIG. 4 is a structural block diagram of a first embodiment of a remote teaching device based on biometrics of the present application.
- the remote teaching device based on biometrics proposed in the embodiment of the present application includes: a playback module 4001, an acquisition module 4002, a determination module 4003, a judgment module 4004, an acquisition module 4005, and a processing module 4006.
- the playing module 4001 is used to receive the learning instruction triggered by the learner and play the teaching streaming media;
- the collecting module 4002 is used to collect the video containing the learner's face during the playing of the teaching streaming media;
- the processing module 4006 is used for keyword extraction of the teaching voice, according to the extracted Keywords identify the objects in the teaching picture that need to be mixed reality, and the mixed reality processing is performed on the objects to obtain a mixed reality teaching picture that can make the learner immersive, so that the learner can interact with the mixed reality The objects in the teaching picture interact.
- the operation of the determination module 4003 to determine the facial expression of the learner in the video may be specifically implemented based on the face recognition technology in biometrics.
- the face sample data stored in the big data platform is trained based on the face feature detection method in face recognition technology to obtain a face feature detection model, Therefore, when determining the facial expression of the learner in the video, the facial feature points of the learner can be extracted from the video according to the facial feature detection model obtained in advance training.
- the facial area of the learner's face may be divided according to each facial feature point to obtain a facial feature area corresponding to each facial feature point.
- each facial feature point is located in a facial feature area.
- facial feature points of the same object are located in one facial feature area, for example, all facial feature points identifying the left eyebrow are located in the same facial feature area, and all facial feature points identifying the right eyebrow are located in the same facial feature area.
- the velocity vectors of the facial feature points in each facial area are determined.
- the speed vector mentioned here is not only used to indicate the motion speed information of the corresponding facial feature point, but also used to indicate the motion direction information of the facial feature point.
- the method of determining the velocity vector of the facial feature points in each face area based on the optical flow method it may specifically be that by traversing each facial feature area, the facial feature points in the current facial feature area traversed are detected in the adjacent two The intensity of pixel changes between image frames; then, based on the intensity of pixel changes, the velocity vector of facial feature points in the current facial feature area is inferred.
- the facial expressions of the learner in the video may be determined based on the obtained velocity vectors of facial feature points.
- the module 4006 performs mixed reality processing on the object to obtain an operation of a mixed reality teaching screen that can enable the learner to be immersed, which will be described in detail below.
- the table is the correspondence between each image feature matrix and the corresponding rendering model and corresponding introduction information; then, image data is extracted from the teaching streaming media in real time to determine the real-time position of the object in the image data and Size; Finally, according to the real-time position and size of the object in the image data, the rendering model and the introduction information are superimposed on the image data in real time to obtain the mixed reality teaching picture.
- the frame is the unit Extract image data from the teaching streaming media in real time, so that when determining the real-time position and size of the object in the image data, you can perform image data for each frame according to the object's feature information Feature detection to determine the real-time position and size of the object in the image data.
- the image feature matrix and the mapping relationship table corresponding to various types of objects used in the above operation can be constructed in advance.
- the set of training sample images includes sample images corresponding to various types of objects and object categories corresponding to each sample image; taking each sample image and the object category corresponding to each sample image as input, for deep learning
- the model performs classification training to obtain image feature matrices corresponding to various types of objects; establishes a correspondence between image feature matrices corresponding to various types of objects, corresponding rendering models and corresponding introduction information, and generates the mapping relationship table.
- the biometrics-based remote teaching device collects videos containing the learner's face in real time while the learner is watching the teaching streaming media, and indeed learns from the video with the help of biometric technology
- the facial expression of the learner determines whether the learner is in the preset low-efficiency learning state by analyzing the facial expression.
- the learner determines whether the learner is in the preset low-efficiency learning state by analyzing the facial expression.
- the learner determines the learner is in the preset low-efficiency learning state, by acquiring the teaching picture currently played by the teaching streaming media And teaching voice, and use keyword extraction technology to extract keywords from the teaching voice, and then determine the objects that need to be mixed reality in the teaching picture according to the extracted keywords, and finally use mixed reality technology to perform mixed reality processing on the determined objects.
- a mixed reality teaching picture that can make the learner immersive can be obtained, so that the learner can interact with the objects in the mixed reality teaching picture while watching the teaching streaming media, thereby enhancing the participation of the learner, making Learners can better use distance teaching methods Line self-learning.
- the second embodiment of the biometrics-based remote teaching device of the present application is proposed.
- the remote teaching device based on biometrics further includes a search module and a push module.
- the search module is used to search corresponding learning materials according to the extracted keywords;
- the push module is used to push the found learning materials to the learners to assist researchers to understand the key Knowledge points corresponding to words.
- the biometrics-based remote teaching device when it is determined that the learner is in a preset low-efficiency learning state, searches for the blind spots with the learner's current knowledge based on the extracted keywords Relevant learning materials, and push the found learning materials to learners, realizing immediate prompts to learners, which can help learners eliminate knowledge blind spots as early as possible, and better assist learners to use distance teaching methods for learning.
Abstract
A physiological sign recognition-based distance learning method, a device, an apparatus, and a storage medium, pertaining to the technical field of physiological sign recognition. The method comprises: receiving a learning instruction triggered by a learner, playing a tutorial streaming media, and acquiring a video containing the face of the learner during playback of the tutorial streaming media (S10); determining a facial expression of the learner (S20); determining, according to the facial expression, whether the learner is in a pre-determined low-effectiveness learning state (S30); if so, acquiring a tutorial frame and tutorial voice data currently being played in the tutorial streaming media (S40); and extracting a keyword from the tutorial voice data, determining, from the tutorial frame, an object requiring mixed reality according to the extracted keyword, and performing mixed reality processing on the determined object to obtain a mixed reality tutorial frame enabling the learner to have immersive experience (S50).
Description
本申请要求于2018年10月25日提交中国专利局、申请号为201811247129.7、发明名称为“基于生物识别的远程教学方法、装置、设备及存储介质”的中国专利申请的优先权,其全部内容通过引用结合在申请中。This application requires the priority of the Chinese patent application submitted to the China Patent Office on October 25, 2018, with the application number 201811247129.7 and the invention titled "Biometric-based remote teaching methods, devices, equipment, and storage media". Incorporated by reference in the application.
本申请涉及生物识别技术领域,尤其涉及一种基于生物识别的远程教学方法、装置、设备及存储介质。The present application relates to the field of biometrics technology, in particular to a remote teaching method, device, equipment and storage medium based on biometrics.
随着互联网技术的发展,远程教学(也称:在线教学)也逐渐走入日常生活,成为人们学习知识的一种常用手段,有效的帮助那些因为种种限制,无法参与到学校等封闭式教学环境中进行学习的学习者。With the development of Internet technology, distance teaching (also known as online teaching) has gradually entered daily life, becoming a common method for people to learn knowledge, effectively helping those who cannot participate in closed teaching environments such as schools due to various restrictions The learner who is learning.
虽然,远程教学给人们带来了便捷性,可以使得学习者在任何时间、任何地点根据自己需要进行学习。但是,发明人意识到,由于远程教学面向的对象比较泛化,没有针对性,因而在具体的教学过程中,学习者不能体验到课堂教学的真实感,也无法参与到教学过程,因而教学效果欠佳。Although, distance teaching brings convenience to people, it can enable learners to study according to their needs at any time and any place. However, the inventor realized that because the object of distance teaching is more generalized and not targeted, in the specific teaching process, learners cannot experience the authenticity of classroom teaching, nor can they participate in the teaching process, so the teaching effect Poor.
所以,亟需提供一种能够提升学习者参与度的远程教学方法。Therefore, there is an urgent need to provide a distance teaching method that can enhance learner participation.
发明内容Summary of the invention
本申请的主要目的在于提供一种基于生物识别的远程教学方法、装置、设备及存储介质,旨在增加远程教学中学习者的参与度。The main purpose of the present application is to provide a distance learning method, device, equipment and storage medium based on biometrics, aiming to increase the participation of learners in distance learning.
为实现上述目的,本申请提供了一种基于生物识别的远程教学方法,所述方法包括以下步骤:In order to achieve the above purpose, the present application provides a distance learning method based on biometrics, which includes the following steps:
接收学习者触发的学习指令,播放教学流媒体,并在播放所述教学流媒体的过程中采集包含所述学习者的人脸的视频;Receiving a learning instruction triggered by a learner, playing teaching streaming media, and collecting videos containing the learner's face during the playing of the teaching streaming media;
确定所述视频中所述学习者的面部表情;Determine the facial expression of the learner in the video;
根据所述面部表情,判断所述学习者是否处于预设的低效学习状态;According to the facial expression, determine whether the learner is in a preset inefficient learning state;
若所述学习者处于预设的低效学习状态,则获取所述教学流媒体当前播放的教学画面和教学语音;If the learner is in a preset low-efficiency learning state, acquire the teaching picture and teaching voice currently played by the teaching streaming media;
对所述教学语音进行关键词提取,根据提取到的关键词确定所述教学画面中需要进行混合现实的对象,对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面,以使学习者能够与所述混合现实教学画面中的对象进行互动。Perform keyword extraction on the teaching speech, determine the objects that need to be mixed reality in the teaching picture according to the extracted keywords, and perform mixed reality processing on the objects to obtain mixed reality that can make the learner immersive Teaching pictures to enable learners to interact with objects in the mixed reality teaching pictures.
此外,为实现上述目的,本申请还提出一种基于生物识别的远程教学装置,所述装置包括:In addition, in order to achieve the above object, the present application also proposes a remote teaching device based on biometrics, the device includes:
播放模块,用于接收学习者触发的学习指令,播放教学流媒体;Play module, used to receive learning instructions triggered by learners and play teaching streaming media;
采集模块,用于在播放所述教学流媒体的过程中采集包含所述学习者的人脸的视频;A collection module, configured to collect a video containing the learner's face during the playback of the teaching streaming media;
确定模块,用于确定所述视频中所述学习者的面部表情;A determination module for determining the facial expression of the learner in the video;
判断模块,用于根据所述面部表情,判断所述学习者是否处于预设的低效学习状态;The judgment module is used to judge whether the learner is in a preset low-efficiency learning state according to the facial expression;
获取模块,用于在所述学习者处于预设的低效学习状态时,获取所述教学流媒体当前播放的教学画面和教学语音;An obtaining module, configured to obtain the teaching picture and teaching voice currently played by the teaching streaming media when the learner is in a preset low-efficiency learning state;
处理模块,用于对所述教学语音进行关键词提取,根据提取到的关键词确定所述教学画面中需要进行混合现实的对象,对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面,以使学习者能够与所述混合现实教学画面中的对象进行互动。The processing module is used for keyword extraction of the teaching speech, determining the object that needs to be mixed reality in the teaching picture according to the extracted keyword, and performing mixed reality processing on the object to obtain the learner's presence The mixed reality teaching picture in the context, so that the learner can interact with the objects in the mixed reality teaching picture.
此外,为实现上述目的,本申请还提出一种基于生物识别的远程教学设备,所述设备包括:存储器、处理器及存储在所述存储器上并可在所述处理器上运行的基于生物识别的远程教学的可读指令,所述基于生物识别的远程教学的可读指令配置为实现如上文所述的基于生物识别的远程教学方法的步骤。In addition, in order to achieve the above object, the present application also proposes a biometrics-based remote teaching device, the device includes: a memory, a processor, and a biometrics-based device stored on the memory and operable on the processor The readable instruction of the distance learning based on biometrics, the readable instruction of the distance learning based on biometrics is configured to implement the steps of the method of distance learning based on biometrics as described above.
此外,为实现上述目的,本申请还提出一种存储介质,所述存储介质上存储有基于生物识别的远程教学的可读指令,所述基于生物识别的远程教学的可读指令被处理器执行时实现如上文所述的基于生物识别的远程教学方法的步骤。In addition, in order to achieve the above object, the present application also proposes a storage medium on which readable instructions for distance learning based on biometrics are stored, and the readable instructions for distance learning based on biometrics are executed by the processor To realize the steps of the remote teaching method based on biometrics as described above.
本实施例的基于生物识别的远程教学方法、装置、设备及存储介质,通过在学习者观看教学流媒体的过程中实时采集包含学习者人脸的视频,借助生物识别技术确实视频中学习者的面部表情,然后通过对面部表情的分析来判断学习者是否处于预设的低效学习状态,当判定学习者处于预设的低效学习状态是,通过获取教学流媒体当前播放的教学画面和教学语音,并利用关键词提取技术从教学语音中提取关键词,然后根据提取到的关键词确定教学画面中需要进行混合现实的对象,最后借助混合现实技术对确定的对象进行混合现实处理,从而可以得到能够让学习者身临其境的混合现实教学画面,使得学习者在观看教学流媒体的过程中能够与混合现实教学画面中的对象进行互动,从而提升了学习者的参与度,使得学习者能够更好的通过远程教学方式进行自主学习。The remote teaching method, device, equipment and storage medium based on biometrics in this embodiment collect video containing the learner's face in real time while the learner is watching the teaching streaming media, and confirm the learner's Facial expressions, and then determine whether the learner is in the preset low-efficiency learning state by analyzing the facial expressions. When it is determined that the learner is in the preset low-efficiency learning state, by acquiring the teaching picture and teaching currently played by the teaching streaming media Speech, and use keyword extraction technology to extract keywords from teaching speech, then determine the objects in the teaching picture that need to be mixed reality according to the extracted keywords, and finally use mixed reality technology to perform mixed reality processing on the determined objects, so that Obtain a mixed reality teaching picture that can make the learner immersive, so that the learner can interact with the objects in the mixed reality teaching picture while watching the teaching streaming media, thereby enhancing the learner's participation and making the learner Can better self-help through distance teaching Learn.
图1是本申请实施例方案涉及的硬件运行环境的基于生物识别的远程教学设备的结构示意图;1 is a schematic structural diagram of a remote teaching device based on biometrics in a hardware operating environment according to an embodiment of the present application;
图2为本申请基于生物识别的远程教学方法第一实施例的流程示意图;FIG. 2 is a schematic flowchart of a first embodiment of a remote teaching method based on biometrics identification;
图3为本申请基于生物识别的远程教学方法第二实施例的流程示意图;FIG. 3 is a schematic flowchart of a second embodiment of a remote teaching method based on biometrics identification;
图4为本申请基于生物识别的远程教学装置第一实施例的结构框图。FIG. 4 is a structural block diagram of a first embodiment of a remote teaching device based on biometrics of the present application.
本申请目的的实现、功能特点及优点将结合实施例,参照附图做进一步说明。The implementation, functional characteristics and advantages of the present application will be further described in conjunction with the embodiments and with reference to the drawings.
应当理解,此处所描述的具体实施例仅用以解释本申请,并不用于限定本申请。It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.
参照图1,图1为本申请实施例方案涉及的硬件运行环境的基于生物识别的远程教学设备结构示意图。Referring to FIG. 1, FIG. 1 is a schematic structural diagram of a remote teaching device based on biometrics in a hardware operating environment according to an embodiment of the present application.
如图1所示,该基于生物识别的远程教学设备可以包括:处理器 1001,例如中央处理器(Central Processing Unit,CPU),通信总线1002、用户接口1003,网络接口1004,存储器1005。其中,通信总线1002用于实现这些组件之间的连接通信。用户接口1003可以包括显示屏(Display)、输入单元比如键盘(Keyboard),可选用户接口1003还可以包括标准的有线接口、无线接口。网络接口1004可选的可以包括标准的有线接口、无线接口(如无线保真(WIreless-FIdelity,WI-FI)接口)。存储器1005可以是高速的随机存取存储器(Random Access Memory,RAM)存储器,也可以是稳定的非易失性存储器(Non-Volatile Memory,NVM),例如磁盘存储器。存储器1005可选的还可以是独立于前述处理器1001的存储装置。As shown in FIG. 1, the remote teaching device based on biometrics may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to implement connection communication between these components. The user interface 1003 may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may optionally be a storage device independent of the foregoing processor 1001.
本领域技术人员可以理解,图1中示出的结构并不构成对基于生物识别的远程教学设备的限定,可以包括比图示更多或更少的部件,或者组合某些部件,或者不同的部件布置。Those skilled in the art may understand that the structure shown in FIG. 1 does not constitute a limitation on the remote teaching device based on biometrics, and may include more or less components than the illustration, or a combination of certain components, or different Parts layout.
如图1所示,作为一种存储介质的存储器1005中可以包括操作系统、网络通信模块、用户接口模块以及基于生物识别的远程教学的可读指令。As shown in FIG. 1, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and readable instructions for distance learning based on biometrics.
在图1所示的基于生物识别的远程教学设备中,网络接口1004主要用于与远程教学平台、互联网平台等进行数据通信;用户接口1003主要用于与用户进行数据交互;本申请基于生物识别的远程教学设备中的处理器1001、存储器1005可以设置在基于生物识别的远程教学设备中,所述基于生物识别的远程教学设备通过处理器1001调用存储器1005中存储的基于生物识别的远程教学的可读指令,并执行本申请实施例提供的基于生物识别的远程教学方法。In the long-distance teaching device based on biometrics shown in FIG. 1, the network interface 1004 is mainly used for data communication with the long-distance teaching platform, Internet platform, etc .; the user interface 1003 is mainly used for data interaction with the user; this application is based on biometrics The processor 1001 in the long-distance teaching device of the remote control device 1001, and the memory 1005 may be provided in a remote teaching device based on biometrics. It can read instructions and execute the remote teaching method based on biometrics provided by the embodiments of the present application.
本申请实施例提供了一种基于生物识别的远程教学方法,参照图2,图2为本申请一种基于生物识别的远程教学方法第一实施例的流程示意图。An embodiment of the present application provides a distance learning method based on biometrics. Referring to FIG. 2, FIG. 2 is a schematic flowchart of a first embodiment of a distance learning method based on biometrics.
本实施例中,所述基于生物识别的远程教学方法包括以下步骤:In this embodiment, the remote teaching method based on biometrics includes the following steps:
步骤S10,接收学习者触发的学习指令,播放教学流媒体,并在播放所述教学流媒体的过程中采集包含所述学习者的人脸的视频。Step S10: Receive a learning instruction triggered by a learner, play teaching streaming media, and collect a video containing the learner's face during the playing of the teaching streaming media.
具体的说,本实例中的执行主体为能够播放教学流媒体的终端设备,比如学习者的个人计算机、智能手机、平板电脑等,此处不再一 一列举,对此也不做任何限制。Specifically, the execution subject in this example is a terminal device capable of playing teaching streaming media, such as a learner's personal computer, smart phone, tablet computer, etc., which will not be enumerated here, and there is no restriction on this.
应当理解的是,在播放教学流媒体的终端设备为上述终端设备时,为了保证后续得到混合现实教学画面能够在学习者面前呈现,学习者在观看教学流媒体的过程中,需要佩戴3D眼镜等交换设备。It should be understood that when the terminal device playing the teaching streaming media is the above-mentioned terminal device, in order to ensure that the subsequent mixed reality teaching picture can be presented in front of the learner, the learner needs to wear 3D glasses while watching the teaching streaming media Switching equipment.
此外,为了方便用户观看,播放教学流媒体的终端设备可以直接选用3D播放器,这样学习者无需佩戴3D眼镜,只需通过交互笔就可以轻松方便的与后续得到的混合现实教学画面中的对象进行互动。In addition, in order to facilitate the user to watch, the terminal device that plays the teaching streaming media can directly use the 3D player, so that the learner does not need to wear 3D glasses, and can simply and conveniently use the interactive pen to get the objects in the subsequent mixed reality teaching screen. Interact.
此外,关于步骤S10中的操作,在实际应用中大致如下:In addition, the operation in step S10 is roughly as follows in practical applications:
比如,学习者需要使用自己的终端设备观看教学流媒体时,先按下了播放按键,这样终端设备内的处理器便可以接收到学习者触发的学习指令,然后根据该学习指令控制终端设备播放所述教学流媒体,同时在播放的过程中开启用终端设备内置的摄像头或学习者所处房间的摄像头(外置摄像头与终端设备预先建立通信连接),实时采集包含所述学习者的人脸的视频。For example, when a learner needs to use his own terminal device to watch teaching streaming media, he first presses the play button, so that the processor in the terminal device can receive the learning instruction triggered by the learner, and then control the terminal device to play according to the learning instruction The teaching streaming media, at the same time, during the playback process, the camera built in the terminal device or the camera in the room where the learner is located (the external camera and the terminal device establish a communication connection in advance), and the face containing the learner is collected in real time Video.
需要说明的是,以上仅为举例说明,对本申请的技术方案并不构成限定,在具体实现中,本领域的技术人员可以根据需要设置上述操作逻辑,此处不做限制。It should be noted that the above is only an example, and does not constitute a limitation on the technical solution of the present application. In a specific implementation, a person skilled in the art may set the above operation logic as needed, without limitation here.
步骤S20,确定所述视频中所述学习者的面部表情。Step S20: Determine the facial expression of the learner in the video.
具体的说,在实际应用中,确定所述视频中学习者的面部表情的操作,可以通过如下步骤实现:Specifically, in practical applications, the operation of determining the facial expression of the learner in the video may be implemented by the following steps:
(1)根据预先训练获得的人脸特征检测模型,从所述视频中提取所述学习者的面部特征点。(1) Extract the learner's facial feature points from the video according to the facial feature detection model obtained in advance training.
应当理解的是,为了保证减少不必要的干扰,可以先根据预先训练获得的人脸检测模型,从所述视频中识别出所述学习者的人脸图像。然后再根据预先训练获得的人脸特征检测模型,从所述人脸图像中提取出所述学习者的面部特征点,比如眼睛、眉毛、嘴巴、下颌等部位的特征点。It should be understood that, in order to ensure that unnecessary interference is reduced, the face image of the learner may be identified from the video according to the face detection model obtained in advance. Then, according to the facial feature detection model obtained in advance, the facial feature points of the learner are extracted from the facial image, such as the feature points of the eyes, eyebrows, mouth, jaw and other parts.
(2)根据各面部特征点,对所述学习者的人脸进行面部区域划分,得到与各面部特征点对应的面部特征区域。(2) According to each facial feature point, the facial area of the learner's face is divided to obtain a facial feature area corresponding to each facial feature point.
比如,规定划分后的面部特征区域中有且仅有一个面部特征点,即每一个面部特征点位于一个面特征区域。For example, there is only one facial feature point in the divided facial feature area, that is, each facial feature point is located in a facial feature area.
或者,规定同一对象的几个面部特征点位于一个面部特征区域,如标识左侧眉毛的所有面部特征点位于同一个面部特征区域,标识右侧眉毛的所有面部特征点位于同一个面部特征区域。Or, it is stipulated that several facial feature points of the same object are located in one facial feature area, for example, all facial feature points identifying the left eyebrow are located in the same facial feature area, and all facial feature points identifying the right eyebrow are located in the same facial feature area.
需要说明的是,以上仅为举例说明,对本申请的技术方案并不构成限定,在具体实现中,本领域的技术人员可以根据需要对人脸进行面部区域划分,此处不做限制。It should be noted that the above is only an example, and does not constitute a limitation on the technical solution of the present application. In a specific implementation, a person skilled in the art can divide the facial area of a human face as needed, and no limitation is made here.
(3)基于光流法,确定各面部区域中的面部特征点的速度向量。(3) Based on the optical flow method, the velocity vectors of the facial feature points in each facial area are determined.
需要说明的是,此处所说的速度向量,不仅仅用于表示对应的面部特征点的运动速度信息,还用于表示该面部特征点的运动方向信息。It should be noted that the speed vector mentioned here is not only used to indicate the motion speed information of the corresponding facial feature point, but also used to indicate the motion direction information of the facial feature point.
此外,关于基于光流法,确定各面部区域中的面部特征点的速度向量的方式,具体可以是通过遍历各面部特征区域,检测遍历到的当前面部特征区域中的面部特征点在相邻两个图像帧之间的像素变化强度;然后根据所述像素变化强度,推断所述当前面部特征区域中的面部特征点的速度向量。In addition, regarding the method of determining the velocity vector of the facial feature points in each face area based on the optical flow method, it may specifically be that by traversing each facial feature area, the facial feature points in the current facial feature area traversed are detected in the adjacent two The intensity of pixel changes between image frames; then, based on the intensity of pixel changes, the velocity vector of facial feature points in the current facial feature area is inferred.
此外,值得一提的是,在计算速度向量时,还需要根据人脸关键点定位技术,确定上述各面部特征的空间位置坐标,然后根据位置坐标的变化确定偏移量。并通过相应的传感设备,确定当前视频的强度。In addition, it is worth mentioning that, when calculating the velocity vector, the spatial position coordinates of the above facial features need to be determined according to the face key point positioning technology, and then the offset amount is determined according to the change of the position coordinates. And through the corresponding sensing device, determine the current video intensity.
为了便于理解,以下进行具体说明。For ease of understanding, specific explanations are provided below.
假设,某一面部特征点的位置坐标为P(x,y,t),强度为I(x,y,t),在两帧之间移动了Δx,Δy,Δt。其中,x为横坐标,y为纵坐标,t为光学量值,则根据亮度恒定约束条件,有:Suppose that the position coordinate of a certain facial feature point is P (x, y, t), the intensity is I (x, y, t), and Δx, Δy, Δt is moved between two frames. Among them, x is the abscissa, y is the ordinate, and t is the optical value. According to the constant constraint of brightness, there are:
公式(1):I(x,y,t)=I(x+Δx,y+Δy,t+Δt);Formula (1): I (x, y, t) = I (x + Δx, y + Δy, t + Δt);
假设移动很小,I(x,y,t)的图像约束可以用泰勒级数来获得:Assuming that the movement is small, the image constraints of I (x, y, t) can be obtained using Taylor series:
其中,τ一个高阶无穷小的。因而,通过对公式(1)和公式(2)进行整理,可以获得:Among them, τ is a high-order infinitesimal. Therefore, by formulating formula (1) and formula (2), we can obtain:
通过对公式(3)和公式(4)进行整理,可以获得:By sorting out formula (3) and formula (4), we can obtain:
其中,V
x和V
y分别是x和y的分量,I(x,y,t)的速度或光流。因此,在距离Δt是两帧之间,上述特征点的光学量值t被表示为一个二维的速度向量
Where V x and V y are the components of x and y, respectively, the velocity or optical flow of I (x, y, t). Therefore, when the distance Δt is between two frames, the optical value t of the above feature point is expressed as a two-dimensional velocity vector
此外,未在本实施例中介绍的内容,可以通过查找光流法的相关资料实现,此处不再赘述。In addition, the content that is not introduced in this embodiment can be realized by searching for relevant materials of the optical flow method, which will not be repeated here.
(4)根据各面部特征点的速度向量,确定所述视频中所述学习者的面部表情。(4) Determine the facial expression of the learner in the video based on the velocity vector of each facial feature point.
比如,在标识眼内角的上眼皮的特征点向下运动,导致眼内角的上眼皮降低,标识嘴巴的特征点向外运动,导致出嘴巴张大时,通常可以认为所述视频中的学习者的面部表情为困倦。For example, when the feature point of the upper eyelid that marks the inner corner of the eye moves downward, which causes the upper eyelid of the inner eye to lower, and the feature point that marks the mouth moves outward, causing the mouth to open wide, it can usually be considered that the learner ’s Facial expression is sleepy.
还比如,在标识上唇的特征点向上运动,标识下唇的特征点跟谁上唇的特征点向上运动,导致上唇抬起,且下唇与上唇紧闭,嘴角下端,唇轻微凸起;标识眉毛内角的特征点向眉心运动,导致眉毛内角皱在一起,且眉毛抬高是,通常认为所述视频中的学习者的面部表情为疑惑。For another example, when the characteristic point of the upper lip is marked upward, the characteristic point of the lower lip and the characteristic point of the upper lip are moved upward, causing the upper lip to lift up, and the lower lip and the upper lip are tightly closed. The characteristic point of the inner corner moves toward the heart of the eyebrow, causing the inner corner of the eyebrows to wrinkle together and the eyebrows to be raised. It is generally considered that the learner's facial expression in the video is doubtful.
还比如,在标识唇角的特征点向后脸颊后上方运动,导致唇角向后拉并抬高,标识嘴巴的特征点向外运动,导致出嘴巴张大时,通常可以认为所述视频中的学习者的面部表情为满意。For another example, when the feature point that marks the corner of the lip moves backward and upward on the cheek, causing the corner of the lips to be pulled back and raised, and the feature point that marks the mouth moves outward, causing the mouth to open wide. The learner's facial expression is satisfied.
需要说明的是,以上仅为举例说明,在具体实现中,本领域的技术人员可以结合微表情的变化特征与获得的各面部特征点的速度向量,确定所述视频中所述学习者的面部表情,此处不再赘述,对此也不做限制。It should be noted that the above is only an example. In a specific implementation, a person skilled in the art can combine the changing features of the micro-expression and the obtained velocity vector of each facial feature point to determine the learner's face in the video Emoticons will not be repeated here, and there is no limit to this.
步骤S30,根据所述面部表情,判断所述学习者是否处于预设的低效学习状态。Step S30, according to the facial expression, determine whether the learner is in a preset inefficient learning state.
具体的说,通过研究表明,在学习者做出困倦、疑惑的表情和动作时,学习者当前的学习效率较低,即处于预设的低效学习状态。因而,在根据面部表情判断学习者是否处于预设的低效学习状态时,只需判断学习者是否做出了困倦、疑惑的面部表情即可。Specifically, studies have shown that when learners make sleepy and puzzled expressions and movements, the learners' current learning efficiency is low, that is, they are in a preset state of inefficient learning. Therefore, when determining whether the learner is in a preset inefficient learning state based on the facial expression, it is only necessary to determine whether the learner has made a sleepy and doubtful facial expression.
需要说明的是,以上给出的仅为两种用于判定为低效学习状态的具体面部表情和肢体动作,在实际应用中,可以根据微表情学和相关的肢体语言学进行设置,此处不做限制。It should be noted that the above are only two specific facial expressions and body movements used to determine the inefficient learning state. In practical applications, they can be set according to micro-expression and related body linguistics. Here No restrictions.
此外,值得一提的是,由于在实际的教学过程中,学习者在长时间的听课过程中往往难以始终保持最佳的学习状态。一般情况下,大脑会经历从能够高效学习的高效期,再到思考,学习状态差的低效期。在低效学习状态下,学习者往往难以很好的理解教学流媒体中播放的教学内容,因而在这种状态下,很容易产生知识“盲点”(即对学习者来说,看不透、想不准、理不清的知识点)。因此,为了提升远程教学质量,就需要尽可能的吸引学习者的注意力,以使学习者的大脑能够长时间的保持高效学习状态,或者能够尽快从低效学习状态恢复到高效学习状态,可以在判定用户进入低效学习状态后,先暂停当前播放的教学流媒体,为学习者播放一段轻松愉悦的音乐,或者讲一个小笑话,让学习者稍作片刻休息。然后在执行步骤S40及之后的操作。In addition, it is worth mentioning that, in the actual teaching process, learners often have difficulty in maintaining the best learning state at all times during the long course of listening. In general, the brain will go from an efficient period of efficient learning to an inefficient period of thinking and poor learning. In the state of inefficient learning, learners often have difficulty in understanding the teaching content played in the teaching streaming media, so in this state, it is easy to produce knowledge "blind spots" (that is, for learners, it is impossible to see through, Inexplicable and unclear knowledge points). Therefore, in order to improve the quality of distance learning, it is necessary to attract the learner's attention as much as possible, so that the learner's brain can maintain an efficient learning state for a long time, or can recover from an inefficient learning state to an efficient learning state as soon as possible. After judging that the user has entered an inefficient learning state, first pause the currently playing teaching streaming media, play a light and pleasant music for the learner, or tell a small joke to give the learner a short break. Then, the operations in and after step S40 are performed.
步骤S40,若所述学习者处于预设的低效学习状态,则获取所述教学流媒体当前播放的教学画面和教学语音。Step S40: If the learner is in a preset low-efficiency learning state, acquire the teaching picture and teaching voice currently played by the teaching streaming media.
具体的说,在判定学习者当前时刻处于预设的低效学习状态时,为了调动学习者的积极性,以使学习者尽快从低效学习状态恢复到高效学习状态,本案通过获取教学流媒体当前播放的教学画面和教学语音,从而可以较为精准的确定当前时刻,导致学习者产生困惑的知识点,使得后续步骤S50中进行的混合现实处理,是专门针对导致当前学习者产生困惑的内容进行处理,从而在远程教学的过程中实现了为不同学习者,提供各自需要的教学方式。Specifically, when it is determined that the learner is currently in a preset inefficient learning state, in order to mobilize the enthusiasm of the learner, so that the learner can recover from the inefficient learning state to the efficient learning state as soon as possible, the The teaching pictures and teaching voices played, so that the current moment can be determined more accurately, and the knowledge points that cause the learner to be confused, so that the mixed reality processing in the subsequent step S50 is specifically for the content that causes the current learner to be confused. In order to provide different learners with the teaching methods they need in the process of distance teaching.
步骤S50,对所述教学语音进行关键词提取,根据提取到的关键词确定所述教学画面中需要进行混合现实的对象,对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面,以使学习者能够与所述混合现实教学画面中的对象进行互动。Step S50: Perform keyword extraction on the teaching voice, determine the object that needs to be mixed reality in the teaching picture according to the extracted keyword, and perform mixed reality processing on the object to obtain an immersive learner Of mixed reality teaching pictures to enable learners to interact with objects in the mixed reality teaching pictures.
具体的说,在从教学语音中提取关键词时,需要先将教学语音转换为文本格式,然后在对得到的文本内容进行关键词提取。Specifically, when extracting keywords from teaching speech, it is necessary to convert the teaching speech into text format first, and then extract keywords from the obtained text content.
为了便于理解,上述所说的根据提取到的关键词确定所述教学画面中需要进行混合现实的对象,以下进行举例说明。For ease of understanding, the above-mentioned determination of objects in the teaching screen that need to be mixed reality is based on the extracted keywords, which will be described below by way of example.
假设学习者学习的课程为地理课程中关于等高线地形判断的内容,学习者对于提取到的关键词“山地山峰”、“盆地洼地”、“山脊山脊线”、“山谷山谷线”、“鞍部”、“陡崖”等内容比较困惑,并且在教学画面中显示的图也均为等高线样式的。这种状态下,学习者可能在大脑中无法想象出具体的立体画面。Assuming that the learner's course is the content of contour terrain judgment in the geography course, the learner has extracted the keywords "mountain peak", "basin depression", "ridge ridge line", "valley valley line", " "Saddle", "steep cliff" and other contents are more confusing, and the pictures displayed in the teaching screen are also contour lines. In this state, the learner may not be able to imagine a specific three-dimensional picture in the brain.
这个时候,便可以根据提取到的关键词来确定教学画面中需要进行混合现实的对象。At this time, you can determine the objects that need to be mixed reality in the teaching picture based on the extracted keywords.
比如,根据“山地山峰”确定的为教学画面中表示山地山峰的等高线图。这时,就可以对山地山峰的等高线图进行虚拟现实处理和增强现实处理,在山地山峰等高线的平面图形上展示出立体的山体图形(虚拟现实),同时在对应的位置,展示出这种地形特征的描述信息及习惯使用的表述方法。这样,学习者便可以看到立体的等高线模型,并且在观看过程中还可以通过手持交互笔做出预设的动作,比如向左滑动,以使等高线模型跟者向左旋转,方便学者看到对立面的画面,从而更好的理解当前所讲解的知识点。For example, the "mountain peak" is determined as the contour map representing the mountain peak in the teaching picture. At this time, it is possible to perform virtual reality processing and augmented reality processing on the contour maps of the mountain peaks, and display the three-dimensional mountain graphics (virtual reality) on the plane graphics of the mountain peak contour lines, and at the same time, display The description information of the terrain features and the expression method used habitually. In this way, the learner can see the three-dimensional contour model, and during the viewing process, he can also make preset actions by holding the interactive pen, such as sliding to the left to rotate the contour model to the left. It is convenient for scholars to see the picture of the opposite, so as to better understand the knowledge points currently explained.
此外,上述对所述对象进行混合现实(Mixed Reality,MR)处理的操作,大致可以是,先对所述对象进行数字化处理,得到所述对象对应的图像矩阵;然后,确定所述图像矩阵与预先训练获得的各类物体对应的图像特征矩阵之间的相似度;接着,根据预设的筛选规则,筛选出相似度满足所述筛选规则的图像特征矩阵;接着,根据预设的映射关系表,获取筛选出的图像特征矩阵对应的渲染模型和对应的介绍信息,所述映射关系表为各图像特征矩阵与对应的渲染模型和对应的介绍信息之间的对应关系;接着,从所述教学流媒体中实时提取图像数据,确定所述对象在所述图像数据中的实时位置及大小;最后,根据所述对象在所述图像数据中的实时位置及大小,在所述图像数据上实时叠加所述渲染模型和所述介绍信息,得到所述混合现实教学画面。In addition, the above-mentioned mixed reality (MR) processing operation on the object may be roughly performed by first digitizing the object to obtain an image matrix corresponding to the object; then, determining the image matrix and The similarity between the image feature matrices corresponding to various types of objects obtained by pre-training; then, according to the preset filtering rules, the image feature matrix whose similarity meets the filtering rules is selected; then, according to the preset mapping relationship table To obtain the rendering model corresponding to the selected image feature matrix and the corresponding introduction information, the mapping relationship table is the correspondence between each image feature matrix and the corresponding rendering model and the corresponding introduction information; then, from the teaching Extract image data in real time from streaming media to determine the real-time position and size of the object in the image data; Finally, according to the real-time position and size of the object in the image data, superimpose on the image data in real time The rendering model and the introduction information obtain the mixed reality teaching picture.
此外,值得一提的是,在实际应用中,为了保证最终得到的混合现实教学画面的效果,在从所述教学流媒体中实时提取图像数据时,具体可以精确到帧,即以帧为单位从所述教学流媒体中实时提取图像数据,这样在确定所述对象在所述图像数据中的实时位置及大小时, 就可以根据所述对象的特征信息,对每一帧所述图像数据进行特征检测,确定所述对象在所述图像数据中的实时位置及大小。In addition, it is worth mentioning that, in practical applications, in order to ensure the effect of the final mixed reality teaching picture, when the image data is extracted from the teaching streaming media in real time, it can be specifically accurate to the frame, that is, the frame is the unit Extract image data from the teaching streaming media in real time, so that when determining the real-time position and size of the object in the image data, you can perform image data for each frame according to the feature information of the object Feature detection to determine the real-time position and size of the object in the image data.
通过这种精确到帧的处理方式,可以有效的保证后续确定的所述对象在所述图像数据中的实时位置及大小的准确性,进而能够准确的在所述图像数据上实时叠加所述渲染模型和所述介绍信息,保证了混合现实效果。Through this frame-accurate processing method, the accuracy of the real-time position and size of the object in the image data determined subsequently can be effectively ensured, so that the rendering can be accurately superimposed on the image data in real time The model and the introductory information ensure a mixed reality effect.
进一步地,为了保证上述操作能够顺利进行,上述操作中用到的各类物体对应的图像特征矩阵及映射关系表可以预先构建。Further, in order to ensure that the above operation can be carried out smoothly, the image feature matrix and the mapping relationship table corresponding to various types of objects used in the above operation can be constructed in advance.
比如,通过获取训练样本图像集合,所述训练样本图像集合包括各类物体对应的样本图像及各样本图像对应的物体类别;以各样本图像以及各样本图像对应的物体类别为输入,对深度学习模型进行分类训练,获得各类物体对应的图像特征矩阵;建立各类物体对应的图像特征矩阵与对应的渲染模型和对应的介绍信息之间的对应关系,生成所述映射关系表。For example, by acquiring a set of training sample images, the set of training sample images includes sample images corresponding to various types of objects and object categories corresponding to each sample image; taking each sample image and the object category corresponding to each sample image as input, for deep learning The model performs classification training to obtain image feature matrices corresponding to various types of objects; establishes a correspondence between image feature matrices corresponding to various types of objects, corresponding rendering models and corresponding introduction information, and generates the mapping relationship table.
需要说明的是,以上给出的仅为一种具体的实现方式,对本申请的技术方案并不构成限定,在具体实现中,本领域的技术人员可以根据需要选取合适的操作方式实现对教学画面的混合现实处理,此处不做限制。It should be noted that the above is only a specific implementation manner, and does not constitute a limitation on the technical solution of the present application. In a specific implementation, a person skilled in the art can select an appropriate operation method as required to realize the teaching screen The mixed reality processing is not limited here.
通过上述描述不难发现,本实施例中提供的基于生物识别的远程教学方法,通过在学习者观看教学流媒体的过程中实时采集包含学习者人脸的视频,借助生物识别技术确实视频中学习者的面部表情,然后通过对面部表情的分析来判断学习者是否处于预设的低效学习状态,当判定学习者处于预设的低效学习状态是,通过获取教学流媒体当前播放的教学画面和教学语音,并利用关键词提取技术从教学语音中提取关键词,然后根据提取到的关键词确定教学画面中需要进行混合现实的对象,最后借助混合现实技术对确定的对象进行混合现实处理,从而可以得到能够让学习者身临其境的混合现实教学画面,使得学习者在观看教学流媒体的过程中能够与混合现实教学画面中的对象进行互动,从而提升了学习者的参与度,使得学习者能够更好的通过远程教学方式进行自主学习。From the above description, it is not difficult to find that the remote teaching method based on biometrics provided in this embodiment collects video containing the learner's face in real time while the learner is watching the teaching streaming media, and indeed learns in the video with the help of biometric technology The facial expression of the learner, and then determine whether the learner is in the preset low-efficiency learning state by analyzing the facial expression. When it is determined that the learner is in the preset low-efficiency learning state, by acquiring the teaching picture currently played by the teaching streaming media And teaching voice, and use keyword extraction technology to extract keywords from the teaching voice, and then determine the objects that need to be mixed reality in the teaching picture according to the extracted keywords, and finally use mixed reality technology to perform mixed reality processing on the determined objects. Thereby, a mixed reality teaching picture that can make the learner immersive can be obtained, so that the learner can interact with the objects in the mixed reality teaching picture while watching the teaching streaming media, thereby enhancing the participation of the learner, making Learners can better use distance teaching methods Line self-learning.
参考图3,图3为本申请一种基于生物识别的远程教学方法第二实施例的流程示意图。Referring to FIG. 3, FIG. 3 is a schematic flowchart of a second embodiment of a remote teaching method based on biometrics of the present application.
基于上述第一实施例,本实施例基于生物识别的远程教学方法在步骤S50之后,还可以包括:Based on the first embodiment described above, the remote teaching method based on biometrics in this embodiment after step S50 may further include:
步骤S60,根据提取到的关键词查找对应的学习资料,将查找到的所述学习资料推送给所述学习者,以辅助学者理解所述关键词对应的知识点。Step S60: Find corresponding learning materials according to the extracted keywords, and push the found learning materials to the learner to assist the scholar to understand the knowledge points corresponding to the keywords.
具体的说,上述查找对应的学习资料的操作,可以是根据提取的关键词在互联网或者学习者使用的终端设备中预先存储的学习案例或中查找对应的学习资料。Specifically, the above-mentioned operation of searching for corresponding learning materials may be based on the extracted keywords to search for corresponding learning materials in a learning case or a pre-stored learning case in the Internet or a terminal device used by the learner.
相应地,在查找到学习资料后,将所述学习资料推送个所述学习者,具体可以是将所述学习资料发送给用户设置的邮箱,或者直接在当前终端设备的用户界面展示,以方便学习者进行查看、学习。Correspondingly, after the learning materials are found, the learning materials are pushed to the learner, which may specifically be sent to the mailbox set by the user, or displayed directly on the user interface of the current terminal device for convenience Learners view and learn.
此外,为了使得远程教学的方式能够更好的辅助学习者学习,可以在为学习者播放完教学流媒体后,在终端设备的用户界面提供反馈入口,以使学习者作出针对所述教学内容(如录制教学流媒体的教学者的教学方式、安排的教学课程等)的反馈信息,从而在用户输入反馈信息并点击界面上的确定按键后,将所述反馈信息上传的远程教学服务平台,这样不仅可以根据学习者反馈的内容适应性的调整教学内容,还可以根据学习者反馈的内容作为评估教学者的教学质量的参考。In addition, in order to enable the distance teaching method to better assist the learner in learning, after playing the teaching streaming media for the learner, a feedback portal may be provided in the user interface of the terminal device to enable the learner to make the content for the teaching ( For example, the teaching method of recording teaching streaming media, the teaching courses arranged, etc.) feedback information, so that after the user enters the feedback information and clicks the OK button on the interface, the feedback information is uploaded to the remote teaching service platform, so Not only can the teaching content be adaptively adjusted according to the content fed back by the learner, but also can be used as a reference to evaluate the teaching quality of the teacher based on the content fed back by the learner.
进一步地,为了能够根据学习者需求为学习者提供为期量身定制的教学流媒体,可以规定教学者在录制教学流媒体时,细化到每一个不同的知识点,从而可以使远程教学服务平台存储的教学流媒体时以知识点为单独,单独存储。这样,学习者在使用终端设备观看教学流媒体时,可以先输入自己想要学习的内容的关键词,然后由终端设备将这些关键词发送给远程教学服务平台,以使远程教学服务平台根据用户提供的学习内容的关键词,查找对应的知识点进行组合,得到符合用户要求的教学流媒体。Further, in order to be able to provide learners with tailor-made teaching streaming media according to the needs of learners, it can be stipulated that when the teaching streaming media is recorded, the teacher can be refined to each different knowledge point, thereby enabling the remote teaching service platform The stored teaching streaming media is based on knowledge points and stored separately. In this way, when the terminal device is used to watch the teaching streaming media, the learner can first input the keywords of the content that he wants to learn, and then the terminal device sends these keywords to the distance teaching service platform, so that the distance teaching service platform according to the user Provide the keywords of the learning content, find the corresponding knowledge points and combine them to obtain the teaching streaming media that meets the user's requirements.
需要说明的是,以上仅为举例说明,并不对本申请的技术方案构成限定,在具体实现中,本领域的技术人员可以根据实际情况进行设置,此处不做限制。It should be noted that the above is only an example and does not limit the technical solution of the present application. In a specific implementation, those skilled in the art can set it according to the actual situation, and there is no limitation here.
此外,应当理解的是,在实际应用中,根据关键词查找学习资料,并为学习者推送所述学习资料的操作,与上述步骤S50中根据关键词确定所述教学画面中需要进行混合现实的对象,对所述对象进行混合现实处理的操作可以并行处理,此处不做限制。In addition, it should be understood that in practical applications, searching for learning materials according to keywords and pushing the learning materials for learners is the same as determining the need for mixed reality in the teaching screen according to the keywords in step S50 above. Objects, operations for performing mixed reality processing on the objects can be processed in parallel, and there is no limitation here.
通过上述描述不难发现,本实施例提供的基于生物识别的远程教学方法,在确定学习者处于预设的低效学习状态时,通过根据提取到的关键词查找与学习者当前面临的知识盲点相关的学习资料,并将查找到的学习资料推送给学习者,实现了对学习者的即时提示,从而可以帮助学习者及早消除知识盲点,更好的辅助学习者使用远程教学方式进行学习。From the above description, it is not difficult to find that the distance learning method based on biometrics provided in this embodiment, when it is determined that the learner is in a preset low-efficiency learning state, searches for the knowledge blind spots that the learner is currently facing based on the extracted keywords Relevant learning materials, and push the found learning materials to learners, realizing immediate prompts to learners, which can help learners eliminate knowledge blind spots as early as possible, and better assist learners to use distance teaching methods for learning.
需要说明的是,本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完成,也可以通过计算机可读指令控制相关的硬件完成,所述的计算机可读指令可以存储于一种非易失性计算机可读存储介质中,上述提到的非易失性可读存储介质可以是只读存储器,磁盘或光盘等。It should be noted that those of ordinary skill in the art may understand that all or part of the steps to implement the above-mentioned embodiments may be completed by hardware, or may be controlled by computer-readable instructions to control related hardware. The computer-readable instructions may be stored in In a non-volatile computer-readable storage medium, the aforementioned non-volatile readable storage medium may be a read-only memory, a magnetic disk, or an optical disk.
参照图4,图4为本申请基于生物识别的远程教学装置第一实施例的结构框图。Referring to FIG. 4, FIG. 4 is a structural block diagram of a first embodiment of a remote teaching device based on biometrics of the present application.
如图4所示,本申请实施例提出的基于生物识别的远程教学装置包括:播放模块4001、采集模块4002、确定模块4003、判断模块4004、获取模块4005和处理模块4006。As shown in FIG. 4, the remote teaching device based on biometrics proposed in the embodiment of the present application includes: a playback module 4001, an acquisition module 4002, a determination module 4003, a judgment module 4004, an acquisition module 4005, and a processing module 4006.
其中,播放模块4001,用于接收学习者触发的学习指令,播放教学流媒体;采集模块4002,用于在播放所述教学流媒体的过程中采集包含所述学习者的人脸的视频;确定模块4003,用于确定所述视频中所述学习者的面部表情;判断模块4004,用于根据所述面部表情,判断所述学习者是否处于预设的低效学习状态;获取模块4005,用于在所述学习者处于预设的低效学习状态时,获取所述教学流媒体当前播放的教学画面和教学语音;处理模块4006,用于对所述教学语音进行关键词提取,根据提取到的关键词确定所述教学画面中需要进行混合现实的对象,对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面,以使学习者能够与所述混合现实教学画面中的对象进行互动。Among them, the playing module 4001 is used to receive the learning instruction triggered by the learner and play the teaching streaming media; the collecting module 4002 is used to collect the video containing the learner's face during the playing of the teaching streaming media; Module 4003, used to determine the facial expression of the learner in the video; Judgment module 4004, used to determine whether the learner is in a preset inefficient learning state based on the facial expression; Acquisition module 4005, used When the learner is in a preset low-efficiency learning state, the teaching picture and teaching voice currently played by the teaching streaming media are obtained; the processing module 4006 is used for keyword extraction of the teaching voice, according to the extracted Keywords identify the objects in the teaching picture that need to be mixed reality, and the mixed reality processing is performed on the objects to obtain a mixed reality teaching picture that can make the learner immersive, so that the learner can interact with the mixed reality The objects in the teaching picture interact.
应当理解的是,在实际应用中,上述确定模块4003确定所述视频中所述学习者的面部表情的操作,具体可以基于生物识别中的人脸识别技术实现。It should be understood that in practical applications, the operation of the determination module 4003 to determine the facial expression of the learner in the video may be specifically implemented based on the face recognition technology in biometrics.
比如,在确定所述视频中所述学习者的面部表情之前,先基于人脸识别技术中人脸特征检测方法对大数据平台中存储的人脸样本数据进行训练,获得人脸特征检测模型,从而在确定所述视频中所述学习者的面部表情时,能够根据预先训练获得的人脸特征检测模型,从所述视频中提取所述学习者的面部特征点。For example, before determining the facial expression of the learner in the video, the face sample data stored in the big data platform is trained based on the face feature detection method in face recognition technology to obtain a face feature detection model, Therefore, when determining the facial expression of the learner in the video, the facial feature points of the learner can be extracted from the video according to the facial feature detection model obtained in advance training.
接着,为了方便后期的对各个面部特征点变化的确定,可以根据各面部特征点,对所述学习者的人脸进行面部区域划分,得到与各面部特征点对应的面部特征区域。Next, in order to facilitate the later determination of the change of each facial feature point, the facial area of the learner's face may be divided according to each facial feature point to obtain a facial feature area corresponding to each facial feature point.
比如,规定划分后的面部特征区域中有且仅有一个面部特征点,即每一个面部特征点位于一个面特征区域。For example, there is only one facial feature point in the divided facial feature area, that is, each facial feature point is located in a facial feature area.
或者,规定同一对象的几个面部特征点位于一个面部特征区域,如标识左侧眉毛的所有面部特征点位于同一个面部特征区域,标识右侧眉毛的所有面部特征点位于同一个面部特征区域。Or, it is stipulated that several facial feature points of the same object are located in one facial feature area, for example, all facial feature points identifying the left eyebrow are located in the same facial feature area, and all facial feature points identifying the right eyebrow are located in the same facial feature area.
需要说明的是,以上仅为举例说明,对本申请的技术方案并不构成限定,在具体实现中,本领域的技术人员可以根据需要对人脸进行面部区域划分,此处不做限制。It should be noted that the above is only an example, and does not constitute a limitation on the technical solution of the present application. In a specific implementation, a person skilled in the art can divide the facial area of a human face as needed, and no limitation is made here.
在完成对人脸进行面部区域划分的操作之后,基于光流法,确定各面部区域中的面部特征点的速度向量。After completing the operation of dividing the facial area of the human face, based on the optical flow method, the velocity vectors of the facial feature points in each facial area are determined.
需要说明的是,此处所说的速度向量,不仅仅用于表示对应的面部特征点的运动速度信息,还用于表示该面部特征点的运动方向信息。It should be noted that the speed vector mentioned here is not only used to indicate the motion speed information of the corresponding facial feature point, but also used to indicate the motion direction information of the facial feature point.
此外,关于基于光流法,确定各面部区域中的面部特征点的速度向量的方式,具体可以是通过遍历各面部特征区域,检测遍历到的当前面部特征区域中的面部特征点在相邻两个图像帧之间的像素变化强度;然后根据所述像素变化强度,推断所述当前面部特征区域中的面部特征点的速度向量。In addition, regarding the method of determining the velocity vector of the facial feature points in each face area based on the optical flow method, it may specifically be that by traversing each facial feature area, the facial feature points in the current facial feature area traversed are detected in the adjacent two The intensity of pixel changes between image frames; then, based on the intensity of pixel changes, the velocity vector of facial feature points in the current facial feature area is inferred.
具体的计算方式,可以参考光流法的相关计算公式,此处不再赘述。For the specific calculation method, reference may be made to the related calculation formula of the optical flow method, which will not be repeated here.
最后,根据得到的各面部特征点的速度向量,确定所述视频中所 述学习者的面部表情即可。Finally, the facial expressions of the learner in the video may be determined based on the obtained velocity vectors of facial feature points.
需要说明的是,以上给出的仅为一种具体的实现方式,对本申请的技术方案并不构成限定,在具体实现中,本领域的技术人员可以根据需要选取合适的人脸识别方式确定学习者的面部表情,此处不做限制。It should be noted that the above is only a specific implementation manner, and does not constitute a limitation on the technical solution of the present application. In a specific implementation, a person skilled in the art may select an appropriate face recognition method to determine learning as required The facial expressions of the person are not limited here.
此外,为了便于理解出来模块4006对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面的操作,以下进行具体说明。In addition, in order to facilitate understanding, the module 4006 performs mixed reality processing on the object to obtain an operation of a mixed reality teaching screen that can enable the learner to be immersed, which will be described in detail below.
比如,先对所述对象进行数字化处理,得到所述对象对应的图像矩阵;然后,确定所述图像矩阵与预先训练获得的各类物体对应的图像特征矩阵之间的相似度;接着,根据预设的筛选规则,筛选出相似度满足所述筛选规则的图像特征矩阵;接着,根据预设的映射关系表,获取筛选出的图像特征矩阵对应的渲染模型和对应的介绍信息,所述映射关系表为各图像特征矩阵与对应的渲染模型和对应的介绍信息之间的对应关系;接着,从所述教学流媒体中实时提取图像数据,确定所述对象在所述图像数据中的实时位置及大小;最后,根据所述对象在所述图像数据中的实时位置及大小,在所述图像数据上实时叠加所述渲染模型和所述介绍信息,得到所述混合现实教学画面。For example, first digitize the object to obtain the image matrix corresponding to the object; then, determine the similarity between the image matrix and the image feature matrix corresponding to various types of objects obtained by pre-training; Set the filtering rules to select the image feature matrix whose similarity meets the filtering rule; then, according to the preset mapping relationship table, obtain the rendering model corresponding to the selected image feature matrix and the corresponding introduction information, the mapping relationship The table is the correspondence between each image feature matrix and the corresponding rendering model and corresponding introduction information; then, image data is extracted from the teaching streaming media in real time to determine the real-time position of the object in the image data and Size; Finally, according to the real-time position and size of the object in the image data, the rendering model and the introduction information are superimposed on the image data in real time to obtain the mixed reality teaching picture.
此外,值得一提的是,在实际应用中,为了保证最终得到的混合现实教学画面的效果,在从所述教学流媒体中实时提取图像数据时,具体可以精确到帧,即以帧为单位从所述教学流媒体中实时提取图像数据,这样在确定所述对象在所述图像数据中的实时位置及大小时,就可以根据所述对象的特征信息,对每一帧所述图像数据进行特征检测,确定所述对象在所述图像数据中的实时位置及大小。In addition, it is worth mentioning that, in practical applications, in order to ensure the effect of the final mixed reality teaching picture, when the image data is extracted from the teaching streaming media in real time, it can be specifically accurate to the frame, that is, the frame is the unit Extract image data from the teaching streaming media in real time, so that when determining the real-time position and size of the object in the image data, you can perform image data for each frame according to the object's feature information Feature detection to determine the real-time position and size of the object in the image data.
通过这种精确到帧的处理方式,可以有效的保证后续确定的所述对象在所述图像数据中的实时位置及大小的准确性,进而能够准确的在所述图像数据上实时叠加所述渲染模型和所述介绍信息,保证了混合现实效果。Through this frame-accurate processing method, the accuracy of the real-time position and size of the object in the image data determined subsequently can be effectively ensured, so that the rendering can be accurately superimposed on the image data in real time The model and the introductory information ensure a mixed reality effect.
进一步地,为了保证上述操作能够顺利进行,上述操作中用到的各类物体对应的图像特征矩阵及映射关系表可以预先构建。Further, in order to ensure that the above operation can be carried out smoothly, the image feature matrix and the mapping relationship table corresponding to various types of objects used in the above operation can be constructed in advance.
比如,通过获取训练样本图像集合,所述训练样本图像集合包括 各类物体对应的样本图像及各样本图像对应的物体类别;以各样本图像以及各样本图像对应的物体类别为输入,对深度学习模型进行分类训练,获得各类物体对应的图像特征矩阵;建立各类物体对应的图像特征矩阵与对应的渲染模型和对应的介绍信息之间的对应关系,生成所述映射关系表。For example, by acquiring a set of training sample images, the set of training sample images includes sample images corresponding to various types of objects and object categories corresponding to each sample image; taking each sample image and the object category corresponding to each sample image as input, for deep learning The model performs classification training to obtain image feature matrices corresponding to various types of objects; establishes a correspondence between image feature matrices corresponding to various types of objects, corresponding rendering models and corresponding introduction information, and generates the mapping relationship table.
需要说明的是,以上给出的仅为一种具体的实现方式,对本申请的技术方案并不构成限定,在具体实现中,本领域的技术人员可以根据需要选取合适的操作方式实现对教学画面的混合现实处理,此处不做限制。It should be noted that the above is only a specific implementation manner, and does not constitute a limitation on the technical solution of the present application. In a specific implementation, a person skilled in the art can select an appropriate operation method as required to realize the teaching screen The mixed reality processing is not limited here.
通过上述描述不难发现,本实施例中提供的基于生物识别的远程教学装置,通过在学习者观看教学流媒体的过程中实时采集包含学习者人脸的视频,借助生物识别技术确实视频中学习者的面部表情,然后通过对面部表情的分析来判断学习者是否处于预设的低效学习状态,当判定学习者处于预设的低效学习状态是,通过获取教学流媒体当前播放的教学画面和教学语音,并利用关键词提取技术从教学语音中提取关键词,然后根据提取到的关键词确定教学画面中需要进行混合现实的对象,最后借助混合现实技术对确定的对象进行混合现实处理,从而可以得到能够让学习者身临其境的混合现实教学画面,使得学习者在观看教学流媒体的过程中能够与混合现实教学画面中的对象进行互动,从而提升了学习者的参与度,使得学习者能够更好的通过远程教学方式进行自主学习。It is not difficult to find from the above description that the biometrics-based remote teaching device provided in this embodiment collects videos containing the learner's face in real time while the learner is watching the teaching streaming media, and indeed learns from the video with the help of biometric technology The facial expression of the learner, and then determine whether the learner is in the preset low-efficiency learning state by analyzing the facial expression. When it is determined that the learner is in the preset low-efficiency learning state, by acquiring the teaching picture currently played by the teaching streaming media And teaching voice, and use keyword extraction technology to extract keywords from the teaching voice, and then determine the objects that need to be mixed reality in the teaching picture according to the extracted keywords, and finally use mixed reality technology to perform mixed reality processing on the determined objects. Thereby, a mixed reality teaching picture that can make the learner immersive can be obtained, so that the learner can interact with the objects in the mixed reality teaching picture while watching the teaching streaming media, thereby enhancing the participation of the learner, making Learners can better use distance teaching methods Line self-learning.
需要说明的是,以上所描述的工作流程仅仅是示意性的,并不对本申请的保护范围构成限定,在实际应用中,本领域的技术人员可以根据实际的需要选择其中的部分或者全部来实现本实施例方案的目的,此处不做限制。It should be noted that the workflow described above is only schematic and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of them according to actual needs. The purpose of the solution of this embodiment is not limited here.
另外,未在本实施例中详尽描述的技术细节,可参见本申请任意实施例所提供的基于生物识别的远程教学方法,此处不再赘述。In addition, for technical details that are not described in detail in this embodiment, reference may be made to the remote teaching method based on biometrics provided in any embodiment of this application, and details are not described herein again.
基于上述基于生物识别的远程教学装置的第一实施例,提出本申请基于生物识别的远程教学装置第二实施例。Based on the above-mentioned first embodiment of the biometrics-based remote teaching device, the second embodiment of the biometrics-based remote teaching device of the present application is proposed.
在本实施例中,所述基于生物识别的远程教学装置还包括查找模 块和推送模块。In this embodiment, the remote teaching device based on biometrics further includes a search module and a push module.
其中,所述查找模块,用于根据提取到的关键词查找对应的学习资料;所述推送模块,用于将查找到的所述学习资料推送给所述学习者,以辅助学者理解所述关键词对应的知识点。Wherein, the search module is used to search corresponding learning materials according to the extracted keywords; the push module is used to push the found learning materials to the learners to assist scholars to understand the key Knowledge points corresponding to words.
通过上述描述不难发现,本实施例提供的基于生物识别的远程教学装置,在确定学习者处于预设的低效学习状态时,通过根据提取到的关键词查找与学习者当前面临的知识盲点相关的学习资料,并将查找到的学习资料推送给学习者,实现了对学习者的即时提示,从而可以帮助学习者及早消除知识盲点,更好的辅助学习者使用远程教学方式进行学习。It is not difficult to find from the above description that the biometrics-based remote teaching device provided in this embodiment, when it is determined that the learner is in a preset low-efficiency learning state, searches for the blind spots with the learner's current knowledge based on the extracted keywords Relevant learning materials, and push the found learning materials to learners, realizing immediate prompts to learners, which can help learners eliminate knowledge blind spots as early as possible, and better assist learners to use distance teaching methods for learning.
需要说明的是,以上所描述的工作流程仅仅是示意性的,并不对本申请的保护范围构成限定,在实际应用中,本领域的技术人员可以根据实际的需要选择其中的部分或者全部来实现本实施例方案的目的,此处不做限制。It should be noted that the workflow described above is only schematic and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of them according to actual needs. The purpose of the solution of this embodiment is not limited here.
另外,未在本实施例中详尽描述的技术细节,可参见本申请任意实施例所提供的基于生物识别的远程教学方法,此处不再赘述。In addition, for technical details that are not described in detail in this embodiment, reference may be made to the remote teaching method based on biometrics provided in any embodiment of this application, and details are not described herein again.
Claims (20)
- 一种基于生物识别的远程教学方法,所述方法包括:A remote teaching method based on biometrics, the method includes:接收学习者触发的学习指令,播放教学流媒体,并在播放所述教学流媒体的过程中采集包含所述学习者的人脸的视频;Receiving a learning instruction triggered by a learner, playing teaching streaming media, and collecting videos containing the learner's face during the playing of the teaching streaming media;确定所述视频中所述学习者的面部表情;Determine the facial expression of the learner in the video;根据所述面部表情,判断所述学习者是否处于预设的低效学习状态;According to the facial expression, determine whether the learner is in a preset inefficient learning state;若所述学习者处于预设的低效学习状态,则获取所述教学流媒体当前播放的教学画面和教学语音;If the learner is in a preset low-efficiency learning state, acquire the teaching picture and teaching voice currently played by the teaching streaming media;对所述教学语音进行关键词提取,根据提取到的关键词确定所述教学画面中需要进行混合现实的对象,对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面,以使学习者能够与所述混合现实教学画面中的对象进行互动。Perform keyword extraction on the teaching speech, determine the objects that need to be mixed reality in the teaching picture according to the extracted keywords, and perform mixed reality processing on the objects to obtain mixed reality that can make the learner immersive Teaching pictures to enable learners to interact with objects in the mixed reality teaching pictures.
- 如权利要求1所述的方法,所述确定所述视频中所述学习者的面部表情的步骤,包括:The method of claim 1, the step of determining the facial expression of the learner in the video comprises:根据预先训练获得的人脸特征检测模型,从所述视频中提取所述学习者的面部特征点;Extract the learner's facial feature points from the video according to the facial feature detection model obtained in advance training;根据各面部特征点,对所述学习者的人脸进行面部区域划分,得到与各面部特征点对应的面部特征区域;Divide the facial area of the learner's face according to each facial feature point to obtain a facial feature area corresponding to each facial feature point;基于光流法,确定各面部区域中的面部特征点的速度向量,所述速度向量用于表示各面部特征点的运动速度信息和运动方向信息;Based on the optical flow method, the velocity vectors of the facial feature points in each facial area are determined, and the velocity vectors are used to represent the movement speed information and the movement direction information of each facial feature point;根据各面部特征点的速度向量,确定所述视频中所述学习者的面部表情。The facial expression of the learner in the video is determined according to the velocity vector of each facial feature point.
- 如权利要求2所述的方法,所述基于光流法,确定各面部区域中的面部特征点的速度向量的步骤,包括:The method of claim 2, the step of determining the velocity vector of facial feature points in each facial area based on the optical flow method includes:遍历各面部特征区域,检测遍历到的当前面部特征区域中的面部特征点在相邻两个图像帧之间的像素变化强度;Traverse each facial feature area, and detect the pixel change intensity of the facial feature points in the current facial feature area between two adjacent image frames;根据所述像素变化强度,推断所述当前面部特征区域中的面部特征点的速度向量。According to the intensity of the pixel change, the velocity vector of the facial feature point in the current facial feature area is inferred.
- 如权利要求1所述的方法,所述对所述对象进行混合现实处 理,得到能够让学习者身临其境的混合现实教学画面的步骤,包括:The method according to claim 1, wherein the step of performing mixed reality processing on the object to obtain a mixed reality teaching picture that enables the learner to be immersive includes:对所述对象进行数字化处理,得到所述对象对应的图像矩阵;Digitizing the object to obtain an image matrix corresponding to the object;确定所述图像矩阵与预先训练获得的各类物体对应的图像特征矩阵之间的相似度;Determine the similarity between the image matrix and the image feature matrix corresponding to various types of objects obtained by pre-training;根据预设的筛选规则,筛选出相似度满足所述筛选规则的图像特征矩阵;According to a preset filtering rule, the image feature matrix whose similarity meets the filtering rule is selected;根据预设的映射关系表,获取筛选出的图像特征矩阵对应的渲染模型和对应的介绍信息,所述映射关系表为各图像特征矩阵与对应的渲染模型和对应的介绍信息之间的对应关系;According to a preset mapping relationship table, the rendering model corresponding to the selected image feature matrix and corresponding introduction information are obtained. The mapping relationship table is a correspondence between each image feature matrix and the corresponding rendering model and corresponding introduction information. ;从所述教学流媒体中实时提取图像数据,确定所述对象在所述图像数据中的实时位置及大小;Extract image data from the teaching streaming media in real time to determine the real-time position and size of the object in the image data;根据所述对象在所述图像数据中的实时位置及大小,在所述图像数据上实时叠加所述渲染模型和所述介绍信息,得到所述混合现实教学画面。According to the real-time position and size of the object in the image data, the rendering model and the introduction information are superimposed on the image data in real time to obtain the mixed reality teaching picture.
- 如权利要求4所述的方法,所述从所述教学流媒体中实时提取图像数据,确定所述对象在所述图像数据中的实时位置及大小的步骤,包括:The method of claim 4, the step of extracting image data from the teaching streaming media in real time and determining the real-time position and size of the object in the image data includes:以帧为单位从所述教学流媒体中实时提取图像数据;Extract image data in real time from the teaching streaming media in units of frames;根据所述对象的特征信息,对每一帧所述图像数据进行特征检测,确定所述对象在所述图像数据中的实时位置及大小。According to the feature information of the object, perform feature detection on each frame of the image data to determine the real-time position and size of the object in the image data.
- 如权利要求4所述的方法,所述对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面的步骤之前,所述方法还包括:The method according to claim 4, before the step of performing mixed reality processing on the object to obtain a mixed reality teaching picture that enables the learner to be immersive, the method further includes:获取训练样本图像集合,所述训练样本图像集合包括各类物体对应的样本图像及各样本图像对应的物体类别;Obtain a training sample image set, where the training sample image set includes sample images corresponding to various types of objects and object categories corresponding to each sample image;以各样本图像以及各样本图像对应的物体类别为输入,对深度学习模型进行分类训练,获得各类物体对应的图像特征矩阵;Take each sample image and the object category corresponding to each sample image as input, perform classification training on the deep learning model, and obtain the image feature matrix corresponding to various objects;建立各类物体对应的图像特征矩阵与对应的渲染模型和对应的介绍信息之间的对应关系,生成所述映射关系表。A correspondence relationship between an image feature matrix corresponding to various objects, a corresponding rendering model and corresponding introduction information is established, and the mapping relationship table is generated.
- 如权利要求1所述的方法,所述得到能够让学习者身临其境的混合现实教学画面的步骤之后,所述方法还包括:The method according to claim 1, after the step of obtaining a mixed reality teaching picture that enables the learner to be immersive, the method further comprises:根据提取到的关键词查找对应的学习资料,将查找到的所述学习资料推送给所述学习者,以辅助学者理解所述关键词对应的知识点。Find corresponding learning materials according to the extracted keywords, and push the found learning materials to the learners to assist the scholar to understand the knowledge points corresponding to the keywords.
- 一种基于生物识别的远程教学装置,所述装置包括:A remote teaching device based on biometrics, the device includes:播放模块,用于接收学习者触发的学习指令,播放教学流媒体;Play module, used to receive learning instructions triggered by learners and play teaching streaming media;采集模块,用于在播放所述教学流媒体的过程中采集包含所述学习者的人脸的视频;A collection module, configured to collect a video containing the learner's face during the playback of the teaching streaming media;确定模块,用于确定所述视频中所述学习者的面部表情;A determination module for determining the facial expression of the learner in the video;判断模块,用于根据所述面部表情,判断所述学习者是否处于预设的低效学习状态;The judgment module is used to judge whether the learner is in a preset low-efficiency learning state according to the facial expression;获取模块,用于在所述学习者处于预设的低效学习状态时,获取所述教学流媒体当前播放的教学画面和教学语音;An obtaining module, configured to obtain the teaching picture and teaching voice currently played by the teaching streaming media when the learner is in a preset low-efficiency learning state;处理模块,用于对所述教学语音进行关键词提取,根据提取到的关键词确定所述教学画面中需要进行混合现实的对象,对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面,以使学习者能够与所述混合现实教学画面中的对象进行互动。The processing module is used for keyword extraction of the teaching speech, determining the object that needs to be mixed reality in the teaching picture according to the extracted keyword, and performing mixed reality processing on the object to obtain the learner's presence The mixed reality teaching picture in the context, so that the learner can interact with the objects in the mixed reality teaching picture.
- 一种基于生物识别的远程教学设备,所述设备包括:存储器、处理器及存储在所述存储器上并可在所述处理器上运行的基于生物识别的远程教学的可读指令,所述基于生物识别的远程教学的可读指令配置为实现以下步骤:A remote teaching device based on biometrics, the device includes: a memory, a processor, and a readable instruction for remote teaching based on biometrics that is stored on the memory and can run on the processor, based on The readable instructions for distance learning in biometrics are configured to implement the following steps:接收学习者触发的学习指令,播放教学流媒体,并在播放所述教学流媒体的过程中采集包含所述学习者的人脸的视频;Receiving a learning instruction triggered by a learner, playing teaching streaming media, and collecting videos containing the learner's face during the playing of the teaching streaming media;确定所述视频中所述学习者的面部表情;Determine the facial expression of the learner in the video;根据所述面部表情,判断所述学习者是否处于预设的低效学习状态;According to the facial expression, determine whether the learner is in a preset inefficient learning state;若所述学习者处于预设的低效学习状态,则获取所述教学流媒体当前播放的教学画面和教学语音;If the learner is in a preset low-efficiency learning state, acquire the teaching picture and teaching voice currently played by the teaching streaming media;对所述教学语音进行关键词提取,根据提取到的关键词确定所述教学画面中需要进行混合现实的对象,对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面,以使学习者能够与所述混合现实教学画面中的对象进行互动。Perform keyword extraction on the teaching speech, determine the objects that need to be mixed reality in the teaching picture according to the extracted keywords, and perform mixed reality processing on the objects to obtain mixed reality that can make the learner immersive Teaching pictures to enable learners to interact with objects in the mixed reality teaching pictures.
- 如权利要求9所述的基于生物识别的远程教学设备,所述确 定所述视频中所述学习者的面部表情的步骤,包括:The distance learning device based on biometric recognition according to claim 9, the step of determining the facial expression of the learner in the video includes:根据预先训练获得的人脸特征检测模型,从所述视频中提取所述学习者的面部特征点;Extract the learner's facial feature points from the video according to the facial feature detection model obtained in advance training;根据各面部特征点,对所述学习者的人脸进行面部区域划分,得到与各面部特征点对应的面部特征区域;Divide the facial area of the learner's face according to each facial feature point to obtain a facial feature area corresponding to each facial feature point;基于光流法,确定各面部区域中的面部特征点的速度向量,所述速度向量用于表示各面部特征点的运动速度信息和运动方向信息;Based on the optical flow method, the velocity vectors of the facial feature points in each facial area are determined, and the velocity vectors are used to represent the movement speed information and the movement direction information of each facial feature point;根据各面部特征点的速度向量,确定所述视频中所述学习者的面部表情。The facial expression of the learner in the video is determined according to the velocity vector of each facial feature point.
- 如权利要求10所述的基于生物识别的远程教学设备,所述基于光流法,确定各面部区域中的面部特征点的速度向量的步骤,包括:The distance learning device based on biometric recognition according to claim 10, the step of determining the velocity vector of facial feature points in each facial area based on the optical flow method includes:遍历各面部特征区域,检测遍历到的当前面部特征区域中的面部特征点在相邻两个图像帧之间的像素变化强度;Traverse each facial feature area, and detect the pixel change intensity of the facial feature points in the current facial feature area between two adjacent image frames;根据所述像素变化强度,推断所述当前面部特征区域中的面部特征点的速度向量。According to the intensity of the pixel change, the velocity vector of the facial feature point in the current facial feature area is inferred.
- 如权利要求9所述的基于生物识别的远程教学设备,所述对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面的步骤,包括:The biometrics-based remote teaching device according to claim 9, the step of performing mixed reality processing on the object to obtain a mixed reality teaching picture that can enable the learner to be immersive includes:对所述对象进行数字化处理,得到所述对象对应的图像矩阵;Digitizing the object to obtain an image matrix corresponding to the object;确定所述图像矩阵与预先训练获得的各类物体对应的图像特征矩阵之间的相似度;Determine the similarity between the image matrix and the image feature matrix corresponding to various types of objects obtained by pre-training;根据预设的筛选规则,筛选出相似度满足所述筛选规则的图像特征矩阵;According to a preset filtering rule, the image feature matrix whose similarity meets the filtering rule is selected;根据预设的映射关系表,获取筛选出的图像特征矩阵对应的渲染模型和对应的介绍信息,所述映射关系表为各图像特征矩阵与对应的渲染模型和对应的介绍信息之间的对应关系;According to a preset mapping relationship table, the rendering model corresponding to the selected image feature matrix and corresponding introduction information are obtained. The mapping relationship table is a correspondence between each image feature matrix and the corresponding rendering model and corresponding introduction information. ;从所述教学流媒体中实时提取图像数据,确定所述对象在所述图像数据中的实时位置及大小;Extract image data from the teaching streaming media in real time to determine the real-time position and size of the object in the image data;根据所述对象在所述图像数据中的实时位置及大小,在所述图像数据上实时叠加所述渲染模型和所述介绍信息,得到所述混合现实教 学画面。According to the real-time position and size of the object in the image data, the rendering model and the introduction information are superimposed on the image data in real time to obtain the mixed reality teaching picture.
- 如权利要求12所述的基于生物识别的远程教学设备,所述从所述教学流媒体中实时提取图像数据,确定所述对象在所述图像数据中的实时位置及大小的步骤,包括:The remote teaching device based on biometrics according to claim 12, the step of extracting image data from the teaching streaming media in real time and determining the real-time position and size of the object in the image data includes:以帧为单位从所述教学流媒体中实时提取图像数据;Extract image data in real time from the teaching streaming media in units of frames;根据所述对象的特征信息,对每一帧所述图像数据进行特征检测,确定所述对象在所述图像数据中的实时位置及大小。According to the feature information of the object, perform feature detection on each frame of the image data to determine the real-time position and size of the object in the image data.
- 如权利要求12所述的基于生物识别的远程教学设备,所述对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面的步骤之前,所述方法还包括:According to the biometrics-based remote teaching device of claim 12, before the step of performing mixed reality processing on the object to obtain a mixed reality teaching picture that enables the learner to be immersive, the method further includes:获取训练样本图像集合,所述训练样本图像集合包括各类物体对应的样本图像及各样本图像对应的物体类别;Obtain a training sample image set, where the training sample image set includes sample images corresponding to various types of objects and object categories corresponding to each sample image;以各样本图像以及各样本图像对应的物体类别为输入,对深度学习模型进行分类训练,获得各类物体对应的图像特征矩阵;Take each sample image and the object category corresponding to each sample image as input, perform classification training on the deep learning model, and obtain the image feature matrix corresponding to various objects;建立各类物体对应的图像特征矩阵与对应的渲染模型和对应的介绍信息之间的对应关系,生成所述映射关系表。A correspondence relationship between an image feature matrix corresponding to various objects, a corresponding rendering model and corresponding introduction information is established, and the mapping relationship table is generated.
- 如权利要求9所述的基于生物识别的远程教学设备,所述得到能够让学习者身临其境的混合现实教学画面的步骤之后,所述方法还包括:The distance learning device based on biometrics according to claim 9, after the step of obtaining a mixed reality teaching picture that enables the learner to be immersive, the method further includes:根据提取到的关键词查找对应的学习资料,将查找到的所述学习资料推送给所述学习者,以辅助学者理解所述关键词对应的知识点。Find corresponding learning materials according to the extracted keywords, and push the found learning materials to the learners to assist the scholar to understand the knowledge points corresponding to the keywords.
- 一种存储介质,所述存储介质上存储有基于生物识别的远程教学的可读指令,所述基于生物识别的远程教学的可读指令被处理器执行时实现以下步骤:A storage medium storing readable instructions for distance learning based on biometrics on a storage medium, the readable instructions for distance learning based on biometrics being executed by a processor to implement the following steps:接收学习者触发的学习指令,播放教学流媒体,并在播放所述教学流媒体的过程中采集包含所述学习者的人脸的视频;Receiving a learning instruction triggered by a learner, playing teaching streaming media, and collecting videos containing the learner's face during the playing of the teaching streaming media;确定所述视频中所述学习者的面部表情;Determine the facial expression of the learner in the video;根据所述面部表情,判断所述学习者是否处于预设的低效学习状态;According to the facial expression, determine whether the learner is in a preset inefficient learning state;若所述学习者处于预设的低效学习状态,则获取所述教学流媒体当前播放的教学画面和教学语音;If the learner is in a preset low-efficiency learning state, acquire the teaching picture and teaching voice currently played by the teaching streaming media;对所述教学语音进行关键词提取,根据提取到的关键词确定所述教学画面中需要进行混合现实的对象,对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面,以使学习者能够与所述混合现实教学画面中的对象进行互动。Perform keyword extraction on the teaching speech, determine the objects that need to be mixed reality in the teaching picture according to the extracted keywords, and perform mixed reality processing on the objects to obtain mixed reality that can make the learner immersive Teaching pictures to enable learners to interact with objects in the mixed reality teaching pictures.
- 如权利要求16所述的存储介质,所述确定所述视频中所述学习者的面部表情的步骤,包括:The storage medium of claim 16, the step of determining the facial expression of the learner in the video comprises:根据预先训练获得的人脸特征检测模型,从所述视频中提取所述学习者的面部特征点;Extract the learner's facial feature points from the video according to the facial feature detection model obtained in advance training;根据各面部特征点,对所述学习者的人脸进行面部区域划分,得到与各面部特征点对应的面部特征区域;Divide the facial area of the learner's face according to each facial feature point to obtain a facial feature area corresponding to each facial feature point;基于光流法,确定各面部区域中的面部特征点的速度向量,所述速度向量用于表示各面部特征点的运动速度信息和运动方向信息;Based on the optical flow method, the velocity vectors of the facial feature points in each facial area are determined, and the velocity vectors are used to represent the movement speed information and the movement direction information of each facial feature point;根据各面部特征点的速度向量,确定所述视频中所述学习者的面部表情。The facial expression of the learner in the video is determined according to the velocity vector of each facial feature point.
- 如权利要求17所述的存储介质,所述基于光流法,确定各面部区域中的面部特征点的速度向量的步骤,包括:The storage medium according to claim 17, wherein the step of determining the velocity vector of the facial feature points in each facial area based on the optical flow method includes:遍历各面部特征区域,检测遍历到的当前面部特征区域中的面部特征点在相邻两个图像帧之间的像素变化强度;Traverse each facial feature area, and detect the pixel change intensity of the facial feature points in the current facial feature area between two adjacent image frames;根据所述像素变化强度,推断所述当前面部特征区域中的面部特征点的速度向量。According to the intensity of the pixel change, the velocity vector of the facial feature point in the current facial feature area is inferred.
- 如权利要求16所述的存储介质,所述对所述对象进行混合现实处理,得到能够让学习者身临其境的混合现实教学画面的步骤,包括:The storage medium according to claim 16, wherein the step of performing mixed reality processing on the object to obtain a mixed reality teaching picture that enables the learner to be immersive includes:对所述对象进行数字化处理,得到所述对象对应的图像矩阵;Digitizing the object to obtain an image matrix corresponding to the object;确定所述图像矩阵与预先训练获得的各类物体对应的图像特征矩阵之间的相似度;Determine the similarity between the image matrix and the image feature matrix corresponding to various types of objects obtained by pre-training;根据预设的筛选规则,筛选出相似度满足所述筛选规则的图像特征矩阵;According to a preset filtering rule, the image feature matrix whose similarity meets the filtering rule is selected;根据预设的映射关系表,获取筛选出的图像特征矩阵对应的渲染模型和对应的介绍信息,所述映射关系表为各图像特征矩阵与对应的渲染模型和对应的介绍信息之间的对应关系;According to a preset mapping relationship table, the rendering model corresponding to the selected image feature matrix and corresponding introduction information are obtained. The mapping relationship table is a correspondence between each image feature matrix and the corresponding rendering model and corresponding introduction information. ;从所述教学流媒体中实时提取图像数据,确定所述对象在所述图像数据中的实时位置及大小;Extract image data from the teaching streaming media in real time to determine the real-time position and size of the object in the image data;根据所述对象在所述图像数据中的实时位置及大小,在所述图像数据上实时叠加所述渲染模型和所述介绍信息,得到所述混合现实教学画面。According to the real-time position and size of the object in the image data, the rendering model and the introduction information are superimposed on the image data in real time to obtain the mixed reality teaching picture.
- 如权利要求16所述的存储介质,所述得到能够让学习者身临其境的混合现实教学画面的步骤之后,所述方法还包括:The storage medium according to claim 16, after the step of obtaining a mixed reality teaching picture that enables the learner to be immersive, the method further includes:根据提取到的关键词查找对应的学习资料,将查找到的所述学习资料推送给所述学习者,以辅助学者理解所述关键词对应的知识点。Find corresponding learning materials according to the extracted keywords, and push the found learning materials to the learners to assist the scholar to understand the knowledge points corresponding to the keywords.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201811247129.7 | 2018-10-25 | ||
CN201811247129.7A CN109614849A (en) | 2018-10-25 | 2018-10-25 | Remote teaching method, apparatus, equipment and storage medium based on bio-identification |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020082566A1 true WO2020082566A1 (en) | 2020-04-30 |
Family
ID=66001650
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2018/123186 WO2020082566A1 (en) | 2018-10-25 | 2018-12-24 | Physiological sign recognition-based distance learning method, device, apparatus, and storage medium |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN109614849A (en) |
WO (1) | WO2020082566A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111914694A (en) * | 2020-07-16 | 2020-11-10 | 哈尔滨工程大学 | Class quality detection method based on face recognition |
CN113283383A (en) * | 2021-06-15 | 2021-08-20 | 北京有竹居网络技术有限公司 | Live broadcast behavior recognition method, device, equipment and readable medium |
CN116824280A (en) * | 2023-08-30 | 2023-09-29 | 安徽爱学堂教育科技有限公司 | Psychological early warning method based on micro-expression change |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110033659B (en) * | 2019-04-26 | 2022-01-21 | 北京大米科技有限公司 | Remote teaching interaction method, server, terminal and system |
CN110390048A (en) * | 2019-06-19 | 2019-10-29 | 深圳壹账通智能科技有限公司 | Information-pushing method, device, equipment and storage medium based on big data analysis |
CN110503582A (en) * | 2019-07-16 | 2019-11-26 | 王霞 | Cloud system is educated based on the group interaction of mixed reality and multidimensional reality technology |
CN110458076A (en) * | 2019-08-05 | 2019-11-15 | 西安瑜乐文化科技股份有限公司 | A kind of teaching method based on computer vision and system |
CN110992222A (en) * | 2019-11-05 | 2020-04-10 | 深圳追一科技有限公司 | Teaching interaction method and device, terminal equipment and storage medium |
CN110992741B (en) * | 2019-11-15 | 2022-05-17 | 深圳算子科技有限公司 | Learning auxiliary method and system based on classroom emotion and behavior analysis |
CN111710030A (en) * | 2020-05-29 | 2020-09-25 | 上海红阵信息科技有限公司 | AI-based system and method for resisting deep forgery portrait |
CN111800646A (en) * | 2020-06-24 | 2020-10-20 | 北京安博盛赢教育科技有限责任公司 | Method, device, medium and electronic equipment for monitoring teaching effect |
CN112652200A (en) * | 2020-11-16 | 2021-04-13 | 北京家有课堂科技有限公司 | Man-machine interaction system, man-machine interaction method, server, interaction control device and storage medium |
CN112382151B (en) * | 2020-11-16 | 2022-11-18 | 深圳市商汤科技有限公司 | Online learning method and device, electronic equipment and storage medium |
CN113435975B (en) * | 2021-06-29 | 2023-09-26 | 平安科技(深圳)有限公司 | Wheelchair leasing processing method and device and related equipment |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080020363A1 (en) * | 2006-07-22 | 2008-01-24 | Yao-Jen Chang | Learning Assessment Method And Device Using A Virtual Tutor |
CN106599881A (en) * | 2016-12-30 | 2017-04-26 | 首都师范大学 | Student state determination method, device and system |
CN106778539A (en) * | 2016-11-25 | 2017-05-31 | 鲁东大学 | Teaching effect information acquisition methods and device |
CN107292271A (en) * | 2017-06-23 | 2017-10-24 | 北京易真学思教育科技有限公司 | Learning-memory behavior method, device and electronic equipment |
CN108009954A (en) * | 2017-12-12 | 2018-05-08 | 联想(北京)有限公司 | A kind of Formulating Teaching Program method, apparatus, system and electronic equipment |
-
2018
- 2018-10-25 CN CN201811247129.7A patent/CN109614849A/en active Pending
- 2018-12-24 WO PCT/CN2018/123186 patent/WO2020082566A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080020363A1 (en) * | 2006-07-22 | 2008-01-24 | Yao-Jen Chang | Learning Assessment Method And Device Using A Virtual Tutor |
CN106778539A (en) * | 2016-11-25 | 2017-05-31 | 鲁东大学 | Teaching effect information acquisition methods and device |
CN106599881A (en) * | 2016-12-30 | 2017-04-26 | 首都师范大学 | Student state determination method, device and system |
CN107292271A (en) * | 2017-06-23 | 2017-10-24 | 北京易真学思教育科技有限公司 | Learning-memory behavior method, device and electronic equipment |
CN108009954A (en) * | 2017-12-12 | 2018-05-08 | 联想(北京)有限公司 | A kind of Formulating Teaching Program method, apparatus, system and electronic equipment |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111914694A (en) * | 2020-07-16 | 2020-11-10 | 哈尔滨工程大学 | Class quality detection method based on face recognition |
CN113283383A (en) * | 2021-06-15 | 2021-08-20 | 北京有竹居网络技术有限公司 | Live broadcast behavior recognition method, device, equipment and readable medium |
CN116824280A (en) * | 2023-08-30 | 2023-09-29 | 安徽爱学堂教育科技有限公司 | Psychological early warning method based on micro-expression change |
CN116824280B (en) * | 2023-08-30 | 2023-11-24 | 安徽爱学堂教育科技有限公司 | Psychological early warning method based on micro-expression change |
Also Published As
Publication number | Publication date |
---|---|
CN109614849A (en) | 2019-04-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020082566A1 (en) | Physiological sign recognition-based distance learning method, device, apparatus, and storage medium | |
CN109271945B (en) | Method and system for realizing job correction on line | |
Gunes et al. | Bodily expression for automatic affect recognition | |
CN109885595A (en) | Course recommended method, device, equipment and storage medium based on artificial intelligence | |
WO2021047185A1 (en) | Monitoring method and apparatus based on facial recognition, and storage medium and computer device | |
Dubbaka et al. | Detecting learner engagement in MOOCs using automatic facial expression recognition | |
Oliveira et al. | Automatic sign language translation to improve communication | |
Sharma et al. | Student concentration evaluation index in an e-learning context using facial emotion analysis | |
Zaletelj | Estimation of students' attention in the classroom from kinect features | |
Wu et al. | Object recognition-based second language learning educational robot system for Chinese preschool children | |
Geng et al. | Learning deep spatiotemporal feature for engagement recognition of online courses | |
Pise et al. | Estimation of learning affects experienced by learners: an approach using relational reasoning and adaptive mapping | |
Duraisamy et al. | Classroom engagement evaluation using computer vision techniques | |
Villegas-Ch et al. | Identification of emotions from facial gestures in a teaching environment with the use of machine learning techniques | |
CN111339878B (en) | Correction type real-time emotion recognition method and system based on eye movement data | |
KR102482841B1 (en) | Artificial intelligence mirroring play bag | |
CN116524772A (en) | Traditional cultural education system based on Internet | |
Hernández Correa et al. | An application of machine learning and image processing to automatically detect teachers’ gestures | |
CN115239533A (en) | Interactive online English teaching system and use method thereof | |
JP2020086075A (en) | Learning support system and program | |
CN114779942A (en) | Virtual reality immersive interaction system, equipment and method | |
CN115132027A (en) | Intelligent programming learning system and method based on multi-mode deep learning | |
JP6980883B1 (en) | Assist system, assist method, and assist program | |
CN116434253A (en) | Image processing method, device, equipment, storage medium and product | |
Komiya et al. | Head pose estimation and movement analysis for speech scene |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18938190 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS (EPO FORM 1205A DATED 31.08.2021) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 18938190 Country of ref document: EP Kind code of ref document: A1 |