CN113296609A - Immersive remote teaching method and system - Google Patents
Immersive remote teaching method and system Download PDFInfo
- Publication number
- CN113296609A CN113296609A CN202110618197.5A CN202110618197A CN113296609A CN 113296609 A CN113296609 A CN 113296609A CN 202110618197 A CN202110618197 A CN 202110618197A CN 113296609 A CN113296609 A CN 113296609A
- Authority
- CN
- China
- Prior art keywords
- module
- remote server
- video
- video signal
- processor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 20
- 230000002194 synthesizing effect Effects 0.000 claims abstract description 29
- 239000002131 composite material Substances 0.000 claims description 20
- 238000006243 chemical reaction Methods 0.000 claims description 9
- 230000004424 eye movement Effects 0.000 claims description 9
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 238000004458 analytical method Methods 0.000 claims description 3
- 230000001960 triggered effect Effects 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 claims description 2
- 238000004513 sizing Methods 0.000 claims 1
- 230000006399 behavior Effects 0.000 abstract description 12
- 241000282414 Homo sapiens Species 0.000 abstract description 3
- 238000007654 immersion Methods 0.000 abstract description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 230000002401 inhibitory effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/20—Education
- G06Q50/205—Education administration or guidance
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Tourism & Hospitality (AREA)
- Strategic Management (AREA)
- Software Systems (AREA)
- Educational Administration (AREA)
- Educational Technology (AREA)
- Marketing (AREA)
- Health & Medical Sciences (AREA)
- General Business, Economics & Management (AREA)
- Primary Health Care (AREA)
- Economics (AREA)
- Human Resources & Organizations (AREA)
- Human Computer Interaction (AREA)
- General Health & Medical Sciences (AREA)
- Electrically Operated Instructional Devices (AREA)
Abstract
The invention discloses a remote teaching method and a remote teaching system, which comprise a camera, a processor, a VR module and an image processing and synthesizing module, wherein the camera, the VR module and the image processing and synthesizing module are all connected with the processor, and the processor is wirelessly connected with a remote server through a 5G module. The invention further processes the video by collecting the behaviors of teachers or students and analyzing whether the behaviors meet the requirement of attention so as to strengthen individual videos to improve the immersion degree, and the invention meets the actual attention rule of human beings to the surrounding classroom environment.
Description
Technical Field
The invention relates to the field of remote teaching, in particular to an immersive remote teaching method and system.
Background
Because the remote teaching has problems in teaching interaction, the application of the virtual reality technology in the teaching can bring immersive experience, and meanwhile, the virtual reality technology is closer to a real classroom in the teaching interaction aspect, however, because the audio and video acquisition equipment involved in the remote teaching is numerous, the processing modes of the audio and video data acquired by various equipment on VR equipment are different, and the problem of audio and video data asynchronization exists;
in the prior art, audio and video synchronization is usually completed by calculating time delay of voice and video data and then adjusting the time sequence of the audio and video, and the adjusting method is only suitable for regular audio and video transmission of stable transmission, however, the effect of completing synchronization only by adopting a time delay mode is not good due to the objective problems of network delay, equipment blockage and the like, and moreover, due to the natural existence of errors of audio and video time delay calculation, the accumulated time errors are increased after long-time accumulation, and further the audio and video data are not synchronized again.
Disclosure of Invention
To solve the problems of the background art, the present invention provides an immersive remote teaching method and system,
one of the technical schemes is as follows: a remote teaching method comprises the following steps:
s1, a teacher sets a teaching mode, wherein the teaching mode comprises a teaching mode, a discussion mode and a question asking mode;
s2, comprehensively analyzing behavior data of students, wherein the behavior data comprises eye movement, head posture, body posture and voice data, and judging whether to trigger an attention signal;
s3, if the attention signal is triggered, sending the attention signal to a remote server;
s4, the remote server reconstructs and strengthens the composite video signal of the student triggering the attention signal and combines the composite video signal into virtual scene data;
and S5, the remote server sends the virtual scene data to each student terminal and/or teacher terminal for display.
The teaching mode setting in S1 is set in an administrator mode, which can control whether and how student data in the remote server is transmitted, and has a function of muting overall or banning individual;
in the teaching mode, audio and video data in the teacher terminal are synchronously transmitted, the teacher audio and video data in the virtual scene data are in a protruding position, the teacher terminal can control whether the audio and video data of the student terminals are collected or not, whether a speech forbidding function is started or not and the like, and can control the remote server to strengthen the video information of the teacher all the time.
In the discussion mode, audio and video data of the student terminal and the teacher terminal are synchronously acquired;
in the questioning mode, the student terminals acquire whether students request questioning, and the teacher terminal controls whether to accept or accept questioning of which student;
the eye movement and the head posture in the behavior data in the S2 are acquired by a micro camera and an MEMS gyroscope in a VR module;
the body posture and the voice data in the S2 are acquired by a camera and a voice module which are used for shooting body image data of students or teachers;
the comprehensive analysis in S2 is to determine whether the student wants to participate in the discussion or answer the question according to whether one or more of the eye movement, the head posture, the body posture and the voice data are abnormal;
the synthesized video signal in S4 is a video signal obtained by synthesizing the video signal near the eye position shot by the micro camera and the video signal of the face and body shot by the camera in step S2, the synthesizing step is realized by an image processing and synthesizing module connected to the processor, the image processing and synthesizing module is further configured to uniformly compress the synthesized video signal into the same resolution for processing, and then send the compressed synthesized video signal to a remote server for processing through the processor;
the remote server in the S4 is used to combine the video in the non-enhanced other student terminals with the enhanced video data to convert into virtual scene data;
enhancing the face features by further reconstructing on the basis that the video enhancement in the S4 substantially retains the resolution of the original video; the reconstruction enhancement comprises processing modes such as contrast, saturation, brightness, video expansion, color art or microspur and the like.
The virtual scene data in S5 is implemented by a virtual scene conversion module, and the virtual scene conversion module can perform unified processing on the received composite video signals of all teachers and students, and perform enhanced reconstruction on the composite video signals of the corresponding students according to the received attention signal, where the composite video signals of the teachers are placed at "lecture stage" positions in the virtual scene, and the composite video signals of the students are arranged at "desk positions".
The invention also provides another technical scheme: a remote teaching system comprises a camera, a processor, a VR module and an image processing and synthesizing module, wherein the camera, the VR module and the image processing and synthesizing module are all connected with the processor;
the processor is wirelessly connected with a remote server through a 5G module;
the VR module comprises a miniature camera connected with the processor, a voice module used for recording voice data of students or teachers and an MEMS gyroscope;
the processor is also connected with a camera for shooting body image data of students or teachers.
The image processing and synthesizing module is used for synthesizing video signals near the eye positions shot by the micro camera and video signals of the face and the body shot by the camera, the image processing and synthesizing module is also used for compressing the synthesized video signals into the same resolution ratio for processing in a unified mode, then sending the compressed synthesized video signals to a remote server for processing through a processor, and the image processing and synthesizing module can be completed through a DSP chip.
The remote server receives the compressed composite video signal transmitted by the student terminal or the teacher terminal and receives the uncompressed composite video signal transmitted by the student terminal or the teacher terminal according to the attention degree condition; the remote server comprises a video enhancement module, and the video enhancement module carries out reconstruction enhancement on a video signal needing enhancement; the remote server further comprises a virtual scene conversion module, wherein the virtual scene conversion module converts the synthesized video signals and the strengthened video signals in each student terminal or teacher terminal into virtual scene data, and then sends the virtual scene data to each student terminal or teacher terminal.
The method and the system can further process the video by collecting the behaviors of teachers or students and analyzing whether the behaviors meet the requirement of attention or not so as to strengthen individual videos to improve the immersion degree and meet the actual attention rule of human beings to the surrounding classroom environment, and the video information recorded by the student terminals or the teacher terminals can be compressed when the attention is not needed so as to reduce the data pressure of a server and a VR module, so that the VR scene can be more smoothly rendered.
Drawings
FIG. 1 is a flow chart of a remote teaching method;
FIG. 2 is a schematic diagram of a distance teaching system.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
As shown in fig. 1, the present embodiment provides a remote teaching method, which includes the following steps:
s1, a teacher sets a teaching mode, wherein the teaching mode comprises a teaching mode, a discussion mode and a question asking mode;
s2, comprehensively analyzing behavior data of students, wherein the behavior data comprises eye movement, head posture, body posture and voice data, and judging whether to trigger an attention signal;
s3, if the attention signal is triggered, sending the attention signal to the remote server 6;
s4, the remote server 6 reconstructs and reinforces the composite video signal of the student triggering the attention signal and combines the composite video signal into virtual scene data;
and S5, the remote server 6 sends the virtual scene data to each student terminal and/or teacher terminal for display.
As a preferred embodiment, the teaching mode setting in S1 is set in an administrator mode, which can control whether and how student data in the remote server 6 is transmitted, and has a function of muting overall or banning individual;
as a preferred embodiment, in the teaching mode, audio and video data in the teacher terminal are synchronously transmitted, and in the virtual scene data, the teacher terminal is in a protruding position, and can control whether audio and video data of the student terminals are collected, whether a speech-inhibiting function is started, and the like, and can control the remote server 6 to always strengthen the video information of the teacher.
As a preferred implementation mode, in the discussion mode, audio and video data of the student terminal and the teacher terminal are synchronously acquired;
in the questioning mode, the student terminals acquire whether the students request questioning, and the teacher terminal controls whether to accept or not to accept questioning of which students;
in a preferred embodiment, the eye movement and the head posture in the behavior data in S2 are acquired by the micro-camera 1-1 and the MEMS gyroscope 1-2 in the VR module 1;
in a preferred embodiment, the body posture and voice data in S2 are acquired by a camera 2 for shooting body image data of students or teachers and voice modules 1-3;
as a preferred embodiment, the comprehensive analysis in S2 is to determine whether the student wants to participate in the discussion or answer the question according to whether one or more of the eye movement, the head posture, the body posture and the voice data are abnormal;
as a preferred embodiment, the synthesized video signal in S4 is a video signal obtained by synthesizing the video signal near the eye position shot by the micro-camera 1-1 and the video signal of the face and body shot by the camera 2 in step S2, the synthesizing step is realized by an image processing and synthesizing module 4 connected to the processor 3, the image processing and synthesizing module 4 is further configured to compress the synthesized video signals into the same resolution processing, and then send the compressed synthesized video signal to the remote server 6 through the processor 3 for processing;
as a preferred embodiment, the remote server 6 in S4 is configured to combine the video in the non-enhanced other student terminals with the enhanced video data to convert into virtual scene data;
in a preferred embodiment, the video enhancement in S4 further enhances the reconstruction of the facial features on the basis of substantially preserving the resolution of the original video; the reconstruction enhancement comprises processing modes such as contrast, saturation, brightness, video expansion, color art or microspur and the like.
As a preferred embodiment, the virtual scene data in S5 is implemented by a virtual scene conversion module, which is capable of uniformly processing the received composite video signals of all teachers and students and performing enhanced reconstruction of the composite video signals of the corresponding students according to the received attention signal, wherein the composite video signals of the teachers are placed at the "lecture stage" position in the virtual scene, and the composite video signals of the students are arranged at the "desk position".
Example 2
As shown in fig. 1, the present embodiment provides a remote teaching system, which includes a VR module 1, a camera 2, a processor 3, and an image processing and synthesizing module 4, where the camera 2, the VR module 1, and the image processing and synthesizing module 4 are all connected to the processor 3;
the processor 3 is wirelessly connected with a remote server 6 through a 5G module 5;
the VR module 1 comprises a micro camera 1-1 connected with the processor 3, an MEMS gyroscope 1-2 and a voice module 1-3 used for recording voice data of students or teachers;
the processor 3 is also connected with a camera 2 for shooting body image data of students or teachers.
As a preferred embodiment, the image processing and synthesizing module 4 is configured to synthesize a video Signal near the eye position captured by the micro camera 1-1 and a video Signal of the face and body captured by the camera 2, the image processing and synthesizing module 4 is further configured to compress the synthesized video signals into the same resolution processing, and then send the compressed synthesized video signals to the remote server 6 through the processor 3 for processing, and the image processing and synthesizing module 4 may be implemented by a dsp (digital Signal processing) chip.
In a preferred embodiment, the remote server 6 receives the compressed composite video signal transmitted from the student terminal or the teacher terminal, and receives the uncompressed composite video signal transmitted from the student terminal or the teacher terminal according to the attention degree; the remote server 6 comprises a video enhancement module, and the video enhancement module carries out reconstruction enhancement on a video signal needing enhancement; the remote server 6 further includes a virtual scene conversion module, and the virtual scene conversion module converts the synthesized video signals and the enhanced video signals in each student terminal or teacher terminal into virtual scene data and then sends the virtual scene data to each student terminal or teacher terminal.
According to the invention, the video is further processed by collecting the behaviors of teachers or students and analyzing whether the behaviors meet the requirement of attention or not so as to strengthen individual videos to improve the immersion degree, the actual attention rules of human beings to the surrounding classroom environment are met, and the video information recorded by student terminals or teacher terminals can be compressed when the attention is not needed so as to reduce the data pressure of a server and a VR module 1, so that a VR scene can be more smoothly rendered.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (10)
1. A remote teaching method is characterized in that: the method comprises the following steps:
s1, a teacher sets a teaching mode, wherein the teaching mode comprises a teaching mode, a discussion mode and a question asking mode;
s2, comprehensively analyzing behavior data of students, wherein the behavior data comprises eye movement, head posture, body posture and voice data, and judging whether to trigger an attention signal;
s3, if the attention signal is triggered, sending the attention signal to a remote server;
s4, the remote server reconstructs and strengthens the composite video signal of the student triggering the attention signal and combines the composite video signal into virtual scene data;
and S5, the remote server sends the virtual scene data to each student terminal and/or teacher terminal for display.
2. The method of claim 1, wherein: the teaching mode setting in S1 is set in the administrator mode, which can control whether and how student data is transmitted in the remote server, and has a function of muting overall or banning individual words.
3. The method of claim 2, wherein: and the eye movement and the head posture in the behavior data in the S2 are acquired by a micro camera and a MEMS gyroscope in the VR module.
4. The method of claim 3, wherein: the comprehensive analysis in S2 is to determine whether the student wants to participate in the discussion or answer the question according to whether one or more of the eye movement, the head posture, the body posture and the voice data are abnormal.
5. The method according to any one of claims 1-4, wherein: the synthesized video signal in S4 is a video signal obtained by synthesizing the video signal near the eye position shot by the micro camera and the video signal of the face and body shot by the camera in step S2, the synthesizing step is realized by an image processing and synthesizing module connected to the processor, the image processing and synthesizing module is further configured to uniformly compress the synthesized video signal into the same resolution for processing, and then send the compressed synthesized video signal to a remote server for processing through the processor.
6. The method of claim 5, wherein: enhancing the face features by further reconstructing on the basis that the video enhancement in the S4 substantially retains the resolution of the original video; the reconstruction enhancement includes at least one of contrast, saturation, brightness, video magnification, colorization, or macro-sizing.
7. A distance teaching system to which the method according to any one of claims 1-6 is applied, characterized in that: the system comprises a camera, a processor, a VR module and an image processing and synthesizing module, wherein the camera, the VR module and the image processing and synthesizing module are all connected with the processor; the processor is wirelessly connected with the remote server through the 5G module.
8. The system of claim 7, wherein: the VR module comprises a micro camera connected with the processor, a voice module used for recording voice data of students or teachers and an MEMS gyroscope.
9. The system of claim 8, wherein: the image processing and synthesizing module is used for synthesizing video signals near the eye positions shot by the micro camera and video signals of the face and the body shot by the camera, and is also used for compressing the synthesized video signals into the same resolution ratio for processing, and then sending the compressed synthesized video signals to the remote server for processing through the processor.
10. The system of claim 8, wherein: the remote server receives the compressed composite video signal transmitted by the student terminal or the teacher terminal and receives the uncompressed composite video signal transmitted by the student terminal or the teacher terminal according to the attention degree condition; the remote server comprises a video enhancement module, and the video enhancement module carries out reconstruction enhancement on a video signal needing enhancement; the remote server further comprises a virtual scene conversion module, wherein the virtual scene conversion module converts the synthesized video signals and the strengthened video signals in each student terminal or teacher terminal into virtual scene data, and then sends the virtual scene data to each student terminal or teacher terminal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110618197.5A CN113296609A (en) | 2021-06-03 | 2021-06-03 | Immersive remote teaching method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110618197.5A CN113296609A (en) | 2021-06-03 | 2021-06-03 | Immersive remote teaching method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113296609A true CN113296609A (en) | 2021-08-24 |
Family
ID=77326932
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110618197.5A Withdrawn CN113296609A (en) | 2021-06-03 | 2021-06-03 | Immersive remote teaching method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113296609A (en) |
-
2021
- 2021-06-03 CN CN202110618197.5A patent/CN113296609A/en not_active Withdrawn
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107911644B (en) | Method and device for carrying out video call based on virtual face expression | |
| CN102006453B (en) | Superposition method and device for auxiliary information of video signals | |
| CN102984496B (en) | The processing method of the audiovisual information in video conference, Apparatus and system | |
| CN109814718A (en) | A kind of multi-modal information acquisition system based on Kinect V2 | |
| EP4099709A1 (en) | Data processing method and apparatus, device, and readable storage medium | |
| CN112235585B (en) | Live broadcasting method, device and system for virtual scene | |
| CN113592985B (en) | Method and device for outputting mixed deformation value, storage medium and electronic device | |
| CN111405234A (en) | Video conference information system and method with integration of cloud computing and edge computing | |
| EP2574050A1 (en) | Method, apparatus and remote video conference system for playing audio of remote participator | |
| WO2019214370A1 (en) | Multimedia information transmission method and apparatus, and terminal | |
| CN113012500A (en) | Remote teaching system | |
| CN111651632A (en) | Method and device for outputting voice and video of speaker in video conference | |
| CN115515016B (en) | Virtual live broadcast method, system and storage medium capable of realizing self-cross reply | |
| CN114040318A (en) | Method and equipment for playing spatial audio | |
| CN110609619A (en) | Multi-screen live broadcast interactive system based on panoramic immersion type teaching | |
| CN114286021B (en) | Rendering method, rendering device, server, storage medium, and program product | |
| JP2002091291A (en) | Data communication system for piano lesson | |
| US10468029B2 (en) | Communication terminal, communication method, and computer program product | |
| CN110933350A (en) | Electronic cloud mirror recording and broadcasting system, method and device | |
| CN115220682A (en) | Method and device for driving virtual portrait by audio and electronic equipment | |
| CN113296609A (en) | Immersive remote teaching method and system | |
| CN110602523A (en) | VR panoramic live multimedia processing and synthesizing system and method | |
| CN113012501B (en) | Remote teaching method | |
| DE102007010662A1 (en) | Method for gesture-based real time control of virtual body model in video communication environment, involves recording video sequence of person in end device | |
| CN115311919A (en) | Spoken language training method and system based on VR technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210824 |