Disclosure of Invention
The invention aims to provide a remote real-practice teaching method and system based on augmented reality projection interaction, and aims to solve the problem that the conventional remote real-practice teaching method is difficult to provide an intuitive, easy-to-use and convenient teaching mode.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
in a first aspect, an embodiment of the present invention provides a remote real-practice teaching method based on augmented reality projection interaction, including the following steps:
step (1), erecting a projector and a camera in a student real-operation environment at a student end, and calculating a relative position relation between a projection coordinate system and a camera coordinate system, wherein the method specifically comprises the following substeps:
step (1.1), adjusting the positions of a projector and a camera according to the actual operation environment of students, so that a projection area and a video shooting area are overlapped and cover the actual operation area;
step (1.2), after the position of a projector and a camera are fixed, the projector projects a preset image I to an actual operation area;
step (1.3), acquiring a scene picture containing the projected image I through a camera, and recording the scene picture as an image F;
step (1.4), calculating a transformation relation matrix H between the images I and F to obtain the relative position relation between a projection coordinate system and a camera coordinate system;
step (2), remote communication between the student end and the teacher end is established through the server;
step (3), after the remote communication is established, the pictures of the real exercise area and the voices of the students are sent to the teacher end in real time through the student end;
step (4), after receiving the picture of the real exercise area, the teacher displays the real exercise environment observed by the student terminal, and the teacher directly adds virtualized guidance information in the area of interest in the displayed real exercise environment; the teacher end simultaneously sends the voice and the interactive guidance information of the teacher to the student end in real time;
and (5) after receiving the virtualized guidance information of the teacher, the student converts the position of the guidance information from a camera observation picture coordinate system to a projection picture coordinate system by transforming a relation matrix H, and projects the position to the real operation area by a projector.
Further, still include:
and (6) recording all video, audio and interactive guidance information data transmitted in the remote teaching at the same time when the server end completes the communication between the student end and the teacher end, and storing the video, audio and interactive guidance information data into a database for data playback and teaching evaluation after the course is finished.
Further, still include:
and (7) repeating the steps (3) to (6) to form a closed loop in the remote teaching process until the course is finished or any party is disconnected.
In a second aspect, an embodiment of the present invention provides a remote real-practice teaching system based on augmented reality projection interaction, where the system is configured to execute the remote real-practice teaching method based on augmented reality projection interaction in the first aspect, and the system includes:
a teacher end, the teacher end comprising: the system comprises a display module for displaying the student end actual operation process, an interaction module for teachers to conduct interaction guidance, a first data acquisition module for acquiring the teacher to conduct the interaction guidance process, and a first network communication module for remotely receiving and sending data, wherein the interaction module conducts interaction based on the student end actual operation process provided by the display module, teachers directly conduct interaction in displayed actual operation pictures, and interaction guidance information is provided at any position specified by the teachers;
a student end, the student end comprising: the system comprises a first data acquisition module, a projection display module, a first network communication module and a processing module, wherein the first data acquisition module is used for acquiring the real-time operation process of students, the projection display module is used for projecting the interaction guide information of teachers in an augmented reality mode in a real-time operation environment, the first network communication module is used for remotely receiving and sending data, and the processing module is used for calculating the relative position relation between a projection coordinate system and a camera coordinate system; the processing module enables the guide information of the teacher to be accurately projected to a position appointed by the teacher during interaction, and fusion of virtual information and a real operation environment is formed;
a server side, the server side comprising: the server is used for realizing remote connection and data transmission between the first network communication module and the second network communication module, and the database is used for storing communication data.
Furthermore, the first data acquisition module at least comprises audio acquisition and interactive acquisition, acquires real-time voice and interactive guidance information of a teacher, and sends the real-time voice and interactive guidance information through the first network communication module.
Furthermore, the second data acquisition module at least comprises audio acquisition and video acquisition, acquires real-time voice of students and monitoring video of an actual operation area, and sends the real-time voice and the monitoring video through the second network communication module.
According to the technical scheme, the method has the advantages that a teacher can visually give remote guidance information in a real exercise area of a student, so that a complex multimedia teaching plan design is avoided, the teacher can quickly master a use method and develop teaching, visual, convenient and cheap projection type augmented reality real exercise teaching experience is provided for the student, and the problems of narrow vision field, dizziness, glasses overlapping and the like caused by using an intelligent helmet and other similar devices are avoided while the student quickly understands the guidance of the teacher. The invention is simple, easy to use and visual, and provides a new solution for remote practice teaching.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail and completely with reference to the following specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Example 1:
referring to fig. 1, the present embodiment provides a remote real-practice teaching system based on augmented reality projection interaction, which includes:
teacher 001, comprising: the system comprises a display module for displaying the student end actual operation process, an interaction module for teachers to conduct interaction guidance, a first data acquisition module for acquiring the teacher to conduct the interaction guidance process, and a first network communication module for remotely receiving and sending data, wherein the interaction module conducts interaction based on the student end actual operation process provided by the display module, teachers directly conduct interaction in displayed actual operation pictures, and interaction guidance information is provided at any position specified by the teachers;
specifically, the teacher end is composed of a notebook computer, a display screen of the notebook computer is a display module, a keyboard, a mouse and a touch pad of the notebook computer are interaction modules, a computer program and a sound card for recording man-machine interaction operation and teaching voice of the teacher are first data acquisition modules, and a wireless network card is a first network communication module;
a student end 003 comprising: the system comprises a first data acquisition module, a projection display module, a first network communication module and a processing module, wherein the first data acquisition module is used for acquiring the real-time operation process of students, the projection display module is used for projecting the interaction guide information of teachers in an augmented reality mode in a real-time operation environment, the first network communication module is used for remotely receiving and sending data, and the processing module is used for calculating the relative position relation between a projection coordinate system and a camera coordinate system; the processing module enables the guide information of the teacher to be accurately projected to a position appointed by the teacher during interaction, and fusion of virtual information and a real operation environment is formed;
specifically, the student end comprises a projector, a camera, a bracket for fixing the projector and the camera, and a notebook computer. The camera connected with the notebook computer and the video and audio acquisition software in the notebook computer are second data acquisition modules, the projector connected with the notebook computer and the projection control software in the notebook computer are projection display modules, and the wireless network card carried by the notebook computer is a second network communication module; the processor of the notebook computer is used as a processing module.
The server 005 includes: the server is used for realizing remote connection and data transmission between the first network communication module and the second network communication module, and the database is used for storing communication data;
specifically, in this embodiment, the server is directly hung on the teacher end, and the notebook computer of the teacher end also serves as the server, directly communicates with the student end, and stores the communication data in the local database.
Example 2:
referring to fig. 2, the present embodiment provides a remote real-practice teaching method based on augmented reality projection interaction, where the method includes the following steps:
a projection correction step S101, namely erecting a projector and a camera in the student real operation environment at the student end, and calculating the relative position relation between a projection coordinate system and a camera coordinate system, and specifically comprising the following substeps:
1.1 adjusting the positions of the projector 301 and the camera 303 according to the environment of the student in actual operation and fixing the projector 301 and the camera 303 through the bracket 305, so that the projection area 311 and the video shooting area 313 are basically overlapped and cover the real operation area 315, and the space relationship among the projector 301, the camera 303 and the real operation area 315 is shown in fig. 3;
1.2 after fixing the position of the projector and the camera by using a bracket and other equipment, the projector projects a preset image I to an actual operation area, wherein in the example, a picture (marked as the image I) with a official website two-dimensional code of Zhejiang electromechanical occupational technology institute is projected by a notebook computer at a student end;
1.3, acquiring a scene picture containing the two-dimensional code pattern through a camera, namely acquiring an image containing a Zhejiang electromechanical technical institute official website two-dimensional code and recording the image as an image F;
1.4 identifying and extracting three positioning graphs and a correction graph of the two-dimensional code in the image I and the image F by adopting an algorithm of an OpenCV software library, calculating the coordinate positions of the central points of the four frame-shaped graphs in the image I and the image F respectively, determining the clockwise sequence of the four central points by judging the left and right regions of a diagonal line, and then calculating the two groups of point sets by adopting a homography solving method provided by OpenCV to obtain a transformation relation matrix H between the image I and the image F, namely a coordinate system position transformation relation between a projection picture and a camera observation picture. And finally, storing the transformation relation matrix H into a file form for subsequent calling.
The projection correction step is the preprocessing and preparation of the subsequent steps, on one hand, the spatial relation between the devices is fixed, and on the other hand, the spatial synchronization of the monitoring picture and the projection picture is realized, namely, the deviation between the projection area and the shooting area shown in the figure 3 is compensated through calculation, so that the guide information of the teacher can be accurately projected to the corresponding position, and the function of augmented reality is completed.
Therefore, the design of the invention has the advantages that the requirement on the fixed position of the equipment can be reduced, and the camera and the projector do not need to be conjugate and coaxial or have strict relative position and posture, so that the usability of the equipment at the student end is enhanced during deployment, and the equipment can be quickly built without professionals. On the other hand, the invention corrects the parallax by the software compensation mode of the projector self-projection, and does not need to use additional solid markers such as cards, stickers and the like, thereby automating the correction process and being more beneficial to the rapid configuration and use of the system.
And a communication establishing step S103, establishing network communication between the teacher end and the student end through the server, and realizing bidirectional data receiving and sending between the first network communication module and the second network communication module.
Specifically, in this embodiment, the teacher-side computer also serves as a server, and four threads are respectively started at the teacher side and the student side to respectively serve as a video sending service, an audio sending service, a video receiving service, and an audio receiving service. And the audio and video are bound with IP by Socket, and the networking is monitored and completed. Firstly, a teacher end starts a remote IP connection application of audio and video sending service, if the audio and video sending service is normally connected, audio and video acquisition information transmission is started, and if the audio and video sending service cannot be normally connected, the teacher end waits for 3 seconds to continuously try connection until the audio and video sending service is normally connected. If the connection cannot be normally performed, the remote connection operation can be finished by forced quitting. And after the teacher end completes the connection of the audio and video sending service, waiting for the connection application of the student end to the local receiving service. The student end also starts four threads, and the operation steps are consistent with those of the teacher end.
And a data acquisition step S105, acquiring and sending data at the student side. After the remote communication between the student end and the teacher end is established, the student end starts video and voice acquisition and sends the pictures of the real exercise area and the voices of the students to the teacher end in real time.
Specifically, the student side starts video and voice acquisition through software libraries such as OpenCV and pyAudio, and sends real-time audio and video contents to the teacher side.
And an interactive guidance step S107, wherein the teacher displays, interactively guides and transmits. After receiving the real-time operation picture, the teacher end displays the real-time operation environment observed by the student end camera in a 2D or 3D mode through the display module, so that the teacher can directly add virtualized guidance information to the region of interest through the interaction module in the displayed real-time operation environment; and the teacher end simultaneously starts voice acquisition and interactive acquisition and sends the voice and interactive guidance information of the teacher to the student end in real time.
Specifically, after receiving the real-time real operation picture sent by the student, the teacher opens a screen window through the OpenCV software library to display the real operation picture observed by the camera. When the teacher conducts interactive guidance, the teacher directly uses the mouse to draw interactive contents on an observed real operation picture, records the moving track of a mouse cursor through the OpenCV software library, marks track points in red and displays the track points in the real operation picture. In the process of drawing by the teacher, all red track points are independently extracted into an image R, the image is consistent with the observed real operation picture in size, and the image R is sent to the student side. Meanwhile, the teacher can conduct interactive guidance through voice at any time and send the interactive guidance to the student end through the network.
The step design has the advantages that teachers can visually mark observed real operation pictures without designing multimedia materials or animation conversion tools in advance, and therefore usability is greatly improved.
And a projection display step S109, wherein the student side performs projection display. After receiving the virtual guide information of the teacher end, the student end converts the position of the guide information from a camera observation picture coordinate system to a projection picture coordinate system by transforming a relation matrix H, and projects the guide information to an actual operation area through a projector, so that virtual and actual registration of the virtual guide information and a real actual operation environment is realized, the interactive guide information of the teacher is accurately projected to an area of interest originally observed by the teacher, and the remote actual operation teaching based on augmented reality is completed.
Specifically, the student side receives a guide image R and interactive voice sent by the teacher side through the second network communication module, converts the image R based on the camera coordinate system into an image under a projection coordinate system through a perspective transformation function and a transformation relation matrix H of the OpenCV software library while playing the voice, and projects the image R to the real operation area through the projector to realize virtual and real registration of virtual guide information and a real operation environment.
Fig. 4 shows the projection teaching contents actually observed by the student side, the teacher draws a triangle around the device to be operated through the teacher side interface, and draws an X at the position where the device should be placed, so that the student side can intuitively and accurately observe the prompt of the teacher at the same position and improve the teaching efficiency in cooperation with the guidance of voice.
The step design has the advantages that the interactive guide information of the teacher can be accurately projected to the remote interactive position of the teacher in real time through the method, so that students can intuitively and accurately understand the intention of the teacher, the remote real-time practice teaching based on the augmented reality is realized, and the teaching quality and the teaching efficiency are improved.
Server data management step S111. And the server side records all data such as video, audio, interactive guidance information and the like transmitted in the remote teaching while finishing the communication between the student side and the teacher side, and stores the data in a database for data playback and teaching evaluation after the course is finished.
Specifically, in this embodiment, the server is built at the teacher end, and the data management is also completed at the teacher end. And the server side records all data such as video, audio, interactive guidance information and the like transmitted in the remote teaching while finishing the communication between the student side and the teacher side, and stores the data in a database for data playback and teaching evaluation after the course is finished.
And circularly executing the steps. And repeating the steps S105 to S111 to form a closed loop of the remote teaching process until the course is finished or any party is disconnected.
Specifically, the closed loop real-time execution of steps S105 to S111 is repeated, so that the information of the teachers and the students can be exchanged in real time, and the remote lecture is realized. If the course is finished or one party exits the course, the connection is disconnected.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.