WO2020045837A1

WO2020045837A1 - Method for smart-remote lecturing using automatic scene-transition technology having artificial intelligence function in virtual and augmented reality lecture room

Info

Publication number: WO2020045837A1
Application number: PCT/KR2019/009449
Authority: WO
Inventors: 김영대
Original assignee: 김영대
Priority date: 2018-08-28
Filing date: 2019-07-29
Publication date: 2020-03-05
Also published as: KR20200024441A

Abstract

Compared to a lecture apparatus according to the prior art, the present invention provides a feature, wherein an operation of an apparatus, that is, scene transition, is automatic and based on an artificial intelligence (AI) function so as to allow a lecturer to focus on a lecture, and a feature of an edge processing scheme based on AR composition without performing chroma keying in a studio environment confined to performing a chroma-keying technique, thereby providing a lecture apparatus based on an AR lecture room method. Accordingly, a scene for delivering a more immersive lecture compared to the existing lecture method may be created in any classroom or lecture location such that a technique may be provided of perfectly implementing a real-time lecture apparatus, a real-time lecture recording apparatus, a real-time remote lecture, and a lecture presentation conference apparatus, which allow a lecture to be delivered in a lecture location, based on the method of the present invention.

Description

Smart-remote lecture method and its device using automatic cutaway technology of AI function in virtual and augmented reality classrooms

The present invention relates to a smart-remote lecture method and apparatus using an automatic scene change technology of an artificial intelligence function in virtual and augmented reality classrooms, and more particularly, instructors can use the AR chroma key in comparison with a conventional lecture apparatus. Smart lecture apparatus and method using automatic scene change technology that can greatly improve the efficiency of space and time by realizing a similar VR environment that can automatically switch scenes to highlight areas that you want to emphasize The present invention relates to a method and apparatus for real-time lecture recording and remote lecture, which is used in conjunction with a conventional video conference.

Conventional lecture apparatuses have been developed in which a projector, a blackboard, and a classroom-based electronic board are used to control lecture materials. With the recent development of the latest VR, AR, and MR technologies, and the popularization of smart devices such as Wi-Fi, many lecture recordings are being made using various existing broadcasting technologies such as classroom lecture recording, e-learning, and internet lectures. The methods using these techniques still require less time and editing than recorded lectures in actual classes, which is why there is no real-time remote lecture, ie Tele Presentation technology or equipment. It is true.

1 shows an example of a conventional lecture recording apparatus.

Referring to FIG. 1, when directly recording with a camera, a problem in which a blackboard is not seen clearly and a viewer who views a video of a lecturer and a material when making a composite screen do not concentrate on a scene to be viewed on a screen are generated.

The distance learning-e-learning and MOOC courses through these videos are recently focused on the smart device Internet, and their technology is being made in three ways.

The first method is to record lecture data only, screen or PPT screen and record voice. More than 80% of the Internet lectures are invisible in this way. It consists of a simple material screen and a greening lecture of a pen. Khan's Academy is a prime example. This is a simple and comfortable way to learn extracurricular meals focused on how to understand the problem rather than the motivation of the instructor's facial training.

Secondly, the e-learning method in which the instructor such as the USB camera of the instructor is shown in the corner on the first screen, which is a method such as WebEx and Connect Pro, which are frequently used in webinars. This method secures the first part, but the small part of the material screen and the instructor continue to exist in the same position, and eventually they do not see the instructor.

The third method, which requires a broadcast technician rather than a self-shooting technique, is a multi-camera switching method based on a so-called switcher mixer that alternates lecture materials, sometimes lecture boards, sometimes composite screens. As a general broadcasting technology that uses broadcasting and editing methods, a classroom class is used by professionals and specialized equipment.

However, the lecture synthesis scene method through the lecture synthesizer and the existing switcher mixer lecture system has a problem in the switching of the recording screen and the scene configuration. Therefore, in the classroom, problems such as teachers and blackboard projectors need to be distracted to determine where the students are concentrated.

This means that general education broadcasting and internet lecture companies take lecture class shooting methods, which are used as a lecture studio shooting editing method of making a studio set without taking classes in the classroom. These studios are widely used for production using actual sets, virtual studio methods, or editing after shooting in green screen.

In recent years, with the creation of Trackless Virtual Studio technology, many virtual studio companies around the world are using lecture production methods using this technology to create various camera effects with just one instructor camera.

In addition, in recent years, in order to minimize the number of shooting personnel in order to automatically shoot the broadcast switching technology in the classroom, the lecturer tracking camera in the classroom is inexpensively supplied, and the classroom recording is required in many classrooms. Automated class recording technology that combines mixer and instructor tracking-PIP or mixer method of recording lecturers and lecture materials has recently been applied to many classrooms.

However, it is difficult for an expert to judge the problem of switching a scene to display a lecturer's expression and a problem that a large portion of the synthesis screen is difficult to use as a recording and transmission method of a lecture using the professional instructor tracking and synthesizing apparatus of FIG. 1. Most of the time, the recording is done in one scene, and the scene camera is not beautiful because of lighting problems in the classroom.

The remote lessons or lectures that take these lectures through the Internet and take lectures have recently emerged suddenly on the Internet and smartphones all over the world. With the move and real-time tele-classes, real-time tele-lectures and presentation meetings being required, most of the Internet can be used to send screens to video conferencing in a variety of ways combined with remote video transmission technology. In the video conferencing system Skype, Hangout, etc., a method of alternately sending computer screens and instructors has been used. In FIG. 1, WebEx, which uses screens, is used as a so-called Webinar method, and a dedicated presentation that makes them more professional. TV with so-called "TELE-Presence" for presentation A screen and a projector presentation with two different screens in the way of teaching in general remote video lecture / presentation system has been used in general coming technologies.

However, all of the methods of taking a remote class are different from those of the actual classroom, so it is recognized that the actual class must go to the classroom and the instructor must always be in the classroom.

Professional broadcast equipment of these broadcasting devices should be used, and even though such equipment is used, it is not possible to use them in general classrooms because it is a way that professionals should be mobilized in a specially designed lecture hall in a studio.

Accordingly, there is a need for development of a virtual studio that is more convenient and implements a similar VR environment without using a conventional AR key chroma by supplementing the problems of the conventional lecture system described above. .

The present invention aims to solve the problems of the prior art as described above and the technical problems that have been requested from the past.

After the inventors of the present application have conducted extensive research and various experiments, the smart lecture apparatus and the remote lecture method and apparatus according to the present invention have similar effects by combining special effects and multiple layers of 2D / 3D as described below. You can use these techniques to create a virtual classroom in a more specific and convenient way using the method that occurs, and to create a similar VR environment without using AR key chroma key, and above all, various scene transitions of these methods. Scenes showing the lecturer in a virtual or augmented reality classroom or scenes centered on presentation materials are automatically converted into scenes that students should see by recognizing the speaker's voice, gesture, location, and pointer environment. To select a scene in an AI way Manner so that students have a remote device to a lecture or a lesson recording device and sends it to look the way that focused only synthetic screen. Thus, the device of the present invention solves the problem of broadcasting equipment, which requires the operator of the equipment by inventing a smart PD function that allows the device to change scenes like a broadcasting PD function for selecting a broadcast scene without a separate equipment operator. To be an automatic lecturer. In other words, the instructor focuses on the lecture and the automatic scene transition that students should see automatically creates the VR classroom scene where the instructor highlights and the virtual monitor or virtual copyboard in the virtual classroom that simultaneously performs the function of the virtual copyboard. Real-time VR / AR lecture device using automatic scene change technology that integrates the technology to make a video, real-time lecture recording device to record the output screen of the device, and smart-remote lecture method and device that sends the screen to a remote place for lecture To provide.

In addition, the lecture method for showing the lecturer and the lecture material in the virtual classroom of the VR / AR technology according to the present invention, the lecture recording device for recording it, the real-time remote VR / AR lecture method and apparatus for showing these scenes to the remote place Is a new teaching method and a new teaching device that radically changes the teaching method of the classroom, which makes classroom teaching more efficient using the blackboards, tables, electronic blackboards, and electronic teaching tables used in the existing classrooms. In addition, the real-time recombination screen of the VR / AR virtual classroom is shown on the screen instead of the PPT screen on the screen of the past method as a lecture showing the VR / AR virtual classroom lecture scene of the main unit of the classroom where the real teacher lectures. Lectures are given in a more immersive way, giving the teacher and the whole class eye contact. In the same way, you will be able to watch the same lecture on the remote place and be able to listen to smart devices around the world in the same way. To provide a new method and device for solving the fundamental problem that listening to remote video lessons that everyone knows is no better than listening to an instructor in the classroom.

Smart remote lecture method using the automatic scene change technology according to the present invention for achieving this object,

(a) Virtual desks, floors, blackboards, ceilings, and classrooms are created by constructing virtual classrooms of 2D and 3D graphics that enable the synthesis of lecturers and lecture materials in virtual classrooms by means of graphics and VR for AR. A virtual classroom making and selecting step of outputting at least one of the ornaments;

(b) a plurality of virtual cameras for selectively outputting a plurality of virtual classroom scenes by arranging a plurality of virtual cameras in which at least one of position, rotation, and field of view (FOV) is freely defined in the virtual classroom; Providing a deployment function;

(c) Cropping, edge frame-transparency processing method of the instructor camera to extract only the instructor by chroma key-depth method in the image of the real lecturer in the virtual classroom scene shown in the virtual camera or AR synthesis Create instructor characters in real time through composition, or face recognition of the instructor, and place the instructor images so that the instructor images are displayed in the required position on the output screen of the corresponding virtual camera. Providing a function capable of displaying the output screen photographed through the corresponding virtual camera so that the instructor is shown in the required position in the required size in a scene to be shown to the viewer;

(d) A plurality of presentation images / images in which at least one of an instructor's image, a desk, and a blackboard are to be independently displayed in the virtual classroom scene, so that the plurality of virtual copyboards can be freely positioned in the virtual classroom. Providing a function of freely arranging the virtual copyboard so that the instructor image of the virtual camera and the virtual chalkboards are displayed on the virtual camera as necessary scenes;

(e) setting the movement of at least one or more of a time and a path at which a camera track is moved to each virtual camera position when an output scene is changed and displayed as a necessary scene while the set scenes of the plurality of virtual cameras are lectured; In the process of providing the function, the instructor image arrangement and the position size of the virtual blackboards set in each virtual camera are synchronized with the camera movement so that each track is automatically generated so that each instructor with these virtual camera change track movements. Providing an automatic or manual track generation function such that the image and the virtual copyboard are transitioned together by natural animation;

(f) Switcher button setting that defines a plurality of cameras as virtual switcher buttons like buttons of general broadcast switcher and arranges them as tile button GUI, and defines ICON of these buttons as preview button to show the scene. Arranging such that the instructor can intuitively switch scenes using only a mouse or pointer button without looking at the GUI switching screen with only a mouse button through the process;

(g) setting GUI button switching of virtual copyboard contents to set content or external input cameras to be displayed on each virtual copyboard together with the button GUI switcher setting and to easily switch and display the contents of the corresponding copyboard; And

(h) An artificial intelligence PD function that remotely or automatically switches the function of the broadcasting PD that presses the selection switching buttons of the GUIs arranged in the plurality of virtual cameras according to the lecture situation, so that the instructor gestures, pointers, sounds, language, Various scene changes and virtual copyboards automatically by the automatic PD function that recognizes the scene by recognizing at least one or more of the voice, the direction of the gazing of the instructor's motion, and the students' voice and direction. It may include a step of automatically performing the content conversion to provide a function to output the lecture scene in real time.

In addition, the present invention,

(a) Synthesis of 2D or 3D graphics into a virtual classroom for video compositing where lecture materials such as lecturers, PPTs, etc. viewed by students in the classroom are received by input devices such as cameras and displayed in a virtual graphics classroom. Virtual or beautiful virtual desks, floors, blackboards, ceilings, or backgrounds created by creating virtual classrooms with 2D and 3D graphics that allow instructors and lecture materials to be synthesized in virtual classrooms using screen or VR. Augmentation classroom production and selection;

(b) a virtual camera in which virtual cameras for photographing various scenes are arranged in the virtual classroom so that output screens of a plurality of virtual classroom scenes are created by arranging a plurality of virtual cameras freely defining a position, rotation, and a field of view (FoV). Placing and setting the field;

(c) In the virtual classroom scene shown in each of the plurality of virtual cameras, the lecturers and lecture materials that each virtual camera should see for each scene are shown in the size and position required by the virtual camera as the students view the classroom lecturer and the lecture materials. The virtual lecturer's camera scene and the presentation material scene are arranged for each virtual camera so that the instructor image can be arranged so that the instructor can be shown at the required size on the output screen of the virtual camera. A scene setting step of each of the plurality of cameras, which allows the viewers / students of the actual classroom or the remote place to view the real-time synthesis scene of the device of the present invention and listen to the lecture, so that the necessary scenes are variously displayed;

(d) The lecture materials of each virtual classroom, namely PPT, video, picture, camera image, real imager camera, etc. are placed on the screen of the virtual copyboard which is placed in the virtual classroom, and the virtual pointer is provided to provide a virtual pointer. Or a virtual copyboard that provides a virtual pen that allows the pointer to be shown in the actual lecture scene while drawing and erasing when the actual pointer is moved and the plurality of virtual copyboards are independently inputted by a plurality of virtual cameras. Steps to set up lecture materials for each virtual copyboard to freely change and set up lecture materials

(e) The virtual camera track moves to each virtual camera position as the actual tracking camera (or like the camera movement of the crane) when the output scene is changed to the required scene while the set scenes of the plurality of virtual cameras are selected as the necessary scenes during the lecture. In this process, it is possible to set the time and path movement that the movement becomes natural in the virtual space. The track size of the instructor image set in each virtual camera and the position size of the virtual copyboards are synchronized with the camera movement to automatically track each track. Or create a preset track so that each virtual camera change track movement and each instructor image and virtual copyboard are transitioned to a natural animation together. Student with transition of screen Steps are provided automatically or manually track generation feature that allows the furnace to concentrate without hitting the scene to look natural and see; And

(f) a decision to select a corresponding virtual camera to show the necessary camera scene according to the situation of the plurality of virtual cameras, that is, a virtual disposed in the virtual classroom like a switcher that selects a camera signal connected to a general broadcast switcher; As a function of artificial intelligence function in the device of the present invention as a person, that is, a broadcast PD, the judgment of selecting a camera scene is a function of PD, and at least one or more recognition methods of a gesture, a pointer, a sound, a language, and a voice of an instructor It automatically performs various scene changes and virtual copyboard contents conversion by AI PD function that recognizes the scene by gazing direction instructor's view and students' sound and direction recognition. Providing a function to output the lecture scene in real time; may include.

In one preferred embodiment of the present invention, in the step (d), the mouse or the pointer is displayed on the output screen when the content output to the virtual copyboard is at least one of the plurality of cameras, notebooks, video lists and picture lists. When the virtual copyboard is instructed, the list is predefined so that external cameras, images, and images placed on the copyboard's contents list can be switched in order, and the following contents are selected through the mouse or the pointer to view the switcher menu. It provides a function that the playlist control can be performed for each corresponding virtual copyboard without having to change the contents or slides such as PPT; and the virtual virtual blackboard selection, characterized in that it further comprises One-touch access to copyboard content scenes with mouse / pointer controls In this case, the switching technology may include a function of automatically switching the scene to the corresponding scene while the pictures of the pointer drawn during the content switching are automatically deleted.

Here, in one preferred embodiment of the present invention, in the automatic scene change, that is, the automatic PD in the function of switching to automatically detect the stop if there is no movement of the pointer to increase the size of the instructor, PPT slide is changed When the virtual copyboard is spontaneously highlighted and becomes a phone in a scene that appears to be large, if the mouse or the pointer is constantly moving the presentation, the virtual copyboard grows somewhat or in a way that is time consuming on a boring screen. The method may further include providing and setting a timer function provided with a function for changing to a similar scene.

In addition, in one preferred embodiment of the present invention, in addition to the above-described automatic and pointer position or gesture recognition, the presenter can easily change the scene of each virtual camera while the presenter wants the scene easily by forcing the presenter to lecture. In order to set it as a touch, it is natural to control each scene through a control panel, and using a device such as a control panel is a broadcasting device that requires an expert. The standard template is provided to make it easy for the instructor to change this. Anyone can see this template by intuitively changing the scenes with the mouse button. If you press the right button, you can click the butt (virtual camera) in that row, and turn the wheel. Three virtual copyboards that can be used in general through the function of changing up and down to be manually adjusted only by clicking the mouse without looking at the menu are added to the virtual camera switcher, that is, the graphic control panel provided in the GUI of the present invention. Allows three or four scene layouts for each virtual copyboard to show virtual copyboards of varying sizes, automatically or manually change scenes, give lectures, change required virtual copyboards, and other presentations. In order to easily switch the function to show a lecture and show the lecturer, it is possible to provide a function to freely arrange the switcher scene, that is, the virtual cameras, with the switcher button of the graphical GUI.

At this time, the scene of each virtual camera is captured and shown as an icon on the virtual switcher button. Unlike the buttons on the general switcher, the preview button is provided on the button itself, and the output is immediately output without pressing the preview of the general switcher to check and output. Real-time lectures can be provided by providing the function to program out.

Further, in one preferred embodiment of the present invention, the arrangement of the virtual cameras of the virtual control panel, that is, the GUI arrangement of the present invention, can be freely determined and arranged on the icons of the switcher GUI in each scene setting step. It is natural that the function is provided. At this time, the virtual camera is arranged in the virtual classroom as if the cameras are placed left and right, and back and forth. The tiles can be arranged as tiles 1x4, 2x3, 3x4, 4x5, etc. In the vertical direction, the position of the camera, that is, the instructor is shown large or small, but the instructor is large. The scenes that make the boards largely arranged When the switcher operates to intuitively change from the current scene to the next scene, select and adjust the left and right virtual cameras by clicking the left or right mouse of the mouse, and use the mouse wheel to vertically move the virtual camera in the column of the corresponding copyboard. Changes can be made in a way that allows changes to be made in a plurality of copyboards, instructors of the copyboard part, and cameras that change the size of the copyboard. It is important to provide a selectable feature that is not intelligent.

In addition, in one preferred embodiment of the present invention, in addition to the function of arranging the scene switching control button in the GUI, it is important to provide a content switching function for changing the contents shown on the corresponding virtual copyboard. Various external inputs to be used in the lecture, such as student cameras, document camera inputs, files such as PPT, video and pictures, as well as various program screen captures, video of the other party can be selected and displayed. Provides the ability to freely select and place content switcher icons that select and order contents by each corresponding virtual camera and rotate their wheels by clicking the mouse's click button, left mouse button, or right mouse button. Press to wake up the content switching on that virtual copyboard It is possible to provide a one-touch function of content switching switching that provides a function for switching the corresponding content in a manner such as to remind.

In one preferred embodiment of the present invention, in step (b), in addition to the instructor and the virtual copyboard shown in the virtual camera in the virtual classroom, a 3D object is called to display the virtual object animation using real-time 3D render technology. It can be freely placed in each virtual camera scene to adjust its size and position, and its animation and interactive controls allow real-time 3D virtual experiments and experiences of heavy and large objects or very small DNS-like objects that cannot be experimented or experienced in actual classrooms. The method may further include controlling the learning using a joystick or a keyboard.

In one preferred embodiment of the present invention, in step (a), the virtual copyboard 3D is adapted to adjust the distortion of the virtual screen to a curved curve screen as well as adjusting the size of the virtual copyboard and cropping of unnecessary portions. The ability to adjust the degree of curvature of each virtual copyboard to show beautiful immersive virtual copyboards so that they can be placed in the scene of each virtual camera and the size of the frames of these virtual copyboards together Providing a function of allowing a curved surface and adjusting a frame color and size so that a beautiful virtual camera output screen can be created.

In addition, in one preferred embodiment of the present invention, each virtual copyboard constitutes a main scene of the output screen, and provides a virtual frame, that is, the outer frame of the copyboard as a variety of external graphics so that it can be beautifully displayed in the virtual classroom. It can be selected and used, and the ability to adjust the thickness, color, and transparency of these frames to provide the ability to adjust in real time to match the virtual classroom.

In addition, in one preferred embodiment of the present invention, such a frame function and the outer cropping (cutting) function are provided not only for the virtual blackboard, but also for the instructor's image, and the cropping of the instructor's image frame and the virtual blackboard is performed. Various shapes such as rectangle or ellipse can be selected, and the AR function is processed through the transparency function of the outer border area even if the frame is not selected so that the background and the virtual copyboard or the lecturer's image are infused. When combined with a virtual classroom or the instructor's camera input screen and virtual desk can be created to create a natural sense of unity.

In one preferred embodiment of the present invention, the virtual electronic blackboard includes an electronic pen function capable of marking by a mouse or a pointer, wherein at least one of thickness, color and transparency of the electronic pen is controlled and drawn by the pen function. A function may be provided that allows the part to be automatically erased when the copyboard content is changed or the PowerPoint scene is switched.

That is, the virtual electronic blackboard, like a pointer used in a general classroom to increase the concentration of lectures in the present invention, includes a virtual electronic pen function that can be marked by a mouse or a pointer, and among the thickness, color, and transparency of the electronic pen. A virtual graphics pointer that allows at least one to be controlled so that the mouse or gyro pointer can be visualized on the virtual copyboard, allowing the size of the virtual pointer graphic to be scaled and colored externally. Providing is an important function of the present invention of the virtual teaching method.

In one preferred embodiment of the present invention, the contents of the PPT slide notes when the scene of the PPT is switched when the PPT scene is imported and released using the device of the present invention in the classroom and the presentation room using the mouse or the pointer equipped with the gyro function. Provides an auto-lamper function that can be shown on this additional screen to be shown on the instructor's classroom screen or on the instructor's camera on a lamp or monitor, allowing the instructor to read and present what is being presented. In addition, the slide notes of the presentation contents may include a function of automatically providing a caption function that is provided as a caption on the virtual camera output screen so that the presentation and the caption are output together.

In one preferred embodiment of the present invention, using the camera to recognize the instructor tracking and the face rotation angle through at least one of the instructor's face recognition, direction recognition and facial expression recognition, it is used for switching of the virtual camera, and shown in the virtual camera The ability to automatically display the corresponding content, such as blackboard writing, projector images, or students asking questions, on the virtual copyboard.

The virtual camera scene switching and the instructor facial image expression extracted from the camera may include a function of generating and using an instructor image without a chroma key by allowing the character to be used in place of the instructor camera image in real time.

In one preferred embodiment of the present invention, when the copyboard and the projector screen are used at the same time according to the rotation angle of the instructor through the wireless signal transmission using the motion sensor of the mobile phone to track the rotation angle of the instructor in addition to tracking the position of the instructor Auto scenes can be switched.

In one preferred embodiment of the present invention, the application provides a change screen of a virtual camera scene and a change of material to show lecture material, in particular a change of a PPT file, to a PPT file of an external input notebook or a virtual copyboard of a PPT file. Anyone who uses a remote lecture controller to watch the remote scene transmitted through the video conferencing function and control the device of the invention through a remote controller that can be controlled remotely using a smart phone linked with a wireless signal Can include a function to upload a PPT and make it a tele-lecture device for instructor-free remote lectures in the field of remote control over the Internet.

In one preferred embodiment of the present invention, the image of the instructor is input to a Tele Precence system in which the instructor sends a lecturer video and a lecture notebook at a remote location using a remote camera image received through a remote camera. The smart-remote lecture method, which is received and output, is natural, and the instructor does not come to the apparatus of the present invention and transmits his image to the apparatus of the invention by video conferencing and receives the image of the apparatus by video conferencing. The cloud service may be provided to a remote presenter lecture device that uses and transmits ppt presentations to and from the device.

Here, in one preferred example of the present invention, in using the virtual pointer, the function of automatically operating the device of the present invention only by using the pointer used in the actual lecture to move the PPT page to the next or previous page. When the button is selected, the function of enabling the virtual switcher to select the virtual camera showing the scene at the same time as switching the PPT screen and the pen function for drawing the virtual pointer in conjunction with the function of the pointer, the pointer or the mouse Allows you to control the function of intuitively selecting a button according to the position of the virtual switcher of the virtual camera by using the left mouse button, the right mouse button, and the rotation wheel provided to the user, and by rotating the rotation wheel on the mouse or the pointer Scenes that cause the switcher to switch up and down This function which enables a straightforward Switching with just all the various scenes, such as 12, 16 are provided are important to the ability to be a ring only a pointer without a control panel, such as a particular broadcast switcher one button is provided.

As one preferred embodiment of the present invention, when using a general blackboard or a real electronic blackboard, notebook, TV, etc. in the classroom during lectures, the instructor indicates the position, posture, and the angle and body position of the instructor to select and describe them. Tracking image processing Considering the information of the camera, the direction that the face looks through the instructor's face recognition, and the position pointed by the hand through the gesture, the corresponding content, that is, the contents of the projector screen, the writing on the blackboard, and the student's question scene camera It is possible to add various artificial intelligence information so that the automatic switching of virtual camera selection switching and the function to intelligently and automatically set the contents of the corresponding virtual copyboard can be used. Switching is provided precisely and precisely. Image processing for measuring angles of various sensors, video cameras, and instructors, and automatic scenes using actual rotation sensors may be switched.

Here, the method of using the virtual camera scene in the lecture room in real time in the lecture of the present invention, the lecture recording device for recording and storing the scene in real time, the step of transmitting the synthesized screen to the remote site in real time, the remote site Naturally, the apparatus of the present invention through the step of receiving an image of a camera of the present invention in a lecture room is automatically provided with a function of allowing a lecture recording device, a remote lecture device, and a lecture broadcasting device to broadcast it on the Internet. In the method of video recording, video lectures record the same lectures used in actual classrooms. Unlike the previous video recording methods, lectures recorded in the lecture room are recorded as they are, and are shown in the classrooms. TelePresentation devices that have been shown to be impossible It can be configured.

Here, in the above-described telepresenter function, the apparatus of the present invention is used to set up the cameras of these video conferencing programs using existing video conferencing S / W, that is, Skype of Microsoft or Hangout of Google, for the function of making a two-way question and answer with a remote place. Provides a virtual camera input function that selects and outputs a virtual camera to be displayed as a standard capture driver so that it can be used without any special h / w setting in any video conferencing program. By providing the function to make the simultaneous use of the device of the present invention, the lecture device, which makes the lecture efficient in the classroom, is a lecture device that allows students to take the same lesson outside the classroom with water. Lecture hall to become smart classroom lecturer to hear In the meantime, the class will be a smart lecture device that allows students to take the same class even if they are not in class.

Here, in the function of the above-described telepresenter, when the public presenting the device of the present invention, such as a general presentation room, a large lecture room, a conference hotel, etc., the instructor will show the virtual classroom composite screen of the method in the conference room. The lecture scene of the synthesizer is synthesized in a conference scene where only the material is displayed on a relatively large screen and the material is not large. Of course, without the operator, the instructor can be used as a large lecture room or conference lecture device that can be used to change the lecture scenes with a single pointer, so that complicated broadcasting devices for large lecture halls or hotel conferences cannot be relayed on the large screen. Intensive lecture A large conference / conference relay system used as a device to create a site, and even a remote conference device, so that you can see a conference that goes beyond the field conference without having to go to the venue conference. A relay device, and even a virtual camera from a remote location, can be used for remote conferences, which can be seen in other conference venues, so that the cost of attending the conference can be seen remotely and simultaneously in multiple locations, cities and countries. It will be used as the remote VR conference technology of the new BM model that enables charging.

At this time, the business field can be made by a bi-directional multi-regional / classroom conference relaying model where the Q & A discussion will be conducted simultaneously with the remote site and the field, and these presentations will be recorded to resell the content after the conference. Losing ways can be included.

In one preferred embodiment of the present invention, the smart-remote teaching method is provided as a system mounted inside the electronic teaching board or electronic blackboard that makes it easy for the direct lecturer to lecture in the classroom,

If necessary, chroma key function can be provided so that the lecture can be shown in a way that shows the composite scene of the lecture with the lecturer and the presentation material. It can be recorded or watched in a remote lecture.

In addition, the present invention using the above method, the lecturer at the remote site to the lecture through the Tele Presence (Tele Presence), and synthesized the lecturer and the lecture materials and send to the lecturer, where the lecture in real time, the lecture in the remote location Provides a remote lecture device using TelePresentation.

In one preferred embodiment of the present invention, the lecturer and the presentation material by interlocking the camera of the instructor tracking function by providing the function to the device mounted inside the electronic teaching table or copyboard, and to provide a chroma key function if necessary It may be a smart lecture apparatus electronic table or an electronic blackboard to be a classroom lesson of the electronic teaching table or electronic blackboard equipped with the lecture apparatus function of the present invention shown together.

In one preferred embodiment of the present invention, the remote lecture apparatus may be placed in any position in the classroom, for example, the students behind the classroom not only in the classroom method in front of the classroom, In the classroom, the teacher or teacher looks upside down and sees the students together, and the class is held upside down so that the teacher can always use it as a teaching device. The teacher's eye is looking at the camera from the instructor tracking camera so that the students in the classroom can see each other at the same time with the teacher. Classes and methods that maximize lecture concentration Even in remote classrooms may be two concurrent virtual lessons eyes Justify how the teaching method that simultaneously Justify eyes with infinite students.

In one preferred embodiment of the present invention, when the instructor lectures behind the classroom, more screen space is secured in front of the classroom so that an additional screen or projector screen is mounted in the additional space and the plurality of remote classrooms are located on the screen or projector screen. Shows the listeners together and allows the instructor and the listeners to conduct lectures in synchronizing the lecturer's lecture screen and the remote listeners with each other. You can proceed.

In one preferred embodiment of the present invention, when the instructor is teaching in front of the classroom, the instructor camera uses a chroma key environment, which is extracted only by the instructor, at the back of the instructor to use a virtual screen synthesis method or when the instructor is not a chroma key environment. Or AR or VR to make a graphic 3D character look like an instructor using the instructor face recognition method, which can be used as a virtual lecturer or a remote class lecturer. If the lecturer is teaching in front of the lecture device, when the instructor looks at the presentation material screen, the virtual electronic blackboard showing the material screen becomes larger and the screen becomes smaller so that the instructor is smaller. Automatic inversion of the instructor's camera is possible while looking at the blackboard It is easy to provide a feature that allows the instructor's face orientation to take a look at the lecture material together, by providing the ability to make the teacher's eyes look.

In one preferred embodiment of the present invention, the remote lecture apparatus is configured as a lecture studio of an All in One studio apparatus that provides a camera, which is equipped with a camera, which is equipped with a background, lighting, and a monitor. By providing a package, users can provide a lecture-only broadcasting station as a DIY within 1 hour without difficulty, such as construction.

As described above, the smart-remote lecture method and apparatus using the automatic scene change technology according to the present invention is a method of producing a virtual classroom in a more specific and convenient manner, such as a special effect, a method of generating a similar effect by combining multiple layers of 2D, In addition, the above techniques can be used as a way to create a feeling of a similar VR environment without using AR-type chroma key, and above all, the transition of these methods is automatically converted so that the instructor does not need an operator. You can apply automatic scene transition technology that integrates the technology to automatically create the scene where the instructor is highlighted and the VR classroom scene where the virtual monitor or virtual copyboards in the virtual classroom functioning as the virtual copyboard are highlighted. It has an effect.

Smart-remote teaching method and apparatus using automatic scene change technology according to the present invention is a new teaching method and classroom to more efficiently use classroom teaching using blackboards, teaching tables, electronic blackboards and electronic teaching tables used in the existing classrooms As a new lecture device that radically changes the teaching method of the class, the classroom as well as the remote class will be able to watch the lecture like the real classroom and will be able to listen to the lectures on the smart devices around the world in the same way. The limitation of being able to take the class correctly and listening to remote video lectures that everyone knows has the effect of providing a better environment than the lecture of the instructor in the classroom.

1 and Fig. 3 are conventional online lecture recording scenes and synthesis screens;

2 is a virtual classroom authoring tool concept and component-specific studio production screen according to an embodiment of the present invention;

3 to 5 are screens showing arrangement scenes for virtual cameras in a virtual classroom according to an embodiment of the present invention;

6 is a screen switching and screen highlighting method using the direction keys according to an embodiment of the present invention;

7 to 9 are composite screens of various screens using augmented reality technology according to an embodiment of the present invention;

10 are screens illustrating automatic scene change according to artificial intelligence according to an embodiment of the present invention;

11 is a screen illustrating a flip teaching method according to an embodiment of the present invention;

12 is a screen showing a remote presentation according to an embodiment of the present invention;

13 is a scene of a tele presentation system according to an embodiment of the present invention;

14 is a screen illustrating a scene through a multi-screen configuration using a video will according to an embodiment of the present invention;

15 is a conceptual view illustrating a configuration capable of bidirectional simultaneous lessons according to an embodiment of the present invention;

16 is a conceptual diagram of a VR and AR classroom presentation apparatus operating according to an embodiment of the present invention;

17 to 18 illustrate an implementation of AR studio according to an embodiment of the present invention;

19 is a lecture recording function screen according to an embodiment of the present invention;

20 is a block diagram of a self-learning mobile assembled studio device according to an embodiment of the present invention;

21 is a block diagram of an all-in-one mobile server using a studio device according to an embodiment of the present invention;

22 is a conceptual diagram of a virtual output screen capture driver technique according to an embodiment of the present invention;

23 is a block diagram of a classroom lecture scene according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. However, the examples are not intended to limit the scope of the present invention, which will be construed as to help the understanding of the present invention.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of examples. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described on the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

On the other hand, in describing the present invention, when it is determined that the detailed description of the related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. Terminology used herein is a term used to properly express an embodiment of the present invention, which may vary depending on a user, an operator's intention, or a custom in the field to which the present invention belongs. Therefore, the definitions of the terms should be made based on the contents throughout the specification.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and are not to be construed in ideal or excessively formal meanings unless expressly defined herein. Do not.

In addition, in the description with reference to the accompanying drawings, the same components regardless of reference numerals will be given the same reference numerals and redundant description thereof will be omitted. In the following description of the embodiment, when it is determined that the detailed description of the related known technology may unnecessarily obscure the subject matter of the embodiment, the detailed description thereof will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention.

As described above, in the conventional lecture apparatus, all the methods of taking a remote class are different from those of a real classroom class, so the actual class has to go to the classroom and use the professional broadcasting equipment of these broadcasting devices. Even with this equipment, there were limitations in terms of space and time, such as the need for specialists to be mobilized in a specially designed lecture hall in the studio.

Therefore, in the present invention, a method of creating a virtual classroom in a more specific and convenient manner, such as a method of generating a similar effect by combining special effects and multiple layers of 3D / 2D, and a method of creating a feeling of a similar VR environment without using an AR method chroma key. As a result, the above-mentioned problems were sought by utilizing the above techniques.

According to the present invention, the lecturer in the virtual classroom to show the scene to emphasize the lecture material when the lecturer needs to show the necessary lecture material when the instructor knows the intention of the instructor in the device of the invention without such a professional broadcast technician And virtual classroom shots (3D space / or 2D layered composite virtual compositing), which relocate the lecture materials and show the necessary scenes with virtual cameras in this virtual classroom and automatically select various scenes for them. The AI PD function provides a way to automatically create the required screen without the need for equipment operators.

In the present invention, in the present invention, the VR / AR method of using the broadcasting technology in the classroom in real time, and furthermore, the MR recording technology can be used to improve the recorded video of the classroom than the actual classroom, which is more than the actual lecture. If it is improved, it is impossible to make eye contact, which is a problem of existing classroom lectures, because the classroom is used as a lecture device to show the lecture in the classroom by the invention method instead of the existing lecture method, and the classroom is not enough for students to see the lecturer. It can be a new lecture device that solves the problems of such a big case, the problem of not being able to decorate the room beautifully and the limitation of taking a class from a remote place.

This way, if the instructor only teaches, many camera scenes are automatically created and shown on the screen using virtual copyboards that grow and shrink in virtual classrooms without the need for equipment operation and multiple camera operators. Instructors can output larger than they actually are and sometimes the material screens are made in beautiful virtual or augmented reality spaces for lectures, and the same screens can be recorded and sent to remote locations, giving lecture videos beautiful and real-time immersion.

In detail, the device of the invention first synthesizes a lecture room into a virtual or augmented reality to create a beautiful lecture room, and assembles a desk, a wall, and a floor directly in a virtual space. Lectures of floors, walls, desks and ornaments are arranged in a multi-layered manner, or in a multi-layered manner, in the form of classroom studios that are given in various ways. By arranging the instructor and the virtual chariot blackboards output by the virtual cameras, various scenes can be freely configured into beautifully necessary scenes.

For example, a lecture device that uses a blackboard, a projector, and the like to synthesize lecturers and lecture materials and displays the composite screen on the screen of a projector, which shows lecture materials in real time. Students alternate between lecturers and lecture materials. It is a device that lectures by watching a composite screen instead of listening to lectures.The lecture recording, remote lectures, two-way simultaneous classes, presentation meetings, etc. do not have to be taken by the actual instructor, and do not see the class from the classroom. It allows students in the classroom to take lectures more intensively than in the traditional way,

Instead of showing a PPT on a projector or TV screen, the present invention is invented so that the lecturer and the lecture material can be shown to show the lecturer largely when the student needs to see the lecturer, and sometimes the lecture material is sometimes shown together when viewing the lecture material. In the device of the lecturer, the lecturer synthesizes the lecture screen in such a way that the students can show the scenes by using the artificial intelligence function judged by the device of the invention as the students judge the scenes to be shown through the lecturer's gesture, voice, etc. , Lecture materials, and beautiful classroom background-Real time synthesis and scene switching can be done by synthesizing it into virtual classroom.

At this time, camera images and lecture material screens (PPT, notebook, blackboard, electronic blackboard, question student, real imager, etc.) that take lecturers are input to show the composite screen of the lecture. Freely arrange instructor images and lecture materials on the virtual copyboard in a virtual classroom with a layer composition method (VR instructor removes the background and extracts only the instructor image, or VR classroom method using edge recognition, or edge method) The virtual classroom can be virtually photographed through a virtual camera that can be freely placed in a virtual classroom, such as an AR classroom that synthesizes a part of the background and synthesizes a part of the background and synthesizes a real classroom image and a virtual space. Virtual classroom screens, moving these cameras to different positions, Freely place virtual blackboards or instructor images for each virtual camera location so you can place virtual blackboards, virtual desks, and lecturer sizes in the virtual classroom to show lectures that students can focus on on the virtual shooting screen. Automatically output their virtual cameras and show them as projectors or smart devices.

As described above, the apparatus of the present invention enables a virtual classroom to be composed of 2D multi-layer or 3D graphics, and allows a plurality of virtual cameras that can be freely positioned at various positions in the virtual classroom of the graphic, For each virtual camera, a camera image of a real lecturer is placed along with a virtual desk, and a virtual copyboard showing the lecturer's image and the lecture material image to be displayed from the virtual camera is placed on the virtual camera. Freely create scenes that the composite screen displays to the students to show, and virtual switchers that select these scenes, or multiple virtual cameras (placed in a virtual classroom as shown by switching multiple cameras into real switchers in a real classroom). One virtual cameras in the device of the invention Switch to the virtual switcher function to show the lecture scene, but the device of the invention automatically recognizes the instructor's gesture, voice, writing, pointer position, and student's question. The AI feature allows the optimal virtual camera scene to be shown to the students. This switching can be activated by the instructor or operations assistant as needed.

According to the present invention, instead of a plurality of virtual cameras freely arranged in a virtual classroom, one virtual camera is set to a desired position, that is, a position where the instructor and the classroom are shown in a required size (ie, a keyframe position), and for each camera position. When there are 12 virtual camera positions, the free placement of the virtual copyboards on which the virtual copyboards to be displayed are placed and the tracks that set their settings and automatically display in real time in the size or angle set to the corresponding position If the number of camera tracks is 132 (12x11), and two corresponding virtual copyboards are additional 2x132, it is impossible to manually set the tracks of them in real time. This simultaneous function is important, in which case various cases of track generation As to the form and time, and freely set to generate a variety of animation effects beautify the image and make the Amy animation, and is also provided with a function that allows to be able to see such as students grow blackboard or instructor's face to focus the lecture.

In the remote class with the students of the remote site, when the video of the remote student, the interrogator is linked with the video conferencing program used in the device of the present invention, the program input function of the screen is provided so that the image of the student in the remote classroom is displayed on the virtual electronic board. Thus, when a plurality of classrooms are taught, the screen showing the presenter and the questioner together is provided as another classroom.

The various output screens of the plurality of virtual cameras are pre-arranged to show the instructor and the virtual copyboards in various VR cameras as shown in Fig. 4 by setting the necessary main scenes in advance and defining them as key frames of the virtual cameras. The location of the virtual camera and the corresponding virtual camera to focus on one of the instructor scenes and lecture materials that the instructor focuses on, and to show various virtual scenes of the scenes that make the corresponding virtual copyboard scene larger and make the instructor relatively less visible. In order to create the necessary scenes by pre-positioning the copyboard to pre-set the lecture scene by displaying the necessary scenes, the scenes are set to a certain rule and the virtual cameras are tiled and the instructor is upward. Size down, virtual blackboard down In the layout, the virtual camera setting switcher is arranged so that even the beginner can understand the scenes output by the virtual cameras intuitively by using the alignment method that makes the virtual blackboard 1 on the left and 2,3 on the right. You can see the switcher button without looking at it, and you can see the current scene and intuitively move the necessary scene by pressing the set button, or by pressing the left, middle, or right button to move the mouse Includes a feature that allows the instructor to simply move the wheel by moving the wheel or move the scene with simple arrow key buttons, etc.The scene transition basically recognizes the movement of the mouse pointer so that the mouse pointer moves as illustrated in Figure 6. When you go up to the virtual copyboard on the output screen, you create a scene that enlarges the chalkboard. The upper camera button is operated automatically, and if the mouse / pointer does not move for a certain time on the virtual blackboard, the instructor is automatically moved to the upper direction button, which makes the instructor larger. When the scene is over the set time, the instructor shows the entire studio scene again, and when the mouse moves with the pointer over a specific virtual copyboard, it causes the virtual copyboard to move to the largely placed scene. If the mouse pointer moves on the virtual copyboard for a long time, the virtual copyboard moves to a larger scene and then moves to a different scene again after a certain amount of time. Auto chapter in way to the scene The ability to make this transition, it is important that the ability to be a pointer to how the artificial intelligence built-in progress.

When the instructor draws a pen on a notebook with a tablet, or writes on an electronic board or a notebook mouse, the function of the mouse pointer and pen is sensed as in the case of claim 5, and the network is sensed by the device of the invention to change the scene. It includes losing.

When the instructor does not use the pointer and is not recognized by the device of the invention as a laser pointer, the instructor is greatly instructed when the instructor's viewing direction is through the image processing of the instructor's face and gaze direction that enters the instructor camera / or instructor tracking camera. When the instructor looks at the blackboard or projector screen, he uses the method of enlarging the projector screen, and the position sensor (xyx) reads the direction of the instructor's face and the position of the lecturer's classroom to know the instructor's position in the classroom. In this case, the tracking of the instructor can be precise and the multi-position sensor is attached to each microphone to move the virtual camera scene using the microphone voice and the position of the questioner's tracking camera and to display the scene where the instructor looks. Tracking camera displays instructor size and lecture materials by position and angle In the case of claim 5, when writing a blackboard by recognizing the direction of the instructor's position face through time and posture as in the case of claim 5, when viewing a projector screen, when viewing a student, when a student asks a question, Automatic lecture recording and automatic remote recording with the function of enabling camera tracking and scene switching at the same time by recognizing the voice, starting, ending, and viewing the scene. Make classes possible.

Like this, it is as if the students judge the place to see and see the teacher's gestures. Based on movement / gestures / questions, the students automatically select the virtual camera scenes that should be automatically output to the AI function of the device of the invention using the information as if they were looking at the lecturer, the blackboard, and the question student. It is a lecture device that allows the presenter's device to create a virtual camera screen of a scene for students to see in an artificial intelligence method once the speaker announces it. Remote classroom interactive lessons for meetings, It could be a tele-presentation conferencing device for video conferences, or a web conference / webinar for webinars that broadcast these lectures worldwide.

If the position of the virtual camera, the placement of the virtual copyboards for each virtual camera, and the position size angle to the 3D object are set for 12 virtual cameras for each of these, about 800 tracks have to be defined, and these tracks are automatically In order to automatically generate tracks in real time, various animations and animations are automatically generated by changing animation time and track generation conditions for each virtual camera scene so that various camera actions and animations are generated without creating their tracks in 3D max. The virtual copyboard and the animation of 3D objects naturally occur so that the students can focus on the screen smoothly and focus naturally.

Install the device of the present invention in a general classroom so that lecturers and lecture materials can be seen in a remote classroom so that lectures can be automatically transferred to a remote location, and students can be seen in a classroom with an instructor. It is possible to use the remote class method, that is, the two-way class or the multi-class class, that is, the Tele-Class solution. It becomes the device that makes the teaching system.

When the instructor stares at the camera using the device of the present invention, the instructor stares at the camera and the instructor looks at the projector screen. The lecturer's eyes look at all the students in the classroom and the class becomes a simultaneous eye-catching lecture, maximizing the concentration of lectures, and further, the instructor teaches behind the classroom (Flipped Teaching). It's a class where the instructor sees the class scenes that the students see in the class that is taught after the class, and the virtual distance class where the students see the faces and gestures and the presentation materials and the remote class students together. And remote classrooms are the same as on-site classrooms with instructors. Sharing the ups and new remote classroom lessons joint that enables simultaneous multi-classroom lessons with students in multiple classrooms can be made. At this time, the use of a response system that allows a remote student to answer a smartphone via the Internet to perfect immersion is a high-tech VR classroom that maximizes immersion and enables perfect remote immersion class through the technology used together. To be.

It provides a capture driver function that links the existing video conferencing, webinar, or two-way interactive video conferencing functions of the telepresence system to an output screen camera driver that allows the output screen of the unit to be used as the camera input of the video conferencing system. By providing or outputting the input of the existing video conferencing device, the composite screen of the present invention is a video input to recognize the camera of the existing device to become a presentation conference, presentation webinar, or telepresentation system. At the same time, the videoconferencing scene is a feature that allows direct output from broadcasts such as YouTube or Facebook to be performed without additional encoder devices. By providing the technology to simultaneously connect existing webinar programs such as WebEx or ConnectPro, and video conferencing programs such as Skype and Zoom, remote lecture meetings and presentation webinars can be implemented in the virtual classroom method of VR / AR technology. Instructors will have a new virtual technology remote classroom, webinar, and presentation broadcast where the lecturer looks at the virtual classroom screen where the lecturer and lecture materials are synthesized in the teaching methods where the lecturer was small and the lecture material was the main method. This approach allows the use of existing Tele-Presence video conferencing presentation systems together, making the TelePresentation system, or remote presentation device, impossible.

The output of the device of the invention to the existing TelePresence system is inputted to the notebook of the telepresence system. The instructor camera of the existing telepresence system is used as the instructor camera of the device. Present the presentation while viewing the output screen, make the other team's meeting team visible on the additional screen, watch the same presentation on the other side, and watch the attendees on additional channels, perfect for the immersive presentation of the TelePresence device. Make Tele-Presentation system a remote presentation conference.

In the case of a video presentation with a large number of regions, the video wall is used together to effectively view a plurality of participants and the presenter effectively configures the screen when multiple devices are used. Freely configurable and include technology that allows presenters to automatically link video wall stretchers to become a large, immersive system for multi-presentation conferences through the videowall multi attendance presentation VR conference system.

In large conference rooms, where large audiences need to take a lecture, such as a large classroom, the output screen is displayed on the projector's projector screen so that the instructor's facial expressions can be shown effectively to multiple people. It can be viewed as another large classroom or classroom, enabling a large conference presentation device and a relay joint conference system that renews the way in which existing large conference venues are inefficient.

In addition, it is possible to output the left and right screens with the virtual camera placed at the eye width in stereo Scopic 3D, and send them to a remote place so that the same stereoscopic lectures can be viewed on various stereo screens or HMDs even from a remote place.

The virtual classroom is streamed to the 360 screen to view it remotely from the 360's HMD, and the 3D classroom of the virtual classroom is sent to each client in advance so that the client can stream instructor cameras and lecture materials, and downgrade their virtual camera locations together. It allows you to show the scene from the 360 reference point of the virtual camera scene, render it on the client at an additional HMD client position and angle, and have a high quality experience. It's no surprise that it's a 360 HMD device that provides a way to pre-provide and switch their scenes from the client rather than auto-switching.

This lecture device is mounted as a lecture table, cameras, lecture computers, and student tracking devices that track lecturers in the classroom are interlocked, and when the lecturer lectures with the device of claim 1, the lecture is automatically shown on the screen in a virtual lecture manner, and recorded. It becomes the electronic VR classroom which becomes the VR classroom which becomes the classroom with several external classrooms.

It is made of the actual copyboard of the above-mentioned method, and the writing screen of the blackboard is recorded and transmitted along with the instructor's face so that it can be viewed from a remote place so that the camera and the linked instructor tracking and the copyboard writing are synthesized in real time. It's no surprise that VR lectures become automatic blackboards for VR lectures.

It is natural that the lecture recording studio will be recorded when the lecturer is watching the screen outputted by the lecture apparatus in the classroom with the device of the present invention. In this case, when the lecturer uses the presentation material, the slide notes of the presentation slide program such as PPT are seen. It is recognized by the device of the present invention, and it is shown as the title subtitle so that the instructor can present the lecture while watching the subtitle, and when the pointer becomes the next slide, the slide is changed to the virtual camera screen where the lecture material becomes larger. While reading and presenting the slide notes, the slide notes are closed and the caption is finished, and the instructor turns the scenes so that the instructor looks larger when the instructor finishes the slide material. And presente It becomes a lecture recorder with the function of changing the slide and changing the contents of the lecturer, and providing the lecturer with the correct presentation. The slide recorder, the scene change, and the presentation slide notes are automatically subtitled. Anyone can bring the presentation material to the PPT and operate it as a pointer without any manipulation, and when the slide is changed, the screen automatically enlarges the presentation material, the slide notes are displayed in the subtitles, and the scrolling speed of the subtitles is synchronized with the presenter's presentation speed to synchronize them. And when the subtitles are finished, it includes a function that makes the scene move to a scene that enlarges the instructor.

Here, the subtitle scrolling speed recognizes the instructor's speech and automatically adjusts the subtitle speed, scrolling subtitle speed, and presenter presentation speed, and the instructor adds descriptions even after the slide notes. In addition, it is natural to provide a function to make the voice recognition.

Here, when the output screen of the device of the present invention is used as the screen of the framter, the text scroll of the framter is made transparent so that the background of the slide note subtitle is made larger on the screen so that the front of the output screen is visible and the note text is read aloud. It provides the ability to make lectures by automatically adjusting the time, such as the ability to make the lectures disappear automatically after scrolling, or to manually turn on and off the lectures.

In addition, if the pointer uses the gyro sensor in the classroom to enable the mouse function and slide change, the lecture scene will be automatically changed to the lecture and the lecture scene will be displayed in the lecture room. The lecture using the present invention is automatically performed in a manner that shows the synthesis scene of the device of the invention by showing the synthesis scene by using the pointer on the screen and using the pointer on the screen to recognize the motion of the pointer mouse. You can make the composite screen switch.

This is a one-person lecture recording system that uses the functions of FIG. 19 using a chroma key screen and two or one screen as shown in FIG. Lecture recording system that provides studio as a system and provides lecture recording as a VR lecture device at any place without studio construction.The studio provides a monitor-mounted and camera-mounted kit as shown in Figure 24. The automatic function of the apparatus of the present invention includes a technology for providing a DIY studio installation that allows anyone to make advanced presentation lectures and a kit that enables a DIY lecture recording studio.

The technology of the present invention outputs the lecture scene directly to the scene by an automatic or manual simple operation to receive a real-time synthesis screen on the existing classroom projector by using a conventional projector and the presentation computer screen as input in a general lecture scene, such as a movie theater. Naturally, it is natural to make the classroom, hotel, and conference site to be used as a device for relaying, recording, and remotely teaching lectures, and recording AR / VR lectures in places where the classrooms are not directly used as lecture devices. It is a device that can record high-quality recordings of high-quality recordings that solve difficult problems of the existing recording service and problems that recording lectures do not have much more than actual classroom classes by using the (Lecture Capture) device. It will be a device used as a VR lecture recording system.

Such a device receives an instructor presentation computer input, a camera photographing an instructor, a camera photographing an instructor, and a camera photographing a student. When you look at the lecturer's material, the lecturer's material is made of one-touch recording to show the virtual camera setting when the lecturer's material is enlarged. The visible device is made. (See Figure 21)

In particular, instead of using the instructor's gestures and facial expressions to synthesize the camera image of the instructor, they express the face recognition as other characters to compose the actual face and hand movements into characters and use them on the composite screen instead of the instructor. (As a 2D or 3.D instructor who may be himself, the facial expressions as well as the voices are generated similarly so that the actual camera expresses the expressions and gestures of the characters so that the actual instructor uses more than the characters moving on the lecture screen, The character replaces the facial expression with the character instead of the actual instructor's video, and creates a virtual character lecture through face recognition and gesture recognition instead of the actual instructor. In a lecture method that makes expressions appear in real time as characters. The facial expression gesture recognition allows the expression of the character of the virtual lecturer by making the facial expression gesture of the Carrier instructor without the chroma key. It is possible to express it and use it as a virtual instructor so that it can be synthesized like a perfect chroma key without chroma key.In this case, if face recognition is difficult to process in real time, lecturer face recognition is lectured for post-editing and new precision recognition and processing You can make a lecture video with a more accurate look.

Hereinafter, with reference to the accompanying drawings will be described in more detail.

First, FIG. 23 illustrates a lecture apparatus environment in a general classroom environment in which the lecture apparatus is used. Tracking can be provided when the instructor moves the instructor camera to take the instructor first. Chroma key can be easily moved in the chroma key color area behind the instructor. The composition of the lecturer will be used on the lecture screen. Secondly, the PPT file or notebook input is used as the instructor's presentation.The two materials are screens made by resynthesizing a virtual camera placed in a space selected or created as a 2D or 3D graphic virtual classroom in a virtual classroom. It will be shown through the projector or TV in the classroom. A variety of virtual camera scenes will be shown automatically or manually as the switcher scene specified in 4, and the lecture device draws or shows pointers and lectures on the virtual copyboard on the presentation screen. The virtual copyboard that shows the data can be displayed simultaneously by using additional cameras that show additional students so that various inputs can be simultaneously displayed using multiple boards instead of just one. If necessary, a large number of scenes such as student questions will be enlarged to show a virtual camera scene to show their screen.

2 is provided with a virtual classroom authoring tool for selecting and using a basic virtual classroom for lectures according to an embodiment of the present invention and allowing anyone to easily and conveniently make or use these modified virtual classrooms. This is very important and the concept of their authoring tools and studio production screens for each component are shown.

Specifically, as shown in FIG. 2, as an example of the screen configuration of the device of the invention, a plurality of virtual (VR) copyboards to show virtual desks and lecture materials in a virtual classroom, and three virtual copyboards freely positioned in the above scene. The lectures can be arranged by adjusting the angle and setting various scenes that allow the instructor to enlarge or enlarge the screen for each of these layouts, and arrange the virtual cameras to make various output screens in real time. It allows the instructor to make the instructor large or the material to be displayed large by sending the scene to the projector, recording, or remote location.These scenes can be explained using the pointer (virtual graphics) while viewing the screen and the instructor's gesture or hand. The direction of the pointer, the position of the pointer, draw, etc., and start according to the sound of the voice If you show a notebook in these scenes by recognizing the voice, stop, look here, judge the case when you see the student, see the student's gesture and recognize it and ask the student through the voice and gesture to ask the corresponding virtual electronic board. To show students in the classroom as well as remotely.

These virtual classrooms can be produced in 3D Max, etc., but studios provide desks and ornaments that can be assembled in lego style so that lecturers can freely create their own classrooms without using special programs. It offers a variety of pre-made items that make it possible and add them to the web for purchase or purchase.

3 to 5 are screens showing the layout scenes for each virtual camera in the virtual classroom according to an embodiment of the present invention.

In detail, FIG. 3 illustrates an assembly studio method of a basic concept, which illustrates a concept of a classroom authoring tool that prefabricates a ceiling, a wall, and desks of a virtual classroom, and various studio productions by adjusting position colors of assembly components. The existing patented technology gives the user the ability to easily change the color of the shader function and adjust the color of each shader for each component.

In this case, the free placement, size, and rotation of the virtual classroom walls, floors, and decoration 3D objects can be scaled with water so that they can be placed freely. Especially, the color of each background or the surface of the 3D object is HSV. It provides color erection, or RGB color adjustment, which allows you to design according to your feeling and provides a shader function that allows you to make adjustments without any other program.

4 is an example of scenes arranged for each of the 12 virtual cameras in the virtual lecture room of the device of the present invention. The board is placed in such a way that it can be enlarged to make it easy to recognize where the scenes are switched intuitively, so that it is possible to intuitively switch things by button or mouse operation. Intuitive switching comes with the ability to adjust with the mouse button.

In other words, FIG. 4 is a scene in which the scenes of each of the twelve virtual cameras are represented in the form of tile switcher buttons in the device of the present invention. If the instructor is large, the virtual blackboard is relatively small and large so that when the instructor sees the instructor, the lecturer makes the lecture material larger and the virtual copyboard 1 on the left side is 3 on the right side and 2 virtual electronics on the center. The scenes can be arranged in such a way that the board can show them, allowing the instructor to move to another scene from the scene as soon as necessary, allowing intuitive scene selection with mouse or keyboard buttons. This is an important feature that enables without the need to learn special device operation during lectures, where scene selection is manual without operator.

In other words, when the scene of the switch button of the device of the invention in which 12 scenes are arranged in this way is automatically set to the scene of the device of the invention, the scene of the button is displayed. In contrast, the intuitive button provides a way of pre-screening the scene to show the output scene without a free part.

5 illustrates a function of detecting a mouse motion and a position of a function of automatically instructing the instructor to move the mouse / pointer to the corresponding virtual camera so that the corresponding virtual copyboard is made larger by viewing the output screen of the corresponding virtual camera scene.

Specifically, the artificial intelligence function that automatically switches the scenes of each virtual camera to the instructor's pointers, postures, blackboards, etc., firstly detects the movement of the pointer or mouse during the lecturer's lecture to show the scene. Demonstrating the function.

When the instructor writes the blackboard The instructor's posture-the camera's posture tracks the writing of the blackboard by detecting the posture for writing, the position of the hand, the gesture, and the facial orientation of the face. It provides the ability of artificial intelligence to show scenes that are synthesized.

Point to each virtual monitor, or when the instructor manipulates a projector, notebook, or physical presenter in a classroom, it will be a transition of artificial intelligence that will recognize them and automatically show you a virtual camera scene that makes the virtual copyboard's scene look larger. Automatic transitions that detect the mouse pointer movement and make the scene appear larger when the mouse pointer moves over the screen, and revert to the original position when the explanation continues without the pointer moving. When the instructor posture is looking at the student, the scene change setting can automatically move to various scenes when the mouse pointer is not moving.

Even if a variety of artificial intelligence functions are provided, if the instructor wants to show a special scene, a new click function is provided so that even a beginner can easily memorize and operate a desired scene with a mouse without any complicated scene transition. In order to intuitively change other scenes required even when setting, the output scene is displayed without looking at the operation menu, and the scene is arranged as shown in FIG. 3 by pressing the mouse wheel or button or using the arrow direction on the keyboard. It can be operated with a mouse or keyboard button that allows the instructor to operate directly while viewing the bay.

FIG. 6 is a screen illustrating a screen change and screen emphasis method using a direction key according to an embodiment of the present invention.

* Specifically, the instructor can intuitively determine the necessary scene manually even during AI switching, and use the mouse to use as a pointer or the arrow keys on the keyboard and the key button arrangement for the left hand used in the game to view the current scene and other scenes. To position the virtual camera in a tiled arrangement, highlighting the virtual copyboard to the left and right, and placing the switcher buttons in a manner such that the instructor size and the virtual blackboard size are adjusted up and down. You can change scenes with only mouse button or arrow button.

This method of manual switching is a function that utilizes the functions of the previous patent together to enable manual switching with a single mouse click even when automatic. In the present invention, the contents of the virtual copyboard of the scene, that is, the external input, the video list, and the pictures are defined in order so that they can be switched to the next or previous contents. It adds the ability to rotate or use the special F1, F2 keys to automatically modify the contents of the copyboard during lectures, and allows the user to look at the scene switching of this function by voice or gestures, as a student looking at the virtual electronics. The chalkboard content will be provided in an AI way that will automatically display it.

In FIG. 7, when a virtual camera scene in which the output of the virtual blackboard is highlighted is selected, the external input or image content to be displayed on the virtual blackboard is automatically changed by the instructor's words or the direction / pointer movement of the instructor in artificial intelligence. It is designed to be switched, and if necessary, the instructor can change the scene change by pressing the mouse wheel or switch using the function key so that the scene can be changed without looking at the menu screen. This makes it possible to switch by means of an AI or pointer button i.e. a mouse.

The number of virtual cameras is three or four, eight, twelve, etc., if necessary, and the necessary buttons are created and arranged in the left and right up and down directions of FIG. 4 so that their arrangement is tiled in the virtual classroom. The position of the tiled virtual camera and the switcher mixer's buttons that make it possible to predict the placement scene by recognizing it as if it is placed with the switcher are the functions that allow the instructor to manually operate the menu when the operation is required without knowing the equipment. It is important.

Also, by switching each of these virtual cameras, the virtual copyboards can be edited to be automatically placed in the corresponding scene so that the virtual cameras can be placed at the most camera position and the corresponding virtual camera. If you set the key frame, the various speeds and tracks are generated so that each virtual camera scene, that is, each keyframe, is automatically generated as a virtual camera movement track of the virtual camera cutaway by animation instead of switching. The blackboards also automatically generate animation time and tracks so that students can focus on the scenes that they need to see slowly when switching scenes. very important.

In addition, the device of the invention is a 3D object that can display real-time interactive 3D animation objects that provide the ability to perform real-time 3D animation, not just 2D images and PPTs, which are seen as external presentation materials, but to move to a virtual experiment or space. It is a function that can load the data of the data and place them in each scene and show them the actual experiment. It overcomes physical constraints, namely size and weight, and shows 3D objects or animations in real time. Providing is a way of becoming an important virtual / or augmented reality presentation. To do this, 3D editors such as 3D Max and Maya read data of animated objects and place them in a virtual classroom (see FIG. 8).

8 shows an AR / VR presentation scene showing a 3D object and an animation scene / virtual interactive experiment, and a scene that is effectively shown by showing differently in 12 scenes for each virtual camera.

This method of displaying 3D virtual objects will enable new lecture apparatuses that enable the 3D virtual classroom lecture apparatus to make impossible part of lecture recording or lecture apparatus.

As shown in FIG. 4, in case of using three instructor screens and three virtual copyboard monitors for 12 scenes, 132 camera tracks are moved from 12 virtual camera tracks to 11 different virtual camera positions, respectively. If 3 copyboards are used and 1 virtual 3D object is used, the total tracking passes will be 132x 6 total, and 792 paths will be generated automatically. The onset device provides the ability to automatically animate each scene with the location of the virtual carema, the location of the virtual copyboards, and the virtual 3D object.

5 is a scene of a classroom in which a device of such a method is arranged in a classroom to display a screen output through a projector in a manner in which each virtual camera scene is displayed.

FIG. 9 shows a virtual classroom in which the instructor and the lecture material are synthesized in a virtual or augmented reality technology in real time in a device of the present invention, in which the instructor shows the material when the instructor looks large when the teacher looks large. By arranging cameras in various ways, they judge artificial intelligence functions such as instructor gestures, voices, pointers, and the like. As the students judge themselves and their materials, the device of the invention judges the lectures in a cool virtual classroom on the screen. This composite screen is recorded and transmitted in such a way as to show the scene to be shown so that the class can be made like the classroom in the classroom as well as in the remote classroom.

10 is a posture recognition function as a function of an artificial intelligence that allows the scene to be automatically changed to the scene and the contents of the virtual electronic board to show the projector or the blackboard with the gesture of the instructor, the attitude of the instructor, the angle of view, and the voice recognition. It shows the scene that is automatically converted to scene by enabling the location recognition to use an image or an external sensor.

In other words, two virtual copyboards are used to cut out the outer part of the part that the instructor sees by using an ellipse rather than the chroma key method, and gradually process the transparent part of the part of the cut out part. An example of a laptop and blackboard camera in an AR classroom scene that is composited with electronic blackboard screens. An instructor's input is a combination of an area and a virtual electronic blackboard-graphic frame with a real instructor's camera input. The following shows an example of switching scenes that are variously arranged and displayed in virtual cameras that are arranged freely.

In other words, when the instructor looks at the projector screen and speaks, a virtual camera scene is output that enlarges the laptop screen shown in the projector, and relatively small instructor's video while composing the background scene with AR concept on the background. This allows the switching of the laptop's screen to be displayed on the virtual copyboard automatically. When the instructor goes back to the blackboard and writes on the blackboard, the tracking camera on the blackboard screen captures that portion. Move the camera and show the screen on the virtual copyboard, using a technique that automatically displays the virtual camera scene.

In the classroom, the instructor places the instructor's camera on the table or in the direction of the student in the classroom, or places the instructor camera at the back of the classroom and sees the output screen in this area. If you take a lecture while staring at the output screen, you will be able to make eye contact with the entire class at the same time as the instructor's eyes always look at the students.

In this way, when the lecturer lectures while looking at the projector screen in front of the classroom (see Fig. 11), the instructor already sees the face several times larger than it actually is in the beautiful virtual classroom on the lecture screen. If you do the lecture, the lecturer will be watching the lectures that the students see, and the lecturer's camera will be placed on top of the projector. The students will be immersed in the tutoring lessons at the same time, and the scenes they should see automatically will be shown automatically. In normal class, students will not see the teacher or the projector screen. Projector virtual classroom lectures , That is a lesson to be concurrent eyes Justify the virtual classroom. At this time, the student camera placed in front of the classroom is displayed on the side of the screen or on one of the virtual electronic boards. When a student asks a question, the student automatically switches to the scene and tracks the student's voice and posture. It is a class that allows the teacher to see the student's face in the class and the class proceeds to the class after the lecture with the students at the back of the class. do.

In this class, the instructor does not occupy the space between the projector screen and the instructor in front of the students, and creates a large screen classroom in front of the classroom. Classroom lessons become virtual eye-catching virtual classroom lessons with maximum immersion.

In this way, the class that the lecturer makes the lecturer behind the scenes becomes the actual lecture video class, and when the class is recorded, the video can be played after the class or in another classroom class. Especially, the lecture screen in the remote classroom is the same as the field class with the instructor. In the meantime, the remote classroom will be a remote classroom that completely solves the problem of technology that is worse than the field class.

As shown in FIG. 11, the screen shows a flipped teaching screen in which a lecturer looks at a lecture scene and a student and a remote classroom behind a lecture room in which a lecturer looks at a virtual classroom screen of the invention.

To this end, an output screen camera for linking a two-way interactive video conferencing function to the device of the present invention to recognize an output screen camera that sets the composite screen of the device of the present invention as a video conference input so that the output screen acts as an instructor camera during the video conference. Driver mode is provided so that any videoconferencing or direct output from broadcasts like YouTube or Facebook can be performed without additional encoder devices, allowing you to use existing webinar programs like WebEx or ConnectPro, video conferencing programs like Skype and Zoom. By providing the technology that enables simultaneous interworking, the remote lecture meeting and presentation webinar is made in the virtual classroom method of VR / AR technology. Take a look at the synthesized virtual classroom screen This allows you remote classroom, Webinar, presentation broadcast of new virtual technologies. This approach allows the use of existing Tele-Presence video conferencing presentation systems together, enabling the TelePresentation system, or remote presentation, which has not been possible.

FIG. 12 shows a device that uses the apparatus of the present invention in conjunction with an existing internet conference, webinar, webconference and telepresence conference system so that an eye-taking lecture at a remote location feels like an instructor comes to the lecture. The scenes that make the presentation Tele-Presentation appear.

In addition, FIG. 13 shows a scene of a tele presentation system in which a presenter and a presentation data are sent from a conventional TelePresence system and the presenter is synthesized by a device of the invention and sent to a single screen. .

In this way, the TelePreseence device displays two inputs, the instructor camera and the instructor notebook, respectively, on two remote monitors. Send it to a channel, or laptop input, and the instructor camera lets you show attendees and send them remotely so that both instructors and attendees are shown remotely.

When there are multiple connections to the output screen, the video conference output is made by using the video wall technology to display a plurality of remote screens in various ways. It enables a Tele-Presentation Multi-conference system.

14 shows an example of using a scene through a multi-screen configuration using a video wall.

In this way, when the classrooms are connected to the remote classrooms using the various methods, the remote relay class that uses the device of the present invention to the existing classroom can enable the effective class to be more immersed in the remote classroom than the field. Since the recording is performed by the device method of the AR method, even if the lecture is not well understood, the recording can be viewed again so that the lecture can be recorded as well as the actual classroom.

The remote virtual lecture system according to the present invention uses a projector screen in a large existing classroom, and if the instructor is small and the projector screen is large, the instructor does not brighten the room light by the brightness of the screen, and thus the instructor is dark and small, and only the presentation materials on the screen are raised. Large classroom lecture equipment used in the lecture room enables lecturers and lecture materials to be effectively used as a device that can be seen in the large classroom.At the same time, large lecture lectures in the remote classroom where the lectures are sent to the remote classroom and the remote cameras are shown to each other in the classroom. As a device, the problems of the existing lecture room and the large lecture room can be linked to the classroom.

FIG. 15 shows a concept of using a lecturer, which is a problem of a large classroom lecture, to make a lecturer and lecture materials appear on a large screen in a dark and small lecture room, and to simultaneously record two-way classes with other large classrooms through recording and real-time transmission.

The video presentation system, which sees each other for two-way question and answer during the remote class, enables the perfect remote class by interworking with the real-time participant question and answer system that asks questions on the laptop screen such as PowerPoint, receives the answers in real time, and displays them in a table. do.

This lecture device is used to make a stereoScopic image using two virtual cameras in a virtual classroom and to make a stereo lecture device that sees a stereoscopic 3D screen in each classroom, and in particular, to send a virtual classroom scene to a 360 screen through a 360-screen HMD. Students will be able to control the view from side to side or back and forth on their favorite VR screen. At this time, the 3D data of the virtual classroom of the device of the present invention is downloaded in order to improve the quality of the actual 360 screen, and the lecture data and the instructor camera video are transmitted to the student HMD device for real-time rendering. It is possible to maximize the immersion by implementing the HMD linkage method with.

This type of lecture is a simultaneous eye-to-learn device that enables simultaneous eye contact with all students in the classroom, the Internet, and remote classrooms, and the output screen becomes a beautiful teacher lecture in a virtual classroom. Even if you do not judge the lecture materials with the lecturer, you can see the necessary scenes by looking at the projector screen, which becomes a state-of-the-art classroom that makes the existing classroom classes more immersive, and the lectures on the VR / AR composite screen are recorded in real time. Advanced VR / AR that allows you to watch the class remotely by automatically transmitting it to the real-time view anywhere in the world, the two-way Q & A through the camera to a large number of classrooms, and the perfect two-way class by applying the real-time response system To become a classroom teaching system All the sciences, international conference space is the association with the remote classroom, conference and presented the VR / AR classroom so the Q & A, and real-time response to the problem.

In the meantime, the class had to go to the classroom and give lectures, and overcome the limitations of the lecture / classroom that had been required to take classes, and no longer have to go to the classroom, classroom, conference hall, Make it a teaching device that allows you to see and listen to lectures that are more spectacular than they are on site or even better than on site.

16 is a conceptual diagram illustrating the operation of the R and AR classroom presentation apparatus. Specifically, the instructor combines the chroma key processing of the instructor camera, the external input camera, and the student tracking cameras that capture the student's position according to the instructor's position and the instructor's position and the camera's position. When the teacher looks at the students and explains them, the teacher enlarges them. When the teacher looks at the material screen, the material screen is synthesized together.

In order to create AR mode lecture scenes, frames are added to the instructor's camera output by cropping the edges of the cropped edges and providing them with transparency and color change functions to combine the screen with the background.

Fig. 17 shows the composition of the VR classroom, AR classroom scene and AR classroom by combining the background with the method of composing transparency and frame transparent parts on the elliptical crop (cut) and the edge (around the crop) of the instructor camera. A scene showing an AR studio is shown.

FIG. 18 illustrates an AR studio technique in which a peripheral frame (elliptical shape) of an instructor camera image and an elliptical frame at a cut edge of the instructor are smoothly combined with a background virtual classroom through transparency processing. After cutting the ellipse around the instructor's camera, add an edge frame at the edge and adjust the transparency and color of the part to make it look natural with the background.

19 shows a function of recording a lecture while viewing a slide note such as a PPT with a prompt screen and subtitle scrolling. Specifically, when the instructor turns over the PPT slide, the slide notes automatically appear as subtitles and prompter screens, and the lectures are read with the subtitles as the subtitles (these screens are displayed on the output screen. Function included).

20 shows an example of a DIY e-learning mobile prefabricated studio, and FIG. 21 shows a lecture relay / recording / VR / AR switcher environment using the mobile prefabricated studio-a laptop mobile system and an all in one mobile server configuration diagram. FIG. 22 shows a virtual output screen capture driver that uses a plurality of applications such as video conferencing, youtube, and facebook at the same time so that the lecture composition screen of the invention can be used without additional devices for broadcasting a video conferencing program or YouTube. The technique is shown as an example.

Although the embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features thereof. You will understand that there is. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the following claims, fall within the scope of the spirit of the present invention. will be.

Claims

(a) Virtual desks, floors, blackboards, ceilings, and classrooms are created by constructing virtual classrooms of 2D and 3D graphics that enable the synthesis of lecturers and lecture materials in virtual classrooms by means of graphics and VR for AR. A virtual classroom making and selecting step of outputting at least one of the ornaments;

(b) a plurality of virtual cameras for selectively outputting a plurality of virtual classroom scenes by arranging a plurality of virtual cameras in which at least one of position, rotation, and field of view (FOV) is freely defined in the virtual classroom; Providing a deployment function;

(c) Cropping, edge frame / transparency processing method of instructor camera to extract only the instructor in chroma key / depth method from the image of the real lecturer in the virtual classroom scene shown in the virtual camera or AR synthesis Create instructor characters in real time through composition, or face recognition of the instructor, and place the instructor images so that the instructor images are displayed in the required position on the output screen of the corresponding virtual camera. Providing a function capable of displaying the output screen photographed through the corresponding virtual camera so that the instructor is shown in the required position in the required size in a scene to be shown to the viewer;

(d) A plurality of presentation images / images in which at least one of an instructor's image, a desk, and a blackboard are to be independently displayed in the virtual classroom scene, so that the plurality of virtual copyboards can be freely positioned in the virtual classroom. Providing a function of freely arranging the virtual copyboard so that the instructor image of the virtual camera and the virtual chalkboards are displayed on the virtual camera as necessary scenes;

(e) setting the movement of at least one or more of a time and a path at which a camera track is moved to each virtual camera position when an output scene is changed and displayed as a necessary scene while the set scenes of the plurality of virtual cameras are lectured; In the process of providing the function, the instructor image arrangement and the position size of the virtual blackboards set in each virtual camera are synchronized with the camera movement so that each track is automatically generated so that each instructor with these virtual camera change track movements. Providing an automatic or manual track generation function such that the image and the virtual copyboard are transitioned together by natural animation; And

(f) Switcher button setting that defines a plurality of cameras as virtual switcher buttons like buttons of general broadcast switcher and arranges them as tile button GUI, and defines ICON of these buttons as preview button to show the scene. Arranging such that the instructor can intuitively switch scenes using only a mouse or pointer button without looking at the GUI switching screen with only a mouse button through the process;

(g) setting GUI button switching of virtual copyboard contents to set content or external input cameras to be displayed on each virtual copyboard together with the button GUI switcher setting and to easily switch and display the contents of the corresponding copyboard; And

(h) An artificial intelligence PD function that remotely or automatically switches the function of the broadcasting PD that presses the selection switching buttons of the GUIs arranged in the plurality of virtual cameras according to the lecture situation, so that the instructor gestures, pointers, sounds, language, Various scene changes and virtual copyboards automatically by the automatic PD function that recognizes the scene by recognizing at least one or more of the voice, the direction of the gazing of the instructor's motion, and the students' voice and direction. A smart-remote lecture method using an automatic scene change technology comprising the step of providing a function to automatically perform the content conversion to output the lecture scene in real time.
The method of claim 1,

In the step (d), when the content output on the virtual copyboard is at least one of the plurality of cameras, notebooks, video lists, and picture lists, when the mouse or the pointer points to the virtual copyboard on the output screen, the corresponding copyboard The list is defined in advance so that external cameras, images, and images placed on the contents list can be switched in order, and the following contents are selected through the mouse or the pointer to control the playlist without viewing the switcher menu. Mouse / pointer control of the corresponding virtual copyboard content scene by automatic virtual copyboard selection, further comprising the step of providing a function that can be performed for each blackboard and changing the contents or slides such as PPT. Using one-touch switching technology Smart-remote teaching method that includes the function to make the transition to the scene while the pictures of the pointer drawn on the transition is automatically erased.
The method of claim 1,

In the step (b), in addition to the instructor and the virtual copyboard showing the virtual camera in the virtual classroom, 3D objects that can be used to show the virtual object animation using real-time 3D rendering technology are freely sized for each virtual camera scene. And arranging positions so as to be able to adjust the positions, and allowing animation and interactive control to control real-time 3D virtual experiments and hands-on learning with a joystick or a keyboard. Smart-remote teaching method further comprising: .
The method of claim 1,

In step (a), bending the virtual copyboard by making the virtual copyboard in 3D to adjust the size of the virtual copyboard and cropping of unnecessary portions, and to adjust the distortion of the virtual screen to a curved curve screen. Provides the ability to adjust the degree of display of immersive virtual copyboards so that they can be placed in the scene of each virtual camera, and the frames of these virtual copyboards can be sized and curved together, frame color and size Providing a function to control the smart-remote teaching method further comprising.
The method of claim 1,

The virtual copyboard includes an electronic pen function that can be marked by a mouse or a pointer, wherein at least one of the thickness, color, and transparency of the electronic pen is controlled, and the portion drawn by the pen function changes or changes the content of the blackboard. Smart-remote teaching method with the ability to automatically erase point scenes during transition.
The method according to claim 1 or 5,

When presenting using PPT files in the classroom and in the presentation room using the mouse or pointer equipped with the gyro function, the contents of the slide notes to write the presentation contents on each slide when the slide transition of the PPT is performed The size of the font can be displayed on the monitor so that the font size can be displayed in a large font so that the slide note screen can be used as a broadcast programmer, and the instructor looks at the lecture to present the lecture. It provides an auto-prompter function that allows you to read what you want to do and to give lectures or presentations.At the same time, if you need a subtitle function, slide notes are automatically provided as subtitles on the virtual camera output screen. PowerPoint slide note ruler with features How remote river - Fram printers and smart with automatic subtitles.
The method of claim 1,

If the instructor's viewing direction is a projector screen using a camera that recognizes the instructor's position, posture tracking, and facial rotation angle through at least one of the instructor's position and face recognition, direction recognition, gesture recognition, and facial expression recognition, the projector screen is a virtual copyboard. If the student is a student, the student moves the selected PTZ camera to select the corresponding student so that it is displayed on the virtual copyboard, and is used for switching the virtual camera to show the instructor and the virtual copyboard together. Includes the ability to show and automatically track the student, projector, or blackboard of an intelligent automatic virtual electronic blackboard to be shown,

By instructing the virtual camera scene switching and facial expression images extracted from the camera in real time to generate 3D images of a character or a real instructor to be used instead of the instructor camera image, the instructor image is generated without a chroma key in the lecture scene. Smart-remote teaching method that includes the ability to use virtual, real-life instructor characters in a virtual classroom in VR.
The method of claim 1,

When the copyboard and the projector screen are used at the same time, the lecturer's position, posture, and motion sensor signals and the position, direction, and motion signals of the mobile phone, which are tracked by the instructor's rotation angle, in addition to the instructor's position tracking, are transmitted through wireless signals. Smart scene, characterized in that the automatic scene according to the rotation angle, the moving direction and the position of the virtual camera and the virtual copyboard contents are switched.
Using the method of any one of claims 1 to 8, the lecturer does not take the class in front of the classroom, but the class while watching the lecture materials and the lecture material synthesis screen shown on the virtual camera-type composite output screen with the students behind the classroom. The instructor's attention is that the instructor looks at the output screen that the instructor sees along with the lecturer's eyes while placing the instructor tracking camera on the output screen. A smart-remote teaching method whereby the composite screen always shows a composite lecture scene with the instructor video facing the front, so that the students in the classroom watching it look as if the lecturer is always staring at the students.
The method of claim 9,

If the instructor lectures behind the scenes, more screen space is secured in front of the classroom, with additional screens or projector screens in the additional space, where students from multiple remote classrooms are shown together, and the instructor and students synthesize the lecturer's lecture Smart-remote teaching method that allows students to see the screens and remote students at the same time.
The method of claim 1,

Change of course materials, change of material to be displayed on virtual copyboard of PPT file of external input notebook, and switching button screen of virtual camera scene as an application of smart device with internet connection. Provides the ability to freely change the presentation materials or change the virtual camera in the classroom so that students who ask questions from the remote site can change the virtual camera or ppt slides of the scenes as needed. Smart-remote instruction method with features that can be controlled in.
The method of claim 11,

The video of the instructor is received using a remote camera image received through a camera at a remote location, the instructor video from the remote site (Tele Precence) system, which sends the lecturer video and lecture notebook as an input and outputs a smart- The method of remote lecture is natural, and the instructor does not come to the device of the present invention, but transmits his video to the device of the invention by video conferencing, receives the video of the device by video conferencing, and remotely views his ppt presentation material. A smart-remote teaching method that enables cloud services with remote presenter teaching devices to send and use.
The method of claim 1,

The smart-remote lecture method is provided as a system mounted inside the electronic teaching table or the electronic blackboard that makes it easy for the direct lecturer to lecture in the classroom,

By providing chroma key function if necessary, lectures are presented in a way that shows the composite scene of lectures with instructor and presentation materials. Smart-remote teaching method that allows recording or viewing by remote lecture.
Smart-remote teaching device comprising a system using the teaching method according to claim 1.