KR20240006171A - System and Method of Online Learning - Google Patents

Abstract

The online learning system according to the present invention includes a learning server that transmits display position information, which is information indicating a position to point or mark a specific part of an actual textbook; A learner terminal that receives the display location information from the learning server and displays a real-time image of an actual textbook held by the learner by overlaying a pointer or marker at a location according to the received display location information and displaying it together. can do.

Description

Online learning system and method {System and Method of Online Learning}

The present invention relates to an online learning system and an online learning method.

Due to the global respiratory epidemic, the demand for non-face-to-face online learning has increased. The remote video conferencing system, which was mainly used by global companies in the past, has also been introduced and used for non-face-to-face online learning. This phenomenon has expanded not only in public education but also in the private education market such as academies and private tutoring.

Although the global respiratory epidemic has disappeared, the demand for non-face-to-face online learning still exists. The reason is that it is efficient and economical in terms of time and cost compared to traditional face-to-face learning.

However, opinions are still divided about the effectiveness of education. The biggest problem pointed out in online non-face-to-face education is that it is difficult to elicit active participation from students. Most online non-face-to-face education conducted using current technology is taught by instructors creating lecture materials and delivering the content to students online. Instructors and students can see each other's faces, so structurally, face-to-face education and online non-face-to-face education seem to be completely the same, but there are two major differences.

First, the image quality is significantly lower than that of face-to-face education. The larger the number of students participating in an online learning classroom, the lower the image quality.

Figure 1 shows the configuration of a general distance education system.

For smooth remote learning, there should be no buffering or interruption due to traffic overload. A simple calculation of how much traffic is required is as follows. The instructor must upload a real-time video of himself/herself to the learning server, and n students must stream the video in real time. Standard high-definition 720p, commonly found in a typical desktop environment, requires a speed of 3~4Mbps, and in order to educate in standard high-definition, the learning server requires a communication bandwidth of 4xn Mbps (if standard high-definition is 4Mbps), for example. If a class has 40 students, including the teacher, approximately 4x41 = 164 Mbps of bandwidth is needed.

Furthermore, as the number of students studying at the same time through the learning server (for example, the number of classes) increases, the required bandwidth increases exponentially. Ultimately, in order to perform the service without interruption within the reserved bandwidth, the transmitted video must be This will be responded to by lowering the quality of the video, ultimately leading to a decrease in the quality of the video the learner is watching and causing a time delay between the instructor and the learner. This increases the learner's sense of distance from the instructor and hinders their concentration in learning, and this also applies to the instructor who receives the videos from the learners.

Second, in current non-face-to-face online education, videos generated by the instructor are unilaterally transmitted to students. Because it is a method of watching videos created by the instructor, as if watching a drama or movie on TV, the learner's concentration decreases.

Although non-face-to-face online education is economically and time-efficient, it also has significant negative effects for the same reasons as above.

The purpose of the present invention is to provide an online learning system and method that can reduce required bandwidth.

Additionally, the purpose of the present invention is to provide an online learning system and method that can further improve learners' concentration.

An online learning method according to an aspect of the present invention includes a first step in which a learner's terminal receives display position information, which is information indicating a position to point or mark a specific part of a physical textbook, from a learning server; A second step in which the learner terminal displays an image captured in real time of an actual textbook held by the learner, and displays it together by overlaying a pointer or marker at a position according to the display location information received in the first step. It is characterized by:

In the above-mentioned online learning method, the second step is the difference between the coordinates when the real textbook is unfolded in a perfect plane and the coordinates in the real-time image of the real textbook held by the learner. Step 2-1 of generating a learner mapping function representing the corresponding relationship; It may include a 2-2 step of obtaining a corrected position by correcting the position according to the display position information using the learner mapping function, and displaying the learner terminal by overlaying a pointer or marker on the corrected position.

In the above-mentioned online learning method, when two or more learners study with the same instructor, the learner mapping function generated in step 2-1 is generated for each learner terminal, and the learner mapping function is generated for each learner terminal from the same display location information. The correction positions generated may be different for each learner terminal.

In the above-described online learning method, the display location information is generated by the instructor's instructor terminal and transmitted to the learning server, and the instructor's instructor terminal displays real-time captured images of the actual textbook held by the instructor. In the third step, the instructor generates the display location information from an input of pointing or marking a specific part of the physical textbook and transmits it to the learning server.

In the above-mentioned online learning method, a page code in which information representing the page number is digitally coded is printed on the physical textbook, and after the page number according to the page code is recognized in a real-time video of the physical textbook held by the instructor, , is transmitted to the learner terminal through the learning server, and the learner terminal can display the received page number together when displaying a video captured in real time of the actual textbook held by the learner.

In the above-described online learning method, the printed page number is recognized in a real-time video of the actual textbook held by the instructor and then transmitted to the learner terminal through the learning server, and the learner terminal is When displaying a video captured in real time of an actual textbook, the received page number can be displayed together.

In the above-described online learning method, the third step is to create an instructor mapping function that represents the correspondence between the coordinates when the actual textbook is unfolded in a perfect plane and the coordinates in the real-time captured image of the actual textbook held by the instructor. Step 3-1 of generating; Step 3-2 of obtaining a corrected position by correcting the position at which the instructor points or marks a specific part of the physical textbook using the instructor mapping function, and generating the display position information based on the corrected position. You can.

In the above-mentioned online learning method, symbols of a certain shape are printed at a certain location on each page of the physical textbook, and when generating the learner mapping function in the 2-1 step, or in the 3-1 step When creating an instructor mapping function, the above symbols can be used.

In the online learning method described above, the printed symbols may be identified by an infrared camera when irradiated with an infrared lamp.

In the online learning method described above, when creating the learner mapping function in step 2-1 or when creating the instructor mapping function in step 3-1, the result of recognizing the outline of the page is used, The result of recognizing the virtual envelope surrounding the letter row can be used.

An online learning system according to an aspect of the present invention includes a learning server that transmits display position information, which is information indicating a position to point or mark a specific part of an actual textbook; A learner terminal that receives the display location information from the learning server and displays a real-time image of an actual textbook held by the learner by overlaying a pointer or marker at a location according to the received display location information and displaying it together. can do.

In the above-mentioned online learning system, the learner terminal generates a learner mapping function that represents the correspondence between the coordinates when the real textbook is unfolded in a complete plane and the coordinates in the real-time captured image of the real textbook held by the learner. In addition, the learner mapping function can be used to obtain a corrected position by correcting the position according to the display position information, and the learner terminal can overlay and display a pointer or marker according to the corrected position.

In the above-mentioned online learning system, while displaying a real-time captured image of an actual textbook held by an instructor, the display location information is generated from an input where the instructor points or marks a specific part of the actual textbook and is sent to the learning server. It may further include a transmitting instructor terminal.

According to the present invention, only small-capacity overlay information (display position information) and audio information can be transmitted, which has the effect of reducing the required bandwidth during online learning. Since there is no need to transmit videos to each other, a small communication bandwidth is used and there are no problems such as transmission quality degradation or time delay. In the existing conventional method, the image quality had to be intentionally lowered due to the communication traffic load, and if the number of people participating online was large, the image quality was inevitably degraded seriously. However, according to the present invention, instead of watching the transmitted video, you can watch the video input from your terminal. Since you learn by watching videos, the high resolution of the videos has the advantage.

In addition, according to the present invention, the instructor's pointer or marker moves and learning audio is played in the textbook (and the learning environment) that the learner feels as his/her own, so the learner has the advantage of feeling as if the instructor is directly teaching him/her ('professor'). It has a realistic feel) and has the effect of increasing the learner's concentration and learning efficiency.

In addition, according to the present invention, a pointer or marker is displayed in accordance with the distorted textbook image for each learner's terminal, thereby improving the realism of teaching for all learners even when multiple learners are learning at the same time.

Also, as an instructor, there is an advantage in being able to teach while watching your own textbook rather than teaching (lecturing) by watching a video of a remote learning material.

In addition, according to the present invention, since the actual textbook can be used as is, there is an advantage in that there is no need to create separate materials or electronic files.

Figure 1 shows the configuration of a general distance education system.
Figure 2 is a block diagram showing an online learning system according to an embodiment of the present invention.
Figure 3 is a block diagram showing the internal configuration of the terminal 20 according to an embodiment of the present invention.
Figure 4 shows an example of a screen displayed on a learner terminal in an online learning system according to an embodiment of the present invention.
Figure 5 shows an example in which additional printing portions (symbols) are additionally printed on the physical textbook according to an embodiment of the present invention.
Figure 6(a) is an example of an image taken of a textbook with symbols printed on it and spread out in a perfect plane, and Figure 6(b) is an example of a virtual grid network formed on the image of Figure 6(a).
Figure 7(a) is an example of an image taken of a curvature when the physical textbook with symbols printed on it is unfolded, and Figure 7(b) is an example of a virtual grid network formed on the image of Figure 7(a).
Figure 8(a) is an example of a virtual grid and markers configured on the image of the actual textbook, and Figure 8(b) is the image of Figure 8(a) converted to a standard image (a flat image with distortion corrected). This is an example of a virtual grid and markers.
Figure 9(a) is an example of configuring a virtual grid and markers on a flat image, and Figure 9(b) is an example of configuring a virtual grid and markers on a distorted image of an actual textbook.

Usually, instructors (e.g. teachers, professors, lecturers) and learners (e.g. students, students) have physical textbooks, but in fact, learners' physical textbooks are not used during online non-face-to-face learning. Usually, instructors use separately created materials (for example, PowerPoint materials) or electronic textbooks, and while the contents of these materials or electronic textbooks are displayed during learning, the display images and the instructor's actions (mouse movements) are displayed as videos. Then, the video is transmitted and displayed on the learner's terminal. However, this method has the problem of traffic overload due to video transmission, but it was analyzed by the inventor of the present invention that it is a factor that reduces learning participation and concentration from the learner's perspective.

Since the main culprit that causes communication traffic overload is real-time video sharing, the online learning system and method according to the present invention starts from not sharing real-time video, and furthermore, the feeling that the learner receives, specifically, the instructor teaching directly or side by side to the learner. The goal is to increase learners' concentration and learning efficiency by improving the feeling of what is being done (hereinafter referred to as 'teaching realism').

FIG. 2 is a block diagram showing an online learning system according to an embodiment of the present invention, and FIG. 3 is a block diagram showing the internal configuration of the terminal 20 according to an embodiment of the present invention.

The online learning system according to an embodiment of the present invention is comprised of a learning server (10) and two or more terminals (20). One of the two or more terminals may be an instructor terminal used by the instructor, and the other may be a learner terminal used by the learner, but this configuration is not necessarily limited. For example, the instructor is not specified and a number of learners form a group. Therefore, it can also be applied to group studies where random learners give presentations or lectures. In this case, in the following description of the present invention, the learner giving the presentation or lecture is considered to be the instructor.

Below, for ease of understanding, it is assumed that one terminal is the instructor's instructor terminal and the other terminal is the learner's learner terminal. However, the scope of equality also includes the instructor's instructor terminal being the learner's learner terminal.

The learning server 10 and the terminal 20 can perform mutual data communication through a communication network such as the Internet. When used by an instructor, the terminal 20 becomes an instructor terminal, and when used by a learner, the terminal 20 becomes a learner terminal.

The learning server 10 performs all functions for operating a typical online learning system, such as registration of instructors and learners, member management, login, and payment. Furthermore, it receives overlay information from some terminal(s) and connects the remaining terminals ( s) can transmit the received overlay information. For example, overlay information is received from the instructor terminal and the received overlay information is transmitted to the learner terminal(s). Of course, the learning server may configure overlay information to be transmitted by reformatting, modifying, correcting, adding, or partially deleting the received overlay information. Details of the overlay information will be described later.

Each terminal 20 (instructor terminal and learner terminal) includes a monitor 22, a control unit 23, an operation input unit 24, an audio input unit 27, an audio output unit 26, a communication unit 25, and a camera 21. ), and can be implemented using ordinary PCs, tablets, smartphones, etc.

The communication unit 25 performs a transmission and reception function for Internet communication, the camera 21 performs a function of photographing actual textbooks held by users (instructors and learners), and the monitor 22 functions to display captured images, etc. Perform. The audio output unit 26 performs the function of outputting an audio signal, but mainly outputs an audio signal input from a remote instructor terminal (or another learner terminal) and transmitted through the learning server 10.

The manipulation input unit 24 performs input from the user, and may be, for example, a mouse, an electronic pen, a touch pad integrated with the touch screen, or a digitizer. The audio input unit 27 is mainly used to input the user's voice. The control unit 23 is equipped with software required by the terminal 20 and performs various functions thereby, including a monitor 22, an operation input unit 24, a communication unit 25, an audio input unit 27, and an audio output unit. (26) and camera (21) are controlled.

The learner terminal according to the present invention displays an image captured in real time of an actual textbook held by the learner, and displays a pointer 33 or a marker linked to the instructor's operation (input) on the image of the actual textbook displayed in real time. It's characteristic is to do something.

The instructor's terminal displays images captured in real time using the camera 21 of the actual textbook held by the instructor, generates overlay information and transmits it to the learning server 10. The overlay information includes owner ID, display location information, It can include pointer/marker tags, deletion tags, and page number information. In addition, the instructor terminal receives the instructor's voice (lecture voice) using the audio input unit 27 and transmits it to each learner terminal through the learning server 10.

Display location information is information generated from the input of the instructor pointing or marking a specific part of the physical textbook, and indicates the location of the pointer or marker in the physical textbook. A pointer or marker is input from an operation input unit 24 such as a mouse, an electronic pen, a touch pad integrated with a touch screen, or a digitizer. A pointer is something that points to a specific part, such as the pointer 33 in FIG. 4, and may have the shape of an arrow, a pointing hand, an indicator, a mouse picture, or a small circle. A marker marks a specific part and is mainly displayed in the form of a line or a dot, for example, 51, 52, and 53 in FIGS. 8 and 9.

The pointer/marker tag is a tag that distinguishes whether the display location information is the location information of a pointing pointer or the location information of a marking marker. For example, if the instructor does not press the mouse button, it is in the pointer state, and when the mouse is moved in that state, the display position information of the pointer is continuously generated and transmitted to the learning server. When the instructor clicks the left button of the mouse, it enters the marking state, and when the mouse is moved in that state, the display position information of the marker is continuously generated and transmitted to the learning server. For example, since it is desirable for the display position information of a marker to be transmitted in real time in accordance with the instructor's input, a single display position information becomes specific position information at a specific sampled point in time, and one marker is a set of continuous display position information. It could be.

The delete tag is a tag that instructs the deletion of a displayed marker, and is information transmitted when the instructor displays a marker using the manipulation input unit 24 and then inputs a command to delete the displayed marker. The owner ID is information that identifies the terminal or user transmitting the overlay information. For example, when displaying pointers or markers of multiple users on a specific learner terminal (or instructor terminal), the user ID is displayed next to these pointers or markers. It can be used to display information that can identify the user. The owner ID can also be implemented in a way that it is first created in the learning server and added to the overlay information.

The page number recognizes the page number of the physical textbook being photographed through the camera 21 of the instructor's terminal. The page number is automatically recognized as the number printed on the physical textbook, or the digitally coded page code is printed separately. It may be that it automatically recognizes what is in place. The page number transmitted from the instructor's terminal is transmitted and displayed on each learner's terminal, allowing each learner to easily know which page the instructor has placed the physical textbook on, and can be used to allow each learner to open the corresponding page accordingly. there is.

The learning server 10 receives overlay information from the instructor terminal and transmits the received overlay information to the learner terminal(s). At this time, the learning server may configure overlay information to be transmitted by reformatting, modifying, or correcting the received overlay information, or adding or partially deleting the received overlay information.

The learner terminal receives overlay information including display position information from the learning server and displays a pointer or marker using the received overlay information. The learner's terminal displays images captured in real time of the actual textbook held by the learner, and displays it together with a pointer or marker overlaid at the location according to the received display location information. The learner terminal generates a learner mapping function that indicates the correspondence between the coordinates when the physical textbook is unfolded in a completely flat plane and the coordinates in the real-time captured video of the physical textbook held by the learner, and uses the learner mapping function to determine the display position. A corrected position is obtained by correcting the position according to the information, and the learner terminal overlays and displays a pointer or marker according to the corrected position. Specific details will be described later.

Then, a pointer is displayed or a marker is displayed according to the pointer/marker tag, and when a delete command is received by the delete tag, the marker being displayed is deleted. If there are two or more markers that can be displayed simultaneously on the learner terminal, the instructor terminal additionally generates and transmits a delete marker identifier in addition to the delete tag, and the learner terminal selectively selects one of the two or more markers being displayed based on this delete marker identifier. You can also delete it.

Figure 4 shows an example of a screen displayed on a learner's terminal in an online learning system according to an embodiment of the present invention.

The camera 21 of the learner's terminal is used to photograph the area containing the learner's actual textbooks. For example, the image at this time is captured together with the desk top image 35 and the writing instrument image 36, etc. to enhance the learner's own learning. The environment can also be displayed. In addition, the actual textbook image 32 is displayed to reflect the state in which the user placed the actual textbook. In the display image 31, the real-time area where the learner's actual textbook is filmed is displayed as it is unfolded by the learner, and along with this, the learner's actual learning environment coming into the shooting angle is also filmed and displayed. Accordingly, the image 31 displayed to the learner becomes his or her own teaching materials and learning environment completely under the control of the learner.

In this situation, the pointer 33 or marker displayed on the actual textbook image 32 among the display images 31 is displayed at a location designated by the instructor in real time in conjunction with the remote instructor's manipulation (input). In addition, the audio signal input through the instructor terminal (or another learner terminal) is output from the audio output unit 26 of the learner terminal through the learning server 10. For example, the instructor conducts remote online learning by having the learner display the pointer 33 or a marker at a desired location and play audio to the learner.

In addition, the page number image 34 is displayed on the display image 31 using the page number information included in the overlay information, so that the learner knows the page of the actual textbook laid out by the instructor (or the actual textbook the instructor is lecturing on). It helps the learner to specify the page accordingly.

As a learner, the instructor's pointer (33) or marker moves and learning audio is played in the teaching materials (your learning environment) that you feel are your own, so you have the effect of feeling as if the instructor is directly teaching you (beside you). Learners have the feeling that the instructor is teaching them directly or next to them ('teacher realism'), which has the effect of increasing the learner's concentration and learning efficiency.

However, even if the instructor and the learner open the same page of the same textbook, the position (area position) of the textbook image 32 in the display image 31 is usually different. In addition, the outline, viewing angle, and size of the textbook image displayed on each monitor are different and distorted depending on the shooting conditions, such as the degree of proximity between the textbook and the camera, the angle at which the textbook is photographed, and the degree of curvature of the unfolded textbook. Even if the physical textbook is in the same location, it will be displayed in different locations on the monitor of the instructor's terminal and the monitor of the learner's terminal.

A simple way to solve this problem is to send the video captured by the learner's terminal to the instructor's terminal and have the instructor point or mark the video in the learner's textbook to solve the problem of the above-mentioned inconsistency. However, this method causes three major problems. First, if there are two or more simultaneous learners, one learner's video must be selected, and other learners, excluding the learner, learn by watching other people's textbook videos and environment videos. Therefore, there is a problem in that the realism of teaching for other learners is not improved. Second, since the instructor must teach (lecture) while watching the learning material remotely through video, there is a problem that the instructor cannot see the notes made in the textbook and conversely, it reduces the instructor's sense of reality. Third, since the videos are transmitted to each other, there is a problem that causes a decrease in quality and time delay in the videos viewed by instructors and especially other learners. Since the resolution of the transmitted video is lower than that of the video directly input by the user's terminal, there is a problem in that the resolution of the video viewed by the instructor and other learners is lowered.

In order to solve the above problems, according to the present invention, a mapping function and position correction are used.

Figure 5 shows an example in which symbols, etc. are additionally printed on an actual textbook according to an embodiment of the present invention.

In order to create a mapping function, an additional printing part is included that is additionally printed on the margin area other than the textbook contents, mainly the border area surrounding the entire contents of the textbook. The additional printing unit has page number information and is printed in the form of a barcode 41 printed long from border to border (see FIG. 5(a)), so that the user's terminal recognizes the barcode 41 and prints the corresponding page. It automatically determines the page number and allows it to be used to extract the mapping function for the page.

In addition, the additional printing unit prints a QR code 42 with page number information and a number of symbols 43 at regular intervals along the border from the border (see FIG. 5(b)), so that the user's terminal can print the QR By recognizing the code (42), the page number can be automatically determined, and the symbol (43) can be used to extract the mapping function of the page. A digitally coded page code, such as a barcode or QR code, which is information indicating the page number, is printed on the physical document. Each page of the physical textbook has symbols of a certain shape printed at a certain location, and these symbols are used when creating a mapping function, which will be described later.

The mapping function represents the correspondence relationship between the coordinates when the actual textbook is unfolded in a complete plane and the coordinates in a real-time captured image of the actual textbook, and may be composed of, for example, a mapping table.

The user's terminal constructs a virtual grid using additional printing parts (barcodes, symbols) from images of actual textbooks captured by a camera. For example, as shown in Figure 6(a), if the actual textbook is perpendicular to the optical axis of the camera, perfectly aligned horizontally and vertically, and spread out in a perfect plane, the additional printing unit can be used as shown in Figure 6(b). A perfectly square lattice network (hereinafter referred to as an 'ideal lattice') will be constructed. However, in reality, the actual textbook may be rotated because it is not perpendicular to the camera's optical axis or perfectly aligned horizontally and vertically, and is not spread out in a perfect plane but is bent due to the elasticity of the bookshelf as shown in Figure 7(b). Most of what is there is. In this situation, a virtual grid network is constructed using additional printing units (barcodes, symbols).

After receiving the image of the textbook as shown in FIG. 7(a), the user's terminal constructs a virtual grid as shown in FIG. 7(b) using the additional printing unit (barcode, symbol), and then prints the actual textbook on a completely flat surface. It is possible to construct a mapping function that represents the correspondence between the coordinates when unfolded and the coordinates in the real-time captured image of the actual textbook. The mapping function can be implemented, for example, as a mapping table.

When the physical textbook is spread out in a complete plane, the intersection point P _ij of the grid network and the intersection point Q _ij of the grid network (which is a distorted grid network) in the real-time captured image of the actual textbook of an actual user have the same pixels in each image. When doing so, you can obtain the coordinates of two intersection points in the image.

When the physical textbook is unfolded in a complete plane, the two intersection coordinate values consisting of the intersection coordinate value V(P _ij ) in the image and the intersection coordinate value V(Q _ij ) in the real-time image of the actual user's physical textbook can be corresponded. (The intersection coordinate value is a two-dimensional value that indicates the location of the intersection in the image).

A pair of intersection coordinate values, V(P _ij ) and V(Q _ij ), can be expressed as, for example, [(214, 51),(214, 38)], and these coordinate value pairs represent all intersection points of the grid. A mapping table is formed by being created in . The mapping table M can be displayed as shown below, for example, and the number of intersections (number of horizontal and vertical lines) and maximum value (range) can be set considering the resolution of the image.

M = {s|s=(V(P _ij ) V(Q _ij )), i= 1,...,n, j=1,..,m}

However, n and m are natural numbers and are the number of horizontal and vertical lines in the grid, P _ij and Q _ij are the intersection points where the ith horizontal line and jth vertical line meet, respectively, and V(P _ij ) and V(Q _ij ) are This is the intersection coordinate value of the intersection points P _ij and Q _ij and is a two-dimensional value indicating the location of the intersection point in the image.

Hereinafter, an online learning method performed in an online learning system according to an embodiment of the present invention will be described, but content that is the same as the content already described above may be omitted.

First, all users (instructor and learner(s)) open the same page of the textbook and take pictures with their own cameras 21. In order to help unfold the same page, as described above, the page number is automatically recognized from the page code included in the barcode 41 or QR code 42 captured by the instructor terminal, and the recognized page number is used as overlay information. It is included in and transmitted to each learner terminal through the learning server 10, and when a real-time captured video of the actual textbook held by the learner is displayed on the monitor of the learner terminal, the page number is displayed as a page number image (see 34 in FIG. 4). It is displayed. Alternatively, the instructor terminal automatically recognizes the printed page number in the captured video, the recognized page number is included in the overlay information and is transmitted to each learner terminal through the learning server 10, and the learner displays the page number on the monitor of the learner terminal. When displaying an image captured in real time of an actual textbook, the page number is displayed as a page number image (see 34 in FIG. 4).

In addition, the learner terminal can automatically recognize the page number of the learner's textbook, compare it with the page number of the instructor's textbook received, and send an alarm when they are different. For example, an alarm method such as blinking the instructor's page number displayed on the monitor of the learner's terminal can be used.

And, as described above, a mapping function is generated on each user's terminal, and on the instructor's terminal, there is a correspondence relationship between the coordinates when the physical textbook is unfolded in a complete plane and the coordinates in the real-time captured video of the physical textbook held by the instructor. Create an instructor mapping function that represents. In addition, each learner terminal also generates a learner mapping function(s) that represents the correspondence between the coordinates when the physical textbook is unfolded in a complete plane and the coordinates in the real-time captured video of the physical textbook held by each learner. . For each learner, the extent of the area occupied by the textbook, the degree of proximity between the textbook and the camera, the angle at which the textbook is photographed, and the degree of curvature of the unfolded textbook vary, so the learner mapping function generated at each learner's terminal is naturally different. do.

Even when two or more learners study with the same instructor, the learner mapping function is generated for each learner terminal, and the correction position, which will be described later, from the same display position information may be different for each learner terminal.

The instructor's terminal displays real-time images of the actual textbook held by the instructor, generates display location information from the input of the instructor pointing or marking a specific part of the physical textbook, and includes this in the overlay information to be sent to the learning server. send to At this time, the instructor terminal uses the instructor mapping function to obtain a correction position that corrects the position where the instructor points or marks a specific part of the physical textbook, and generates the display position information based on the correction position.

When the real-time video of the instructor's terminal captures the actual textbook held by the instructor is the same as the one shown in Figure 8(a) (excluding the grid), the marker 51 shown by the instructor over a certain period of time is shown in Figure 8(a). When drawing the marker 51, at each sampling cycle, the instructor recognizes the location (coordinates) of the point where the actual textbook is being marked at that timing, and the recognized location (coordinates) is used to map the actual textbook using the instructor mapping function described above. When the textbook is unfolded in a perfect plane, the position (coordinates) is corrected. This correction position is included in the overlay information as display position information and is transmitted to each learner terminal through the learning server. If the instructor draws the marker 51 as shown in FIG. 8(a) for a certain period of time, the marker 52 is drawn at the same position as shown in FIG. 8(b) on the virtual standard image (flat image with correction for distortion). Information (more precisely, a set of continuous location information) is transmitted to each learner.

In this way, the coordinate values in the actual grid reflecting the distortion are converted to coordinate values in the ideal grid. For example, the coordinate value of the intersection point Q _ij is converted to the coordinate value of the intersection P _ij , and at a position other than the intersection point, the coordinate value is converted to the coordinate value of the ideal grid network. If it is a coordinate value, it can be converted by a known interpolation method. Therefore, the display location information transmitted to each learner terminal through the learning server is transmitted as coordinate values according to an ideal grid (that is, coordinates when the physical textbook is unfolded in a complete plane and coordinate values in the ideal state).

In addition, each learner's learner terminal receives display position information, which is information indicating the position to point or mark a specific part of the physical textbook, from the learning server, and overlays a pointer or marker at a position according to the received display position information. Display.

The learner terminal displays an image captured in real time of the actual textbook held by the learner, and extracts a displacement value (dx, dy) indicating the starting position of the textbook image (page image) (32; see Figure 4) from the display image. However, this can also be created when creating a learner mapping function. Then, the learner terminal obtains a corrected position by correcting the position according to the display position information using its own learner mapping function, shifts the corrected position by the displacement value, and displays a pointer or marker overlaid on the textbook image.

For example, as shown in Figure 9(b), in a state where a learner's terminal is displaying an image of his/her physical textbook (excluding the grid), overlay information is displayed through the learning server. When receiving the display position information of the included marker (display position information when the pointer/marker tag indicates that it is a marker), first use the learner mapping function to display the marker (part of the marker, to be precise) according to the display position information. to perform position correction. The virtual marker 52 shown in Figure 9(a) is displayed like the marker 53 shown in Figure 9(b) according to the position correction according to the learner mapping function. If any part has already been drawn so far, It may be displayed in an additional form.

Additional printing parts such as barcodes, QR codes and symbols are printed on the textbook, but can be made invisible to the user by being printed, for example, with infrared fluorescent ink, which is identified by an infrared camera when irradiated by an infrared lamp. there is.

In addition, a mapping function can be created without having to configure an additional printing part in the textbook, and the result of recognizing the outline of the page can be used. Usually, the outline of a page reflects the curvature of the page, so a mapping function can be created by recognizing the outlines that make up the four sides of the page. In addition, since each page of each textbook usually consists of a number of character lines, there is a virtual envelope surrounding each character line (or a virtual envelope that surrounds only one direction from the top or bottom of each character line). A mapping function can be created using the result of recognizing the virtual envelope.

In the above, a pointer or marker operated by an instructor is provided to the learner terminal, but simultaneously or separately, a pointer or marker operated by one learner is displayed on the instructor terminal and another learner terminal in a similar manner. For example, if a learner (student) wants to ask the instructor a question about a specific part of the textbook and underlines a specific part of the textbook with his or her mouse, a line will be drawn at the same location on the instructor's textbook screen.

Additionally, a plurality of pointers or markers can be displayed simultaneously on the user's terminal. In this case, each pointer/marker can be tagged with the user's name, nickname, or ID and displayed together.

According to the present invention, a pointer or marker is displayed in accordance with the distorted teaching material image for each learner's terminal, so that the learner feels as if the instructor is teaching by directly looking at his/her teaching material. In particular, multiple learners Even when students are learning at the same time, the realism of teaching is improved for all learners. In addition, as an instructor, there is an advantage in being able to teach while watching your own textbook rather than teaching (lecturing) by watching a video of a remote learning material. Since there is no need to transmit videos to each other, the communication bandwidth is used small, reducing transmission quality and time. It has the effect of eliminating problems such as delay. Additionally, since instructors and learners learn by watching videos input from their own terminals, there is an advantage in that the video resolution is high. According to the present invention, a clear online learning material screen can be secured. In the existing method, image quality had to be intentionally degraded due to communication traffic load, and if the number of people participating online was large, image quality deterioration was bound to occur seriously. However, according to the present invention, rather than transmitting a video, you can view a locally created video in high definition on your PC screen.

Additionally, according to the present invention, since the physical textbook can be used as is, there is no need to create separate materials or electronic files. Textbooks and reference books have already gone through a detailed writing and editing process, so they can be used as is. Nevertheless, in most conventional remote video education systems, instructors created textbooks separately using PowerPoint, etc., because the resolution was greatly reduced when images captured from textbooks were transmitted. For that reason, even though I had a textbook, I had to go through the trouble of creating separate lecture materials using PowerPoint, etc. The problem is that despite these additional efforts, the educational effect on learners (students) is low. During the pandemic, not only universities but also middle and high schools conducted remote classes in this way, but students said that the learning effectiveness was reduced because they had to take classes while looking at textbooks and other PowerPoint presentations.

10: Learning server 20: User terminal (instructor terminal, learner terminal)
21: Camera 22: Monitor
23: control unit 24: operation input unit
25: communication unit 26: audio output unit
27: Audio input unit

Claims (13)

In online learning methods,
A first step in which the learner's learner terminal receives display position information, which is information indicating a position to point or mark a specific part of the physical textbook, from the learning server;
A second step in which the learner terminal displays an image captured in real time of an actual textbook held by the learner, and displays it together by overlaying a pointer or marker at a position according to the display location information received in the first step. Characterized in that,
How to learn online. In claim 1,
The second step is,
A 2-1 step of generating a learner mapping function that represents the correspondence between the coordinates when the physical textbook is unfolded in a complete plane and the coordinates in a real-time captured image of the physical textbook held by the learner;
A 2-2 step of obtaining a corrected position by correcting the position according to the display position information using the learner mapping function, and displaying the learner terminal by overlaying a pointer or marker based on the corrected position.
How to learn online. In claim 2,
When two or more learners study with the same instructor,
The learner mapping function generated in step 2-1 is generated for each learner terminal, and the correction position generated in step 2-1 from the same display location information may be different for each learner terminal.
How to learn online. In claim 1,
The display location information is generated by the instructor's instructor terminal and transmitted to the learning server,
In a state where the instructor's terminal is displaying a real-time captured video of the physical textbook held by the instructor, the display location information is generated from the input of the instructor pointing or marking a specific part of the physical textbook and transmitted to the learning server. A third step; further comprising,
How to learn online. In claim 4,
The physical document has a digitally coded page code printed with information indicating the page number.
After the page number according to the page code is recognized in the real-time video of the actual textbook held by the instructor, it is transmitted to the learner terminal through the learning server,
The learner terminal displays the received page number together when displaying a video captured in real time of the actual textbook held by the learner.
How to learn online. In claim 4,
After the printed page number is recognized in the real-time video of the actual textbook held by the instructor, it is transmitted to the learner's terminal through the learning server,
The learner terminal displays the received page number together when displaying a video captured in real time of the actual textbook held by the learner.
How to learn online. In claim 4,
The third step is,
Step 3-1 of generating an instructor mapping function that represents the correspondence between the coordinates when the physical textbook is unfolded in a complete plane and the coordinates in the real-time captured image of the physical textbook held by the instructor;
A 3-2 step of using the instructor mapping function to obtain a correction position that corrects the position at which the instructor points or marks a specific part of the physical textbook, and generating the display position information based on the correction position. ,
How to learn online. In claim 2 or claim 7,
On each page of the physical textbook, symbols of a certain shape are printed at a certain location,
When creating the learner mapping function in step 2-1, or creating the instructor mapping function in step 3-1, the symbols are used,
How to learn online. In claim 8,
wherein the printed symbols are identified by an infrared camera upon irradiation with an infrared lamp,
How to learn online. In claim 2 or claim 7,
When creating the learner mapping function in step 2-1, or when creating the instructor mapping function in step 3-1,
Using the results of recognizing the outline of the page, or using the results of recognizing the virtual envelope surrounding the row of letters,
How to learn online. A learning server that transmits display position information, which is information indicating the position to point or mark a specific part of the physical textbook;
A learner terminal that receives the display location information from the learning server and displays a real-time image of an actual textbook held by the learner by overlaying a pointer or marker at a location according to the received display location information and displaying it together. Characterized in that,
Online learning system. In claim 11,
The learner terminal is,
Generates a learner mapping function that represents the correspondence between the coordinates when the physical textbook is unfolded in a complete plane and the coordinates in the real-time video of the physical textbook held by the learner,
Obtaining a corrected position by correcting the position according to the display position information using the learner mapping function, and displaying the learner terminal by overlaying a pointer or marker based on the corrected position,
online learning system In claim 11,
It further includes; an instructor terminal that generates the display location information from an input of the instructor pointing or marking a specific part of the physical textbook and transmits it to the learning server while displaying a real-time captured image of the physical textbook held by the instructor; doing,
Online learning system.