KR102442783B1 - Apparatus for Learning Service based on Motion and Driving Method Thereof - Google Patents

Abstract

The present invention relates to a motion-based learning service apparatus and a method of driving the apparatus, wherein the motion-based learning service apparatus according to an embodiment of the present invention acquires the instructor's motion at each step with respect to the practice sequence of the practice subject. Based on the storage unit that stores motion data, and the analysis results of the recorded images of students practicing the practical subject and the stored motion data, the degree of agreement at each stage of the students is analyzed and, when deviating from the standard, the student who deviates from the standard The student terminal device may include a control unit that provides a video of the instructor's practice.

Description

Apparatus for Learning Service based on Motion and Driving Method Thereof

The present invention relates to a motion-based learning service device and a driving method of the device, and more particularly, to a motion-based learning service device that enables efficient class guidance of students even through a single professor during a class in a designated practical course at a university. It relates to a learning service apparatus and a driving method of the apparatus.

In university classes, there are many cases where learning is carried out through various practical exercises. Each practice course is usually taught individually or in groups. However, there is a problem in that sufficient one-to-one classes cannot be conducted in guidance for each student or group because the number of students that one professor has to take charge of during these class times is large. For example, in departments such as the Department of Dental Engineering, work must be done in a certain order in the process of making dentures, and it is impossible to teach this process to all the students participating in the class within a limited time.

Korean Patent Publication No. 10-2016-0040750 (2016.04.15)

An object of the present invention is to provide a learning service device capable of efficiently teaching students even through one professor (self) during a class in a designated practical course of a university, and a method of driving the device.

A motion-based learning service apparatus according to an embodiment of the present invention includes a storage unit that acquires the instructor's motion for each step in the practice sequence of a practical subject and stores it as motion data, and a photographed image of students practicing the practical subject Based on the analysis result and the stored motion data, it analyzes the degree of agreement at each stage of the students and, when deviating from the standard, includes a control unit that provides a video of the instructor's practice to the student terminal device of the student who deviates from the standard do.

The control unit may transmit information to the students deviating from the standard to the instructor's teaching terminal device so that the guidance of the students with errors is made prior to the provision of the practice video.

The learning service device may further include a communication interface unit configured to receive the captured image by performing communication with at least one device among a wearable device worn by the students, a smartphone, and a camera installed in a practice space where the training is performed. .

The control unit stores motion data acquired through a motion sensor attached to the body of the instructor before the practice as reference motion data in the storage unit, and stores the reference motion data as a result of motion analysis for students of the captured image Concordance can be analyzed by comparing

In addition, the method of driving a motion-based learning service apparatus according to an embodiment of the present invention is a method of driving a motion-based learning service apparatus including a storage unit and a control unit. Acquiring motion and storing it in the storage unit as motion data, and the control unit in each step of the students based on the analysis result of the captured images of students practicing the practical subject and the stored motion data and, when deviating from the standard by analyzing the degree of agreement, providing a video of the instructor's practice to the student terminal device of the student deviating from the standard.

In the providing step, prior to the provision of the practice video, information of students deviating from the standard may be transmitted to a teaching terminal device of the instructor so that the guidance of students with errors is made.

The driving method may further include receiving the captured image by performing communication with at least one device among a wearable device worn by the students, a smartphone, and a camera installed in a practice space where the training is performed.

The driving method includes the steps of storing motion data acquired through a motion sensor attached to the body of the instructor before the practice as reference motion data in the storage unit, and storing the reference motion data for students of the captured image. The method may further include analyzing the degree of agreement by comparing it with the motion analysis result.

According to an embodiment of the present invention, after the instructor stores data based on motion recognition in advance for a designated practice subject, compares it with the photographed images of the students taken during the practice, checks the work order of each step of the students, and the confirmation result Accordingly, the instructor's student guidance will be very easy by making the guidance centering on the students with errors.

1 is a view showing a learning service system according to an embodiment of the present invention;
2 is a diagram schematically showing the operation process of the learning service system of FIG. 1;
3 is a block diagram illustrating a detailed structure of the learning service apparatus of FIG. 1, and
4 is a flowchart illustrating a learning service process according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a diagram illustrating a learning service system according to an embodiment of the present invention, and FIG. 2 is a diagram schematically illustrating an operation process of the learning service system of FIG. 1 .

As shown in FIG. 1 , the learning service system 90 according to an embodiment of the present invention includes some or all of the instructor device 100 , the learner device 110 , the communication network 120 , and the learning service device 130 . include

Here, "including some or all" means that some components such as the communication network 120 are omitted so that the instructor device 100 or the learning service device 130 performs direct communication (eg, P2P communication) or learning It means that a part or all of the service device 130 can be configured by being integrated into a network device (eg, a wireless switching device, etc.) in the communication network 120, and includes all of it to help a sufficient understanding of the invention. Explain.

The instructor device 100 includes various devices such as a motion acquisition device (eg, a sensor, a marker, etc.) of the instructor, a desktop computer, a laptop computer, a tablet PC, a smart phone, etc., and may further include a photographing device such as a camera. . For example, in the process of manufacturing dentures made in departments such as the dental engineering department of the university, the operation is performed according to a predetermined sequence, and the instructor device 100 stores motion data generated by the instructor in relation to the sequence of these operations. Of course, motion data may be collected through various paths even if it is not directly produced by the instructor. In order to produce motion data, the instructor directly performs the corresponding work sequence, and in this case, the instructor device 100 acquires motion data to secure reference motion data that can be a standard for practice evaluation.

In this way, in order to generate motion reference data in relation to each sequence of tasks that students need to perform in practice, the instructor uses data acquired through a motion sensor attached to the body in advance, or uses markers ( After attaching the marker), it may be possible to generate motion data based on the marker in the captured image taken with the camera. For example, since the corresponding marker will be photographed in the image with the marker attached, the motion of each sequence may be determined by analyzing the movement based on the coordinates of the corresponding marker for each unit frame image. Of course, the coordinates are preferably three-dimensional coordinates, but in the case of two-dimensional coordinates, it may be desirable to secure the three-dimensional coordinates. For example, after converting a 2D image into a 3D image, it may be secured through this.

In addition, for motion acquisition, a marker-free motion capture method that does not use a motion sensor or marker may be possible. It is a technology that can capture motion in real time for an operator in a free dress state, that is, an instructor. There are almost no restrictions on movement and natural features as instructors can work without attaching optical, magnetic, or acoustic markers. A method of recovering motion by analyzing the silhouette of a human body or a method of detecting each part of the body separately by analyzing a blob, and capturing the motion of the human body through the process of restoring their three-dimensional position and structuring the motion may be utilized.

In addition, the instructor device 100 stores the motion data produced and generated in advance in the DB 130a of FIG. 1 , and then receives the information of the students provided based thereon. In other words, when the practice time progresses, the shooting of the students' practice can be performed at the practice location, and the learning service device 130 receives and analyzes the captured images of the students' work order and stores them in the DB 130a. It compares the instructor's motion data and judges the degree of motion matching through this. If, as a result of the determination of the degree of agreement, it is determined that there is an error, the user information about the student is transmitted to the instructor device 100, for example, a terminal device such as a smartphone of the instructor in the practice place, so that the corresponding You will be able to conduct student guidance. It may be desirable to transmit such information in synchronization with the operation sequence of each step.

In this way, when the instructor instructs the practice of many students at the practice location, since it is possible to observe the progress of the practice even for students who go out of their field of view, the effectiveness of the practice guidance may be increased.

For example, since the instructor device 100 can receive information in real time in relation to the work order of students in the same space, it is possible to immediately correct errors in the work order of the students, but for more efficiency, It may be desirable to receive information about which task sequence an error occurred when the information of Therefore, it becomes possible for the instructor in the same space to lead the students with sufficient time.

The learner device 110 may include a wearable device, a smart phone, or a camera installed in the corresponding practice space, which the learner possesses, more precisely, which can photograph the learner's practice process. Of course, the camera can be a camera owned by the learner, but it is desirable to be installed on the wall or ceiling in the practice space so that all students can be photographed together. It is also possible to take pictures for each divided area.

Through such various methods, the sequence of the students' practice work performed in the practice space is photographed, and the photographed image is transmitted to the learning service device 130 . Accordingly, if an error occurs in their work sequence, students can receive guidance from the instructor immediately or within a short time. In addition, for example, if data related to the work sequence of the corresponding practice is required even after the class is over, the instructor's practice video may be received from the learning service device 130 .

The communication network 120 includes both wired and wireless communication networks. For example, a wired/wireless Internet network may be used or interlocked as the communication network 120 . Here, the wired network includes an Internet network such as a cable network or a public telephone network (PSTN), and the wireless communication network includes CDMA, WCDMA, GSM, Evolved Packet Core (EPC), Long Term Evolution (LTE), and Wibro networks. meaning to include Of course, the communication network 120 according to the embodiment of the present invention is not limited thereto, and may be used as an access network of a next-generation mobile communication system to be implemented in the future, for example, a cloud computing network under a cloud computing environment, a 5G network, and the like. For example, when the communication network 120 is a wired communication network, the access point in the communication network 120 can connect to a switching center of a telephone company, etc., but in the case of a wireless communication network, it accesses the SGSN or GGSN (Gateway GPRS Support Node) operated by the communication company to transmit data. or by connecting to various repeaters such as Base Station Transmission (BTS), NodeB, and e-NodeB to process data.

The communication network 120 may include an access point. Here, the access point includes a small base station, such as a femto or pico base station, which is often installed in a building. Here, the femto or pico base station is classified according to the maximum number of access to the instructor device 100, the learner device 110, etc., in terms of the classification of the small base station. Of course, the access point includes a short-distance communication module for performing short-distance communication such as Zigbee and Wi-Fi with the instructor device 100 and the learner device 110 . The access point may use TCP/IP or Real-Time Streaming Protocol (RTSP) for wireless communication. Here, short-range communication may be performed in various standards such as Bluetooth, Zigbee, infrared, radio frequency (RF) such as ultra high frequency (UHF) and very high frequency (VHF), and ultra-wideband communication (UWB) in addition to Wi-Fi. Accordingly, the access point may extract the location of the data packet, designate the best communication path for the extracted location, and deliver the data packet to the next device, for example, the learning service device 130 along the designated communication path. The access point may share several lines in a general network environment, and includes, for example, a router, a repeater, and a repeater.

The learning service device 130 may provide a motion-based learning service. More precisely, for example, when a practice is performed at a university or a private institute, the learning service device 130 provides information about the practice stored in advance so as to quickly guide students, that is, learners, who may have errors in the process of the operation sequence of the practice, etc. The motion data related to the work order is compared with the students' recorded images to analyze the degree of motion matching, and guidance is made based on the results. In the students' captured images, the method of calculating the coordinate values by tracking the joint area through image analysis may be used.

Referring to FIG. 2 in more detail, the instructor first inputs a specific practice sequence to the learning service device 130 through motion-based technology for each step. The method of generating data based on the motion technology for the operation sequence has been sufficiently described above. And during the actual class, students can wear a wearable device or use their own smart phone to record their practice process as a video in real time and transmit it to the learning service device 130 . The learning service device 130 evaluates whether the image received from each student matches the motion entered by the instructor. If a predetermined degree of agreement is not reached by analyzing the training sequence and the motion corresponding to each sequence, for example, out of a reference value, the learning service device 130 provides a training video of the instructor or through the instructor's smart terminal. By providing the information of the student in error, the professor can guide the student in real time without the student having to ask the instructor a separate question or ask a question.

As mentioned above, the learning service device 130 may operate in various ways according to the intention of the system designer. In other words, it is assumed that an error has occurred for a plurality of students based on the analysis result of the student's captured image and the pre-stored motion data. In this case, the learning service device 130 directly transmits information about a plurality of students to the instructor device 100 . The instructor can provide guidance based on the information. Therefore, in order to acquire students' information from the captured image, pre-registered user information (eg, name) may be transmitted through face recognition at the beginning of class. In this case, since it may be difficult to teach a plurality of students rather than simply transmitting user information such as a name, information on the work order may be transmitted together. Accordingly, the learning service device 130 may determine the form of transmitting information according to the number of students who have made an error in each sequence.

In addition, the learning service device 130 may provide a video of the instructor's practice in the case of a student who has an error in the work order in the practice process or when the student separately requests it. When providing a video, a full video may be provided, but it may be possible to edit and provide an error-prone part. As described above, the learning service device 130 according to the embodiment of the present invention can be operated in various ways according to the intention of the system designer, and above all, the embodiment of the present invention allows the instructor who must guide many students in practice. Based on this pre-constructed task sequence data, it may be in any form as long as it is possible to make the students' guidance more efficient.

FIG. 3 is a block diagram illustrating a detailed structure of the learning service apparatus of FIG. 1 .

As shown in FIG. 3 , the learning service apparatus 130 of FIG. 1 according to an embodiment of the present invention includes a communication interface unit 300 , a control unit 310 , a motion-based learning processing unit 320 and a storage unit 330 . includes some or all of

Here, "including some or all" means that some components such as the storage unit 330 are omitted, or some or all of the components such as the motion-based learning processing unit 320 are the same as the control unit 310. It means that it can be configured by being integrated with other components, and it will be described as including all in order to help a sufficient understanding of the invention.

The communication interface unit 300 communicates with the instructor device 100 and the learner device 110 via the communication network 120 of FIG. 1 . In this process, the communication interface unit 300 performs operations such as modulation/demodulation, muxing/demuxing, encoding/decoding, and encryption/decryption. The communication interface unit 300 receives the captured image or sensing data from the instructor device 100 , for example, a camera or a motion acquisition device (eg, a motion sensor, etc.) and provides it to the control unit 310 .

In addition, the communication interface unit 300 receives the captured images of the learners in practice from the learner device 110 and provides them to the control unit 310 . To this end, the communication interface unit 300 is a terminal device possessed by students in the practice space via the communication network 120 of FIG. 1 or a fixed camera installed in the practice space, furthermore, a plurality of fixed cameras, and in addition, the students directly installed can communicate with the camera.

Furthermore, the communication interface unit 300 transmits, to the instructor device 100 , information about students having errors in the work sequence analyzed based on the pre-stored motion data of the instructor and the photographed images of the students. The information about the students may be information about the students' names, etc., and may be a photograph of a captured face. For example, before a practical class starts, the learning service device 130 may recognize a student's face through a photographed image and extract information about the student's name and the like by comparing it with a pre-stored photo. In other words, in the DB 130a of FIG. 1 , the students' photos, that is, the photo data and information about their names can be matched and stored, so the students' faces in the captured image are compared with the data stored in the DB 130a and matched thereto. Student information may be transmitted to the instructor device 100 .

Of course, while transmitting student information, the communication interface unit 300 may also provide information on the work order. For example, if a particular exercise consists of tasks A, B, and C, send the title or information for that particular task. This information could be transmitted in real time to correct student work order errors in the lab space.

In addition, when the control unit 310 or the motion-based learning processing unit 320 checks the students who have errors in the work order, etc., and the class ends, the communication interface unit 300 may transmit the instructor's practice video. Here, the timing or method of providing the practice video may be, for example, provided immediately during class, provided after the class is over, provided by editing only the relevant part, or provided as a full video, etc. .

The control unit 310 is in charge of overall control operations of the communication interface unit 300 , the motion-based learning processing unit 320 , and the storage unit 330 . As mentioned above, the control unit 310 receives from the communication interface unit 300 the instructor's motion data (or photographed images), the students' photographed images, etc. for the work sequence of the practice subject, and temporarily stores it in the storage unit 330 . After doing so, it may be called and provided to the motion-based learning processing unit 320 .

In addition, the control unit 310 controls the communication interface unit 300 to provide information about students who have an error in the work order to the instructor device 100 according to the request of the motion-based learning processing unit 320, or It will be possible to control the communication interface unit 300 to provide the instructor's practice video to the generated students.

The motion-based learning processing unit 320 evaluates whether motion is consistent by comparing pre-made motion data, or captured images, and captured images of students practicing in the practice space in relation to the work sequence of the practice process. If the motion data is, for example, information about movement of an arm or head for performing a specific task sequence, the motion-based learning processing unit 320 analyzes the students' captured images to determine whether the motion data is performed in each student's motion. can judge Since motion data can be stored in the form of three-dimensional coordinate values (x, y, z), it is preferable that a three-dimensional image is used for the captured image, and even if it is a two-dimensional image, it can be changed to a three-dimensional image or a three-dimensional image. After acquiring 3D coordinates through a data acquisition program, it will be possible to compare them based on this.

The motion-based learning processing unit 320 analyzes and compares the task sequence of a specific task in real time, and when an error occurs in a specific student, extracts student information through face recognition of the student, etc., and transmits the error data together with the student information to the instructor A request may be made to the controller 310 to transmit to the device 100 . Here, the error data may indicate a title related to a work sequence or may be a shot image edited for a certain amount of time based on time change.

In addition, the motion-based learning processing unit 320 may provide or provide a video of the instructor's practice to the student's learner device 110 at the time the practice is finished according to the intention of the system designer, more precisely, the service provider. Of course, it would be possible to send it by e-mail as well.

Furthermore, the motion-based learning processing unit 320 may process data to have a specified format in order to provide a service and store it in the DB 130a of FIG. 1 .

The storage unit 330 may store information or data processed under the control of the control unit 310 . Information may mean simple control signals such as request and response, and data may mean pixel data, etc., but the embodiment of the present invention will not specifically limit to these terms. The stored data may be output according to the request of the control unit 310 and provided to the motion-based learning processing unit 320 and the like. Of course, the reverse operation is also possible.

Meanwhile, as another embodiment of the present invention, the control unit 310 of FIG. 3 may include a CPU and a memory, and the CPU and the memory may be formed into a single chip. The CPU includes a control circuit, an arithmetic unit (ALU), a command interpreter and a registry, and the memory may include a RAM. The control circuit may include a control operation, the operation unit may include an operation of binary bit information, and the instruction interpretation unit may include a program such as a compiler. can do. Also, the registry may be involved in software data storage.

According to the above configuration, the CPU will be able to quickly increase the data operation processing speed by copying the data stored in the motion-based learning processing unit 320 of FIG. 3 and loading it into the memory and then executing it.

4 is a flowchart illustrating a learning service process according to an embodiment of the present invention.

Referring to FIG. 4 together with FIG. 1 for convenience of explanation, the learning service device 130 according to an embodiment of the present invention acquires the instructor's motion at each step with respect to the practice sequence of the practice subject and stores it in the storage unit as motion data. do (S400). For example, in the case of the dental engineering department of a university, the work must be done in a certain order in the process of manufacturing dentures in the department, and therefore "each step with respect to the order of practice" above means the work in a certain order in the process of making dentures. can do.

In the embodiment of the present invention, it has been previously described that there may be various methods of acquiring the instructor's motion and generating and constructing motion data. Using the sensing data of the motion sensor attached to the body of the instructor, especially the joints, attaching a marker to the body and photographing it to acquire the instructor's motion through the captured image, or even marker-free motion capture may be possible. Various methods may be used, such as simply wearing clothes, or by analyzing the human body's silhouette or blobs as mentioned above.

In addition, when the learning service device 130 of FIG. 1 deviates from the standard by analyzing the degree of agreement at each stage of the students based on the analysis results of the captured images of the students practicing the practical subject and the (group) stored motion data, the learning service device 130 of FIG. The instructor's practice video is provided to the student terminal device of the student deviating from the standard, for example, the learner device 110 of FIG. 1 (S410).

Of course, the learning service device 130 transmits the student information of the students who have made an error to the instructor device 100 before providing the practice video to recognize the students who have made an error right away in the practice space and to make corrections. In other words, the practice video may or may not be provided at the end of the practice class.

In addition to the above, the learning service device 130 may perform various operations, and since this has been sufficiently described above, it will be replaced with the contents.

On the other hand, even though it has been described that all components constituting the embodiment of the present invention are combined or operated in combination, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more. In addition, all of the components may be implemented as one independent hardware, but a part or all of each component is selectively combined to perform some or all functions of the combined components in one or a plurality of hardware program modules It may be implemented as a computer program having Codes and code segments constituting the computer program can be easily deduced by those skilled in the art of the present invention. Such a computer program is stored in a computer-readable non-transitory computer readable media, read and executed by a computer, thereby implementing an embodiment of the present invention.

Here, the non-transitory readable recording medium refers to a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, memory, etc. . Specifically, the above-described programs may be provided by being stored in a non-transitory readable recording medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, preferred embodiments of the present invention have been illustrated and described, but the present invention is not limited to the specific embodiments described above, and it is common in the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims. Various modifications may be made by those having the knowledge of, of course, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

100: instructor device 110: learner device
120: communication network 130: learning service device
300: communication interface unit 310: control unit
320: motion-based learning processing unit 330: storage unit

Claims (5)

a storage unit that acquires the instructor's motion for each step of the work sequence of the practice subject and stores it as motion data; and
Based on the analysis result of the captured images of the students practicing the practical subject and the stored motion data, the degree of agreement at each stage of the students is analyzed and, when deviating from the standard, the student terminal device of the student deviating from the standard A control unit that provides a video of the instructor's practice; including,
The control unit provides the training video when there is a request from a student or a student who has an error in the work sequence in the practice process,
The control unit transmits, prior to the provision of the practice video, information on students deviating from the standard to the instructor's teaching terminal device in the practice location of the students so that the guidance of the students with errors is made,
The control unit transmits information of students deviating from the standard in synchronization with the work sequence for each step, and transmits information about which task an error occurred when transmitting the information together,
The control unit, a motion-based learning service device for determining a form of transmitting the information according to the number of students who have an error in each task sequence. delete The method of claim 1,
A motion-based learning service device further comprising a wearable device worn by the students, a smart phone, and a communication interface unit for receiving the captured image by communicating with at least one device among a camera installed in a practice space where the practice is performed; . The method of claim 1,
The controller stores motion data acquired through a motion sensor attached to the body of the instructor before the practice as reference motion data in the storage unit, and stores the reference motion data as a result of motion analysis for students of the captured image A motion-based learning service device that compares and analyzes the degree of matching. A method of driving a motion-based learning service device including a storage unit and a control unit, the method comprising:
Acquiring the motion of the instructor at each step with respect to the work sequence of the practice subject and storing the motion data in the storage unit; and
When the control unit deviates from the standard by analyzing the degree of agreement at each stage of the students based on the analysis result of the captured images of students practicing the practice subject and the stored motion data, the student of the student who deviates from the standard Including; providing a video of the instructor's practice to a terminal device;
The control unit provides the training video when there is a request from a student or a student who has an error in the work sequence in the practice process,
The control unit transmits, prior to the provision of the practice video, information on students deviating from the standard to the instructor's teaching terminal device in the practice location of the students so that the guidance of the students with errors is made,
The control unit transmits information of students deviating from the standard in synchronization with the work sequence for each step, and transmits information about which task an error occurred when transmitting the information together,
The control unit, a method of driving a motion-based learning service device comprising determining a form of transmitting the information according to the number of students who have made an error in each task sequence.

Images