KR20210076432A - Apparatus and method for providing immersive learning content through text recognition - Google Patents

Abstract

The present invention relates to a device and a method for providing an immersive learning content through text recognition. According to an embodiment of the present invention, the device for providing the immersive learning content includes: a character recognition module which recognizes handwritten text by a user; a control module for identifying the immersive learning content matching the recognized handwritten text; and a display module for outputting the identified immersive learning content.

Description

Apparatus and method for providing realistic learning content through text recognition {APPARATUS AND METHOD FOR PROVIDING IMMERSIVE LEARNING CONTENT THROUGH TEXT RECOGNITION}

The present invention relates to an apparatus and method for providing immersive learning content based on virtual reality, complex reality, augmented reality, etc. through recognition of text written by a learner.

Due to the recent development of various digital devices, there is a trend to adopt an educational method incorporating digital devices in an educational environment.

Reflecting this trend, the government has been distributing digital textbooks for the upper grades of elementary school and the first grade of middle school since 2018 as part of a smart education policy, and is presenting an educational curriculum using digital textbooks.

In line with this, recently, educational methods using various digital devices are being developed.

However, the education method up to now is a method in which a digital device is simply grafted onto the existing one-way education method in which the teacher injects knowledge into the learner, and the learner cannot interact with the digital device through his/her own actions, thereby reducing the learner's concentration. There was a limit to how difficult it was to draw.

An object of the present invention is to provide an apparatus and method for providing immersive learning content based on virtual reality, complex reality, augmented reality, etc. through recognition of text written by a learner.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned may be understood by the following description, and will be more clearly understood by the examples of the present invention. Moreover, it will be readily apparent that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

To achieve the above object, an apparatus for providing immersive learning content according to an embodiment of the present invention includes a character recognition module for recognizing handwritten text written by a user, and a method for identifying immersive learning content matching the recognized handwritten text. and a control module and a display module for outputting the identified immersive learning content.

In one embodiment, the character recognition module is characterized in that it comprises a camera for photographing the handwritten text.

In one embodiment, the character recognition module converts a file corresponding to the image captured by the camera into a file to which an OCR (Optical Character Recognition) function can be applied, and recognizes handwritten text in the converted file. characterized in that

In an embodiment, the character recognition module captures an area including the handwritten text, extracts a handwriting area within the photographed area, and recognizes the handwritten text from the extracted handwriting area.

In an embodiment, the character recognition module extracts the writing area by pattern matching a file corresponding to the photographed area and a reference document file.

In an embodiment, the control module identifies the realistic learning content matching the handwritten text by referring to a look-up table (LUT) stored in a memory.

In an embodiment, the display module outputs the immersive learning content as a virtual image.

In an embodiment, the display module is characterized in that the immersive learning content and the live-action image are output together.

In one embodiment, the display module is characterized in that the output of the immersive learning content to the live-action image as an augmented image.

In an embodiment, the display module is a wearable device.

In an embodiment, the display module includes at least one of a Head Mounted Device (HMD), a video see-through device, an optical see-through device, and a projector. characterized in that

In addition, the method for providing realistic learning content according to an embodiment of the present invention for achieving the above object includes the steps of recognizing a handwritten text handwritten by a user, and realistic learning matching the recognized handwritten text with reference to a memory It characterized in that it comprises the step of identifying the content and outputting the identified immersive learning content.

In an embodiment, the step of recognizing the handwritten text written by the user may include photographing an area including the handwritten text, extracting a handwriting area from within the photographed area, and extracting the handwritten text from the extracted handwriting area. and recognizing the handwritten text.

The present invention provides immersive learning content based on virtual reality, complex reality, augmented reality, etc. through recognition of text written by the learner, so that the learner can use his/her own handwritten text to create immersive learning content. By controlling it, not only can the learning efficiency be improved by inducing the learner's participation in education, but also various contents can be used for learning and education in an interactive way, thereby improving the leadership of the educator.

In addition to the above-described effects, the specific effects of the present invention will be described together while describing specific details for carrying out the invention below.

1 is a diagram schematically illustrating components of an apparatus for providing immersive learning content according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating an embodiment of the apparatus for providing immersive learning content shown in FIG. 1;
3 is a flowchart illustrating a method for providing immersive learning content according to an embodiment of the present invention.

The above-described objects, features and advantages will be described below in detail with reference to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains will be able to easily implement the technical idea of the present invention. In describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to indicate the same or similar components.

The terms used in the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in specific cases, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

When a part "includes" a certain element throughout the specification, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention illustrated below may be modified in various other forms, and the scope of the present invention is not limited to the embodiments described below. The embodiments of the present invention are provided to more completely explain the present invention to those of ordinary skill in the art.

The present invention relates to an apparatus and method for providing immersive learning content based on virtual reality, complex reality, augmented reality, and the like through text recognition.

Hereinafter, an apparatus for providing immersive learning content according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 and 2 .

1 is a diagram schematically illustrating components of an apparatus for providing immersive learning content according to an embodiment of the present invention, and FIG. 2 is a diagram illustrating an embodiment of the apparatus for providing immersive learning content shown in FIG. 1 to be.

Referring to FIG. 1 , an apparatus 100 for providing immersive learning content according to an embodiment of the present invention may include a character recognition module 110 , a control module 120 , and a display module 130 . The immersive learning content providing apparatus 100 illustrated in FIG. 1 is according to an embodiment, and its components are not limited to the embodiment illustrated in FIG. 1 , and some components may be added, changed, or may be deleted.

Each component constituting the immersive learning content providing apparatus 100 is ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gates) arrays), processors, controllers, micro controllers, and microprocessors.

First, the structure of the display module 130 and the immersive learning content providing apparatus 100 will be described.

The display module 130 may be any module capable of displaying an image to a user. For example, the display module 130 may be implemented as a mobile-based device or a wearable device according to a display method (display method).

When the display module 130 is implemented as a mobile-based device, the display module 130 is, for example, an Ultra Mobile PC (UMPC), a workstation, a net-book, a Personal Digital Assistants (PDA), or a portable device. It may be a computer, a web tablet, a wireless phone, a mobile phone, a smart phone, an e-book, a portable multimedia player (PMP), or the like.

When the display module 130 is implemented as a wearable device, the display module 130 includes a head mounted device (HMD), a video see-through device, an optical see-through device, It may be implemented as either a screen-based device or a projector-based device.

Referring to FIG. 2 , the immersive learning content providing apparatus 100 may be implemented as a wearable device designed to be worn on the head. In this case, the display module 130 may be positioned in front of the user's eyes when the user wears the immersive learning content providing apparatus 100 , and may output an image in the direction of the user's eyes.

The immersive learning content providing apparatus 100 and the display module 130 constituting the same may be implemented in various shapes other than the embodiment shown in FIG. 2 , but below, for convenience of description, the immersive learning content providing apparatus 100 ) will be described on the assumption that it is implemented as shown in FIG. 2 .

The character recognition module 110 may recognize the handwritten text handwritten by the user. Here, the handwritten text may be defined as a character handwritten by a user.

The user may write characters on the worksheet according to the learning curriculum, and the character recognition module 110 may recognize the handwritten text written on the worksheet. More specifically, the character recognition module 110 may selectively recognize only the handwritten text handwritten by the user among all the texts described in the study sheet.

To this end, the character recognition module 110 may include a camera 110a that captures the handwritten text. The camera 110a may be provided at an arbitrary position of the immersive learning content providing apparatus 100 to capture handwritten text.

Referring to FIG. 2 as an example, the camera 110a may be provided in front of the immersive learning content providing apparatus 100 in order to photograph the handwritten text located in the user's viewing direction.

The camera 110a may photograph the handwritten text and generate a file corresponding to the photographed image (hereinafter, referred to as a photographed image file). At this time, if the captured image file is a format to which an OCR (Optical Character Recognition) function can be applied (eg, JPG, TIF, BMP, GIF, PNG, PDF format), the character recognition module 110 uses the OCR function to Recognize handwritten text in files.

On the other hand, if the captured image file is a format to which the OCR (Optical Character Recognition) function cannot be applied, the character recognition module 110 converts the captured image file into a file to which the OCR (Optical Character Recognition) function can be applied, Handwritten text can be recognized by applying OCR function within the converted file.

Since the text recognition method through the OCR function follows any method used in the art, a detailed description thereof will be omitted.

Meanwhile, the camera 110a may photograph various texts (eg, pre-printed text on the study sheet) in addition to the handwritten text handwritten by the user. In this case, the character recognition module 110 may selectively recognize only the handwritten text.

More specifically, when an area including handwritten text is photographed by the camera 110a, the character recognition module 110 may extract a handwriting area within the photographed area and recognize the handwritten text from the extracted handwriting area. .

In one example, the camera 110a may photograph a preset area to be photographed. For example, a photographing target area may be preset in the study sheet, and the camera 110a may photograph the photographing target area. In this case, the photographing target area may be divided into a printing area in which text is pre-printed and a writing area in which handwritten text is written according to a position within the area.

The character recognition module 110 may selectively recognize only the handwritten text by performing the OCR recognition operation only within a preset writing area among the photographing target areas.

In another example, the camera 110a may capture an arbitrary area. For example, the camera 110a may generate a photographed image file by photographing an arbitrary area within the study sheet. In this case, the character recognition module 110 may extract the writing area by pattern matching the captured image file and the reference document file stored in the memory.

Here, the reference document file is a data of the reference document, and the reference document may be a document having the same format as the study sheet, but in which the user's handwriting is required in the study sheet is blank.

The character recognition module 110 may pattern-match the file generated by the camera 110a and the reference document file to identify a difference between the two. For example, the character recognition module 110 can identify the text described in each item by pattern matching a reference document in which only items according to the format of the study sheet are described and a photographed image file in which handwritten text is written in the corresponding item, and OCR function can be read and recognized as handwritten text.

When the handwritten text is recognized by the character recognition module 110 , the control module 120 may identify immersive learning content matching the handwritten text.

In the present invention, immersive learning content is content for the user's learning according to the learning curriculum, and a user through virtual reality (VR), mixed reality (MR), augmented reality (AR), etc. It may be any content that gives a sense of immersion to the user.

Here, augmented reality (AR) refers to a computer graphics technique that synthesizes a virtual object or information in a real environment to make it look like an object existing in the original environment. Unlike conventional virtual reality, which targets only virtual space and objects, augmented reality has a feature of providing additional information that is difficult to obtain only in the real world by synthesizing virtual objects on the basis of the real world.

Virtual reality (VR) refers to an interface between a human and a computer that makes a specific environment or situation into a computer and makes it as if the person using it is interacting with the actual surrounding situation/environment.

On the other hand, mixed reality (MR) refers to a technology that creates a space that fuses the two worlds by combining information from the real world and the virtual world. Complex reality (MR) refers to a technology implemented by bringing only the advantages of augmented reality (AR) and virtual reality (VR).

The control module 120 may identify immersive learning content matching the handwritten text with reference to the memory. To this end, realistic learning content corresponding to each handwritten text may be stored in advance in the form of a look-up table (LUT).

For example, a lookup table as shown in Table 1 below may be previously stored in the memory.

[handwritten text] [Realistic learning content] One "dinosaur" “Video About Dinosaurs” 2 "insect" “Video about the life of an insect” 3 "volcano" “Video about the eruption process of a volcano”

When the handwritten text “dinosaur” is recognized by the character recognition module 110 , the control module 120 may identify immersive learning content matching the handwritten text as “an image about dinosaur introduction”.

Similarly, when the handwritten text “insect” or “volcano” is recognized by the character recognition module 110 , the control module 120 displays realistic learning content matching each handwritten text as an “image about the life of an insect”, respectively. ” or “images of the eruption process of a volcano”.

The control module 120 may provide the identified immersive learning content to the display module 130 , and the display module 130 may output the immersive learning content as an image.

As described above, the realistic learning content may be image content generated based on virtual reality (VR), mixed reality (MR), augmented reality (AR), and the like.

Accordingly, in an example, the display module 130 may output realistic learning content as a virtual image based on virtual reality (VR) technology.

More specifically, the display module 130 may convert and output the immersive learning content provided by the control module 120 into a virtual image that is output through the entire screen. Accordingly, a user wearing the immersive learning content providing apparatus 100 shown in FIG. 2 cannot secure a view of the real environment in front of them, but instead displays a virtual image corresponding to the immersive learning content through the entire screen. can be checked

In another example, the display module 130 may output immersive learning content and a live-action image together based on augmented reality (AR) technology.

More specifically, the display module 130 may convert and output the immersive learning content provided by the control module 120 into a virtual image output through a portion of the screen.

In this case, the display module 130 may be made of a transparent material to project an image in front of the user. In other words, the user may secure a view of the real environment in front through the transparent display module 130 .

Accordingly, a user who wears the immersive learning content providing apparatus 100 as shown in FIG. 2 secures a view of the real environment in front, and at the same time, uses a portion of the screen of the display module 130 to provide an immersive experience. A virtual image corresponding to the learning content can be checked.

Meanwhile, the display module 130 may output realistic learning content as an augmented image on the live-action image.

More specifically, the display module 130 may overlap and output an augmented image corresponding to the realistic learning content according to the learning curriculum on the live-action image of the real environment around the user.

The camera 110a included in the character recognition module 110 may be used to generate a live-action image. 2 as an example, the camera 110a may photograph an actual environment in front of the user, and the display module 130 may generate a live-action image corresponding to the photographed actual environment.

Subsequently, the display module 130 may output the live-action image through the screen and overlap and output the augmented image corresponding to the immersive learning content in the live-action image.

Hereinafter, a method for providing immersive learning content according to an embodiment of the present invention will be described with reference to FIG. 3 . The method for providing immersive learning content shown in FIG. 3 may be performed by each component in the apparatus 100 for providing immersive learning content shown in FIG. 1 .

3 is a flowchart illustrating a method for providing immersive learning content according to an embodiment of the present invention.

Referring to FIG. 3 , the method for providing immersive learning content includes the steps of recognizing handwritten text handwritten by a user (S10), identifying immersive learning content matching the handwritten text (S20), and outputting the realistic learning content It may include a step (S30) of doing. The method for providing immersive learning content shown in FIG. 3 is according to an embodiment, and each step constituting the method is not limited to the embodiment shown in FIG. 3, and some steps may be added, changed, or deleted as necessary. can

When each step is described in detail, first, the character recognition module 110 may recognize the handwritten text written by the user on a worksheet or the like ( S10 ).

More specifically, the character recognition module 110 may photograph an area including the handwritten text by using the camera 110a, extract a handwriting area within the captured area, and recognize the handwritten text from the extracted handwriting area. have. Since the operation process of the character recognition module 110 has been described above, a detailed description thereof will be omitted here.

Then, the control module 120 may identify the realistic learning content matching the handwritten text with reference to the memory ( S20 ). To this end, the handwritten text and the immersive learning content may be stored in advance so that they correspond to each other, as described above.

Then, the display module 130 may output the immersive learning content identified by the control module 120 (S30). Since the content of the display module 130 to output the immersive learning content in various ways has been described above, a detailed description thereof will be omitted herein.

As described above, the present invention provides immersive learning content based on virtual reality, complex reality, augmented reality, etc. through recognition of text written by the learner, so that the learner can feel himself through his or her handwritten text. By controlling the type of learning content, it is possible not only to induce learners' participation in education to improve learning efficiency, but also to be able to use various contents for learning and education in an interactive way, thereby improving the leadership of the educator.

As described above, the present invention has been described with reference to the illustrated drawings, but the present invention is not limited by the embodiments and drawings disclosed in the present specification. It is obvious that variations can be made. In addition, although the effects according to the configuration of the present invention are not explicitly described and described while describing the embodiments of the present invention, it is natural that the effects predictable by the configuration should also be recognized.

Claims (13)

a character recognition module for recognizing handwritten text written by a user;
a control module for identifying immersive learning content matching the recognized handwritten text; and
including a display module for outputting the identified immersive learning content
A device for providing realistic learning content.
According to claim 1,
The character recognition module
comprising a camera for photographing the handwritten text
A device for providing realistic learning content.
3. The method of claim 2,
The character recognition module
Converting a file corresponding to the image captured by the camera into a file to which an OCR (Optical Character Recognition) function can be applied, and recognizing handwritten text in the converted file
A device for providing realistic learning content.
3. The method of claim 2,
The character recognition module
photographing an area including the handwritten text, extracting a handwriting area within the photographed area, and recognizing the handwritten text from the extracted handwriting area
A device for providing realistic learning content.
5. The method of claim 4,
The character recognition module
extracting the writing area by pattern matching a file corresponding to the photographed area and a reference document file
A device for providing realistic learning content.
According to claim 1,
The control module is
Identifying realistic learning content matching the handwritten text by referring to a look-up table (LUT) stored in a memory
A device for providing realistic learning content.
According to claim 1,
The display module is
Outputting the immersive learning content as a virtual image
A device for providing realistic learning content.
According to claim 1,
The display module is
Outputting the immersive learning content and the live-action video together
A device for providing realistic learning content.
9. The method of claim 8,
The display module is
Outputting the immersive learning content as an augmented image to the live-action image
A device for providing realistic learning content.
According to claim 1,
The display module is
A wearable device
A device for providing realistic learning content.
11. The method of claim 10,
The display module is
HMD (Head Mounted Device), video see-through device (video see-through device), optical see-through device (optical see-through device), including at least one of the projector (projector)
A device for providing realistic learning content.
recognizing the handwritten text handwritten by the user;
identifying immersive learning content matching the recognized handwritten text with reference to a memory; and
outputting the identified immersive learning content
How to provide realistic learning content.
13. The method of claim 12,
Recognizing the handwritten text handwritten by the user includes:
photographing an area including the handwritten text;
extracting a writing area within the photographed area; and
Recognizing the handwritten text in the extracted handwriting area
How to provide realistic learning content.

Images