KR101916146B1 - Method and system for providing book reading experience service based on augmented reality and virtual reality - Google Patents

Abstract

According to the present invention, a reading-experience service providing method for a reading-experience service providing system including a head mounted display (HMD) includes: a step of providing an augmented reality (AR) service including a first virtual character describing a recognized character as a three-dimensional virtual image when the character is recognized on a two-dimensional plane through the HMD; a step of making a switch to a virtual reality (VR) service having a scenario based on children′s story associated with the character, while providing the AR service; a step of generating a second virtual character, which is a virtual character of a user wearing the HMD, when the switch to the VR service has been completed; a step of providing learning content as an interactive game, while providing the VR service in accordance with the scenario using the first and second characters as main components; and a step of providing a reading state report including information diagnosing and evaluating a reading ability of the user when the scenario has ended. According to the present invention, the method is capable of inducing children to read books.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and system for providing a reading experience service based on AR and VR,

The present invention relates to a reading experience service technique, and more particularly, to a technique of providing a reading experience service based on Augmented Reality (AR) and Virtual Reality (VR) to a user wearing an HMD.

AR (Augmented Reality) is one of the virtual reality in which the user views the real world viewed by the user and the virtual world having the additional information as one image. AR, a concept that complements the real world with a virtual world, uses a virtual environment created by computer graphics, but the protagonist is a real environment. The computer graphics serve to provide information necessary for the real environment, and overlap the three-dimensional virtual image with the real image that the user is viewing, thereby making the distinction between the real environment and the virtual screen ambiguous.

Virtual reality (AR) is a technology that overlaps 3D virtual images on a realistic image or background and displays them as a single image, while virtual objects use virtual images instead of reality.

Augmented Reality (MR), also known as Mixed Reality (MR), was introduced to the world for the first time as augmented reality was added to the world by adding a virtual image to the airplane assembly process in 1990 at Boeing.

Augmented Reality and Virtual Reality (VR) seem to be similar, but they are clearly distinguished according to whether they are virtual or real. For example, in a virtual reality fighting game, a character instead of me is confronted with a virtual enemy in a virtual space, but an augmented reality fighting game is a form in which I confront a virtual enemy in a real space. Therefore, the augmented reality is more realistic than the virtual reality.

In addition, virtual reality is generally used only in special environments such as film and video, but augmented reality is widely popularized by the general public.

For example, map retrieval via the Internet, location retrieval, etc. are also included in augmented reality in a broad sense.

Generally, infants and children grow up by watching puzzles or playing puzzles or building blocks. As such, children's fairy tale or playgrounds are an important medium for naturally acquiring various information and knowledge. Fairy tale puzzles, puzzles, or blocks are used to help children develop their emotions, imagination and creativity, Cognitive development and so on.

According to the National Survey on National Reading in 2015, communication through books (reading books, playing games, reading conversation, etc.) during the growth process plays an important role in developing intimacy and reading habits.

As a result of recent research, it has been found that children with high reading abilities have high thinking skills and creativity as well as academic performance. In addition, with the development of digital media, there is a need to respond to a rapidly changing reading environment and a decrease in the reading population.

The government has established a basic plan for promoting reading culture, and has been promoting projects for promoting reading and spreading the reading culture. However, it is not enough for the public to feel it, and it is required to continuously expand reading contents.

As such, as the infant market has emerged as a niche (angel) market in the niche market, the content consumption market for learning mobile game edutainment targeting mobile kids is growing.

As virtual reality (VR) emerges as a blue ocean that will lead the future industry, application cases are increasing in various fields such as games, medical education, education and exhibition, but it is necessary to find killer contents.

Korean Patent Publication No. 10-2009-0003445

SUMMARY OF THE INVENTION The present invention provides a real-time edutainment assimilation content and a smart learning solution capable of interacting with a virtual reality (VR) and an augmented reality (AR) It has its purpose.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, in a method of providing a reading experience service in a reading experience service providing system including an HMD (Head Mounted Display) of the present invention, when a character existing on a 2D plane is recognized through the HMD, Providing an AR (Augmented Reality) service including a first virtual character representing a character in a 3D virtual image, converting the virtual reality service into a scenario based on a moving image content related to the character during the provision of the AR service, Generating a second virtual character, which is a virtual character of a user wearing the HMD, when the VR service is switched to the VR service; and providing a VR service based on the first character and the second character in accordance with the scenario , Providing learning contents in the form of a game capable of interacting with each other, and when the scenario ends, And providing a reading status report including information assessed and evaluated by the user's reading ability.

In the step of providing the game-type learning contents in which the interaction can be performed, the user can recognize the hand motion of the user and reflect the interaction to the user, or recognize the user's voice and reflect the user's voice in the interaction.

In the step of providing the Augmented Reality (AR) service, a character colored in a specific color can be recognized and displayed as a first virtual character.

In the step of providing the reading status report, the reading status report may be transmitted to the predetermined terminal.

The present invention provides a reading experience service providing system comprising a hand movement recognition sensor for recognizing a hand motion of a user, a body movement recognition sensor for recognizing motion of the whole body of the user, a head mounted display (HMD) And a first virtual character that recognizes a character existing on the 2D plane through the 3D virtual image and expresses the recognized character as a 3D virtual image, and provides the AR service based on the animation content related to the character during the provision of the AR service A VR service of the scenario, and a control module for generating a second virtual character, which is a virtual character of a user wearing the HMD.

Wherein the control module controls the hand movement of the user recognized by the hand movement recognition sensor while the VR service is being provided based on the first character and the second character according to the scenario, And provides a reading status report including information on the evaluation of the user's reading ability when the scenario ends, and information including the evaluated information.

The HMD includes a communication interface for communicating with an external device, a voice recognition unit for recognizing voice, a voice output unit for outputting voice, a display unit for displaying an AR or VR image, and an overall operation of the HMD And a control unit for processing a signal received through the communication interface unit and reflecting the voice of the user recognized by the voice recognition unit during the provision of the VR service to the interaction to proceed with the game.

The control module may recognize a character colored in a specific color and display the character on the HMD as a first virtual character.

The control module may transmit the reading status report to the predetermined terminal.

According to the present invention, by providing a reading experience service based on AR and VR, children can develop reading habits, develop self-directed learning ability, develop communication and empathy ability, There is an effect that can help strengthen.

In addition, according to the present invention, in the VR market environment where there is no killer content, there is an advantage that the VR industry can contribute to the revitalization of the VR industry through the development of new concept creative accomplishments in which advanced IT technology and education of the VR and AR are fused.

Further, according to the present invention, it is possible to contribute to the promotion of the reading culture and the growth of the publishing industry through reading living and habituation, and it is expected that new industries such as reading culture industry, contents industry and publishing industry can be created.

1 and 2 are views schematically showing a concept of a reading experience service providing system according to an embodiment of the present invention.
3 is a flowchart illustrating a recognition process in two sensors according to an exemplary embodiment of the present invention.
4 is a block diagram illustrating an internal structure of an HMD according to an embodiment of the present invention.
5 is a flowchart illustrating a method of providing a reading experience service according to an embodiment of the present invention.
FIG. 6 is a diagram briefly showing a process of providing a reading experience service according to an embodiment of the present invention.
7 to 9 are illustrations of a reading experience scenario according to an embodiment of the present invention.
10 is a diagram illustrating a reading status report according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted in an ideal or overly formal sense unless expressly defined in the present application Do not.

In the following description of the present invention with reference to the accompanying drawings, the same components are denoted by the same reference numerals regardless of the reference numerals, and redundant explanations thereof will be omitted. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

1 and 2 are views schematically showing a concept of a reading experience service providing system according to an embodiment of the present invention.

1 and 2, a reading experience service providing system according to an embodiment of the present invention includes an HMD (Head Mounted Display) 100, a hand movement recognition sensor 200, a body motion recognition sensor 300, a control module (Not shown).

The HMD 100 is a device worn on the user's head, and displays Augmented Reality (AR) and Virtual Reality (VR).

The hand movement recognition sensor 200 recognizes the hand movement of the user. For example, the hand movement recognition sensor 200 may be implemented as a lip motion. In an embodiment of the present invention, the hand movement recognition sensor 200 may be mounted together on the HMD.

The body motion recognition sensor 300 recognizes the motion of the whole body of the user. For example, the body motion recognition sensor 300 may be implemented as a Kinect.

The control module (not shown) provides an AR service including a first virtual character that recognizes a character existing on the 2D plane through the HMD 100 and expresses the recognized character as a 3D virtual image. Then, during the provision of the AR service, the system switches to the VR service of the scenario based on the animation contents related to the character, and generates the second virtual character which is the virtual character of the user wearing the HMD.

In an embodiment of the present invention, the control module may recognize a character colored in a specific color and display it on the HMD 100 as a first virtual character. For example, the control module may recognize a character painted in green and display it on the HMD 100 as a first virtual character.

Then, while the VR service is being provided according to the scenario based on the first character and the second character, the control module determines whether or not the user's hand movement recognized from the hand movement recognition sensor 200 and the hand movement of the user recognized from the body movement recognition sensor 300 Provides learning contents in a game form capable of interacting with the body by reflecting the body motion, and provides a reading status report including information obtained by diagnosing and evaluating the user's reading ability when the scenario ends.

In an embodiment of the present invention, the control module may transmit the reading status report to the predetermined terminal. For example, a reading status report can be sent to a mobile phone used by a child's caregiver using a reading experience service.

In the present invention, the control module may be implemented as a separate server or may be embedded in the HMD 100. That is, the control module is not limited to the implementation form or the installation place.

In the present invention, a technique for interlocking the hand movement recognition module 200 and the body motion recognition module 300 is proposed in order to improve accuracy and diversity of motion recognition.

2, a recognition range (a) in the case where the lip motion is used and a recognition range (b) in the case where the Kinect is used as an example of the body movement recognition module 300 are shown as an example of the hand movement recognition module 200 have.

3 is a flowchart illustrating a recognition process in two sensors according to an exemplary embodiment of the present invention.

Referring to FIG. 3 (a), the motion recognition process of the present invention can roughly be divided into a motion recognition step S310 and a synchronization step S320.

The motion recognition step S310 recognizes the effective distance from the hand movement recognition sensor 200 at S311, performs the user motion recognition S312, recognizes the effective distance at the body motion recognition sensor 300 at S313, And performs motion recognition (S314). For reference, the effective recognition range for lip motion is 150 ° within the viewing angle, within 600 mm for the vertical (z-axis), and 0.4 to 0.8m for Kinect v2.

If simultaneous recognition is impossible in the synchronization step S320, a valid recognition value is processed in a priority order (S321). In other words, when the user's motion recognition by the gesture recognition sensor 300 is impossible, the user's motion recognition by the gesture recognition sensor 200 is prioritized. On the other hand, if the user's motion recognition by the gesture recognition sensor 300 is possible but the user's motion recognition by the gesture recognition sensor 200 is impossible, user motion recognition by the gesture recognition sensor 300 is prioritized. In case of the same recognition, synchronization processing is performed (S322). In other words, when the user motion recognition by the hand movement recognition sensor 200 and the user motion recognition by the body movement recognition sensor 300 are valid and the operation is recognized in the same operation category, Thereby synchronizing the motion recognition and the user's motion recognition by the body motion recognition sensor 300. If the user's motion recognition by the hand movement recognition sensor 200 is different from the user's motion recognition by the body motion recognition sensor 300, the recognized motion is segmented and processed into two motions (S323). According to the present invention, it is possible to perform individual recognition processing by the hand movement recognition sensor 200 and the body movement recognition sensor 300, thereby recognizing and handling other motions of the user in addition to the hand movement recognition of the hand movement recognition sensor 200 .

At this time, the user motion by the hand movement recognition sensor 200 and the user motion by the body motion recognition sensor 300 may be learned through the machine learning algorithm. For example, when a feature of a recognizable range is extracted for each motion having a low probability of accurate motion recognition by the hand movement recognition sensor 200 or the body movement recognition sensor 300, It is possible to give a weight by learning and learn such motion to be recognized as a specific motion. Through this process, it becomes possible to recognize and distinguish the motion that is difficult to recognize accurately.

Fig. 3 (b) shows a gesture machine learning algorithm by a hand gesture recognition sensor. Referring to FIG. 3B, when a specific hand gesture (gesture a) is input to the hand movement recognition sensor 200 in off-line (S341), the recognition probability of the corresponding hand gesture is smaller than a predetermined recognition probability value A If it is judged (S343), the feature of the hand gesture is extracted (S345). If the recognition probability of the corresponding hand gesture is determined to be equal to or greater than the predetermined recognition probability value A (S343), an interaction process corresponding to the corresponding hand gesture is performed (S347). The characteristic extracted at this time is a characteristic that the hand movement recognition sensor 200 can recognize through the corresponding hand movement gesture. The learning is performed in a manner of weighting the extracted features (S349). The feature extracted through the repeated learning is learned as corresponding to the corresponding gesture (a). If the gesture (a) is input to the gesture recognition sensor 200 after the off-line learning (S342), the gesture recognition sensor 200 only recognizes the extracted feature, And processes the interaction corresponding to the gesture (a) (S344).

3 (c) shows a gesture machine learning algorithm by a gesture recognition sensor. 3C, when a specific gesture bubble (gesture b) is input to the gesture recognition sensor 300 in off-line (S351), the recognition probability of the gesture gesture is smaller than a predetermined recognition probability value B If it is judged (S353), the feature of the body movement gesture is extracted (S355). If it is determined that the recognition probability of the gesture gesture is equal to or greater than the predetermined recognition probability value B (S353), the interaction process corresponding to the gesture gesture is performed (S357). The feature extracted at this time is a characteristic that the body motion recognition sensor 300 can recognize through the body movement gesture. The learning is performed in such a manner that a weight is assigned to the extracted features (S359). The feature extracted through the repeated learning is learned as corresponding to the corresponding gesture (b). When the body movement gesture b is input to the body movement recognition sensor 300 after the offline training, the body movement recognition sensor 300 recognizes only the extracted features, but when the learned gesture b is input, And processes the interaction corresponding to the gesture (b) (S354).

The gesture (a) recognized by the hand movement recognition sensor 200 and the gesture (b) recognized by the body movement recognition sensor 300 are combined and recognized as a gesture (c) May be processed.

This motion recognition is performed, and the interaction is finally processed (S330).

4 is a block diagram illustrating an internal structure of an HMD according to an embodiment of the present invention.

Referring to FIG. 4, the HMD 100 of the present invention includes a communication interface unit 110, a voice recognition unit 120, a voice output unit 130, a display unit 140, and a controller 150.

The communication interface unit 110 communicates with an external device.

The voice recognition unit 120 recognizes the voice.

The voice output unit 130 serves to output voice.

The display unit 140 serves to display an AR or VR image.

The control unit 150 controls the overall operation of the HMD 100.

The control unit 150 processes the signal received through the communication interface unit 110 and reflects the voice of the user recognized by the voice recognition unit 120 in the interaction during the provision of the VR service to proceed the game.

5 is a flowchart illustrating a method of providing a reading experience service according to an embodiment of the present invention.

Referring to FIG. 5, an AR service including a first virtual character that recognizes a character existing on a 2D plane through the HMD 100 (S501) and representing the recognized character as a 3D virtual image is provided (S503) .

Then, during the provision of the AR service, the scenario is switched to the VR service based on the moving picture contents related to the character (S505).

When the VR service is switched to the VR service, a second virtual character, which is a virtual character of the user wearing the HMD, is generated. While providing the VR service according to the scenario based on the first character and the second character, And provides learning contents in game form (S507).

Then, when the scenario ends (S509), a reading status report including information on the evaluation of the user's reading ability is provided (S511).

In one embodiment of the present invention, in step S507 of providing a game-type learning content capable of interacting, the user's hand motion is detected and reflected in the interaction, or the user's voice is recognized and reflected in the interaction can do.

In step S503 of providing the Augmented Reality (AR) service, a character colored in a specific color can be recognized and displayed as a first virtual character. For example, a character colored in green may be recognized and displayed as a first virtual character.

In the step S511 of providing the reading status report, the reading status report can be transmitted to the predetermined terminal. For example, a reading status report can be transmitted to a terminal (for example, a mobile phone) used by a child's guardian using a reading experience service.

FIG. 6 is a diagram briefly showing a process of providing a reading experience service according to an embodiment of the present invention.

FIG. 6 illustrates realistic edutainment assimilation contents and smart learning solutions capable of interacting experience based on virtual reality (VR) and augmented reality (AR) of the present invention.

In the embodiment of Fig. 6, the user paints a Peter Pan character, which is a moving picture character drawn on a drawing sheet (1). Through this coloring process, users can develop creativity.

If the painted Peter Pan character is recognized while the user wears the HMD, the Peter Pan character painted by the user through the AR is greeted by the user and greeted to the user (2).

Then, as the screen is switched to the virtual reality (VR), the user travels with the Peter Pan to the real world in the virtual reality (③).

Then, the user becomes a fairy tale country character in virtual reality, and performs various experiential game type learning contents along with Peter Pan (④). In this process, it is possible to apply a general language learning method and chunk / task / situation-based learning methods, or to introduce a system for ranking items by scoring items by acquiring items or solving problems during a game.

Then, after clearing the game and ending the scenario, I will diagnose creativity, imagination, and intimacy of reading through storytelling experiences (⑤).

Then, a reading status report is provided (⑥). Users can be motivated to achieve achievement by watching the reading status report, and can help spontaneous book reading habits through memories they have experienced while playing with characters in a fairy tale country in virtual reality. You can also provide your parents with a reading status report to check your progress.

7 to 9 are illustrations of a reading experience scenario according to an embodiment of the present invention.

7 to 9 illustrate screens displayed when a user wears an HMD.

In FIG. 7, a character 1 advancing a story appears. This screen is an augmented reality (AR) screen where the background music can be played back and the character 1 can discriminate between a large and a small voice of a user through voice input through a microphone, have.

After a certain period of time, the screen is switched to the screen shown in Fig.

In FIG. 8, the character 1 performs a circle drawing operation with a finger at the central portion of the screen, and the circle 810 drawn by the character 1 becomes larger. At this time, when the user looks at the circle 810, he can recognize it and generate a special sound effect. Then, the circle 810 gradually grows to the next screen. The screen in the circle is a virtual reality (VR) screen, and the virtual reality screen is displayed naturally on the next screen. The conversion from the augmented reality (AR) screen to the virtual reality (VR) screen may be performed by recognizing a predetermined motion of the user by the gesture recognition sensor 200 or the gesture recognition sensor 300. [

The switching between the real space in which the augmented reality AR is implemented and the virtual space in which the virtual reality VR is implemented can be realized by using the camera, the hand movement recognition sensor 200, and the body movement recognition sensor 300 of the HMD 100, And generating spatial mesh information by recognizing the depth information of the object, the floor, the wall, the ceiling, and the object, and performing image calibration using RGB information, depth information, and mesh information. In other words, the switching between real space and virtual space can be realized through spatial and object recognition, 3D model extraction (mesh generation), combination of RGB information and 3D model, and image calibration.

In FIG. 9, the virtual reality screen is switched to a full-fledged virtual reality story along with various background music and sound effects.

10 is a diagram illustrating a reading status report according to an embodiment of the present invention.

Referring to FIG. 10, the present invention provides a reading status report including a result of diagnosing a user's reading ability and a graph and evaluating a user's reading ability using a predetermined reading ability evaluation element.

As described above, the present invention provides a reading learning method and provides various languages and communication learning methods through learning contents. In other words, in the present invention, while interacting with a non-player character (NPC), it is guided and provides a feeling of being in a fairy tale while playing, and experiences a variety of fairy tale situations directly as a main character , And acquire language and communication skills naturally through playful learning.

In the present invention, chunk-based learning, process-based learning, and situation-based learning are provided as a total language learning method that enables students to acquire language learning and communication ability naturally while experiencing contents.

First, in chunk-based learning, a chunk is a chunk of words with a single meaning, and is learned through a chunk card represented by a picture in the context of chunks. And because you learn a meaningful chunk of text without learning each word and grammar, you will use more accurate and natural expressions, and you will understand the language more easily and quickly. This approach has the advantage of reducing the burden of learning words and grammar separately.

In addition, the present invention provides task-based learning using gamification to enhance the educational effect of games and a fairy tale at the same time.

In the present invention, various tasks such as challenge, competition, score acquisition, compensation, level-up, and mission are performed through the game, and a new expression is naturally learned through the process of solving.

In addition, the present invention provides context-based learning, so that a learner communicates with various characters and various places and situations through various VR experience assimilation. Through this process, various sentences are spoken and naturally accustomed to language expression, so that they can communicate confidently even if they encounter various situations and opponents in everyday life.

While the present invention has been described with reference to several preferred embodiments, these embodiments are illustrative and not restrictive. It will be understood by those skilled in the art that various changes and modifications may be made therein without departing from the spirit of the invention and the scope of the appended claims.

100 HMD
200 Handset recognition sensor
300 Gesture recognition sensor

Claims (8)

In a reading experience service providing method in a reading experience service providing system including an HMD (Head Mounted Display)
Providing an Augmented Reality (AR) service including a first virtual character representing a recognized character on a 2D plane through the HMD as a 3D virtual image;
Switching to a VR (Virtual Reality) service of a scenario based on a moving picture content related to the character while providing the AR service;
Generating a second virtual character that is a virtual character of a user wearing the HMD when the VR service is switched to the VR service;
Providing learning contents in a game form capable of interacting while providing a VR service based on the first character and the second character according to the scenario; And
And providing a reading status report including information obtained by diagnosing a user's reading ability when the scenario ends.
The method according to claim 1,
Wherein the step of providing the learning contents providing interaction with the game includes sensing the user's hand movements and reflecting the same in the interaction or recognizing the user's voice and reflecting the user's interaction in the interaction Way.
The method according to claim 1,
In the step of providing the Augmented Reality (AR) service,
Recognizing a character colored in a specific color and displaying the character as a first virtual character.
The method according to claim 1,
In providing the reading status report,
Wherein the reading status report is transmitted to a predetermined terminal.
A hand movement recognition sensor for recognizing a hand motion of a user;
A body motion recognition sensor for recognizing the motion of the entire body of the user;
An HMD (Head Mounted Display) worn on the user's head; And
Providing an AR service including a first virtual character that recognizes a character existing on a 2D plane through the HMD and expresses the recognized character as a 3D virtual image; And a control module for generating a second virtual character, which is a virtual character of a user wearing the HMD,
Wherein the control module controls the hand movement of the user recognized by the hand movement recognition sensor while the VR service is being provided based on the first character and the second character according to the scenario, And providing a reading status report including information obtained by evaluating a user's reading ability when the scenario is terminated and evaluating the reading performance report.
The method of claim 5,
In the HMD,
A communication interface unit for communicating with an external device;
A speech recognition unit for recognizing speech;
An audio output unit for outputting audio;
A display unit for displaying an AR or VR image; And
A control unit for controlling the overall operation of the HMD, processing a signal received through the communication interface unit, and reflecting the voice of the user recognized from the voice recognition unit during the provision of the VR service to the interaction to proceed with the game Wherein the service providing system comprises:
The method of claim 5,
Wherein the control module recognizes a character colored in a specific color and displays the recognized character in the HMD as a first virtual character.
The method of claim 5,
Wherein the control module transmits the reading status report to a predetermined terminal.

Images