KR20190128962A - METHOD AND WEARABLE DISPLAY APPARATUS FOR PROVIDING eBOOK BASED ON AUGMENTED REALLITY - Google Patents

Abstract

Provided are a method and wearable display device for providing an electronic book based on augmented reality. The method for providing an electronic book based on augmented reality by the wearable display device operated by at least one processor comprises the steps of: tracking gaze of a user wearing the wearable display device; obtaining a real image positioned in a gaze direction of the user; augmenting a virtual image that the electronic book is displayed on three-dimensional coordinates of the real image; recognizing a gesture opening the electronic book with a hand of the user; and augmenting the virtual image on which a plurality of book pages of the electronic book is displayed according to the gesture recognition.

Description

Augmented Reality-based e-book providing method and wearable display device {METHOD AND WEARABLE DISPLAY APPARATUS FOR PROVIDING eBOOK BASED ON AUGMENTED REALLITY}

The present invention relates to a method for providing an augmented reality based e-book and a wearable display device.

Books have a great meaning as the oldest media that human beings store, search and acquire information for thousands of years. These book media have evolved into eBooks with the development of information and communication technology, providing more powerful functions of information storage, retrieval and acquisition, and are now replacing paper books in many places. However, with the development of information and communication technology, the possibility of further improved information media is opening up and the situation needs to be explored.

Because there are many aspects of reading behavior, there have been various researches aimed at enhancing the reading experience by utilizing information and communications technologies (ICT). In addition to e-book terminals, new types of information media used were new conductors, tablet PCs, and notebook computers.

Recently, various attempts have been made to enhance the reading experience using AR (Augmented Reality) technology, but there are limitations in the genre where immersion and interest are important, such as narrative writing, coloring games, and textbooks.

In addition, even when the usability was improved, it was difficult to expand to the case where it was difficult to use both hands such as medical, military training, and manufacturing industry because touch interaction was required.

On the other hand, when reading, users make various attempts to distinguish the work area from the viewing area in order to avoid repetitive tap switching. For example, half-screen, using a smartphone or tablet PC.

In addition, users read the book and use paragraph highlighting when the paragraphs related to the topic appear, or copy the contents and mark them as if they are recorded elsewhere. However, you may have difficulty remembering where you have a page or paragraph to revisit, or make a supplementary record of page numbers or phrases elsewhere.

In addition, users try various strategies to remember the author's name or title. For example, write down the author's name or title, or go to the first chapter of the book and double-check the author or library information.

As described above, various inconveniences exist to the user when reading, but a method for effectively solving the problem has not been proposed.

The present invention has been made in an effort to provide an augmented reality-based e-book providing method and a wearable display device that assists a user's reading activity according to a user input read through gesture recognition.

According to an embodiment of the present invention, a method of providing an e-book provides a method of providing an augmented reality-based e-book by a wearable display device operated by at least one processor, wherein the user wears the wearable display device. Tracking an image, acquiring a photorealistic image located in the user's gaze direction, augmenting a virtual image in which an e-book is displayed on a three-dimensional coordinate system of the photorealistic image, and recognizing a gesture of opening a book by a user's hand And augmenting a virtual image on which the plurality of book pages of the e-book is displayed on the real world 3D coordinate system according to gesture recognition.

The outputting may include generating real-real image data by photographing a real world indicated by the user's gaze direction, determining a user device among objects identified from the real-real image data, and And outputting a grid interface around the grid interface, wherein the grid interface is a hologram image obtained by dividing a virtual space overlaid on the real world three-dimensional coordinate system into a plurality of sections, and the arranging includes the plurality of books. Pages may be displayed in the page order in the plurality of sections, respectively.

After outputting the grid interface, the method may further include recognizing a gesture of clicking the grid interface, and adjusting and outputting an angle and a size of the grid interface according to the gesture recognition.

The listing may include displaying a tag on each of the plurality of book pages, and after the listing, recognizing a gesture of folding and then clicking the tag, and matching and storing the tag with the corresponding page. In addition, the tag may be used for page grouping.

After the listing, the method may further include recognizing a user voice, and outputting the book data according to the voice recognition on the grid interface.

According to another feature of the present invention, the wearable display device includes an image acquisition unit for capturing an image, a gaze interface for detecting a movement of a user's eyes, a gesture interface for recognizing a gesture indicated by a movement of a user's hand, and an image in a real world 3D coordinate system An augmented reality screen output unit for augmenting a virtual image displayed on the e-book, a storage unit for storing a program for providing an augmented reality-based e-book, and at least one processor for executing the program, the program, When the gesture of opening the book with the user's hand is recognized, the virtual image displaying the e-book is projected on the 3D coordinate system of the real world image which is located in the direction of the user's gaze detected by the gaze interface and acquired from the image acquisition. Recalling a plurality of book pages of the e-book according to gesture recognition It includes the commands that are listed in the real world three-dimensional coordinate system (Instructions).

The program identifies the user device from the actual reality image data indicated by the user's gaze direction generated from the camera, projects a grid interface around the user device, and the plurality of books in a plurality of sections of the grid interface. And a command to control the augmented reality screen output unit to display pages in page order, respectively, and the grid interface may be a hologram image obtained by dividing a virtual space overlaid on a real world three-dimensional coordinate system into a plurality of sections.

If the program recognizes a gesture of displaying each tag on the plurality of book pages and folding and clicking one of the respective tags, the program recognizes a gesture and a corresponding tag and stores the page. The storage unit may store grouping information in which at least one page corresponding to each tag in which a gesture is recognized is matched.

The wearable display apparatus may further include a microphone configured to receive a user voice, and the processor may recognize the user voice to derive a command, and overlay the book data according to the command on the grid interface and output the command. It may include commands to control the reality screen output unit.

The processor may include instructions for outputting a user interface for adjusting an angle and a size of the grid interface when the gesture of clicking the grid interface is recognized and for changing the grid interface according to the gesture recognized through the user interface. can do.

According to the embodiment of the present invention, the electronic book can be read in a state where both hands can be freely written, thereby efficiently supporting a situation in which a high degree of expertise is required.

In addition, users can refer to multiple texts simultaneously and read user input using gesture recognition rather than touch method, which is effective for multitasking and can be used for specific tasks such as medical, military training, and manufacturing, as well as general text reading. It can enhance the user experience in situations where you are reading multiple texts.

In addition, it is possible to solve the conventional problems of manual reading, such as fatigue of repeated tap switching in general reading, inconvenience of reloading the displayed paragraph, and manual writing such as difficult to find author information and study information. .

1 is an exemplary view for providing an e-book based on augmented reality according to an embodiment of the present invention.
2 is a block diagram illustrating a hardware configuration of a wearable display device according to an exemplary embodiment.
3, 4 and 5 are exemplary views of a grid interface according to an embodiment of the present invention.
6 is a flowchart illustrating a method of providing an e-book according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless otherwise stated.

The expressions used in the singular herein may be interpreted in the singular or the plural, unless an explicit expression such as “one” or “single” is used.

In this specification, user interaction is defined as a term referring to a user input operation that causes a certain result.

1 is an exemplary view for providing an e-book based on augmented reality according to an embodiment of the present invention.

Referring to FIG. 1, a grid interface 200 in which an e-book is displayed around the device 100 used by a user is augmented. The wearable display device 300 worn by a user acquires an actual image to which the user's eyes are directed, and outputs the grid interface 200 on the 3D coordinate system of the actual image. It is designed to make it easier for users to write or read other materials while doing integrated reading.

The book pages are arranged in the grid interface 200 in order from the inside to the outside. The user may click on the grid interface 200 to adjust the angle and size of the grid interface 200 according to his device 100.

Here, the wearable display apparatus 300 may perform page layout, page classification, or obtain information on the e-book by voice through gesture recognition. The wearable display device 300 is a display device that can be mounted on a head and present an image directly in front of a user's eyes. The wearable display device 300 may also be referred to as a head mounted display (HMD). A configuration of such a wearable display device will be described with reference to FIG. 2.

2 is a block diagram illustrating a hardware configuration of a wearable display device according to an embodiment of the present invention, and FIGS. 3, 4, and 5 are exemplary views of a grid interface according to an embodiment of the present invention.

Referring to FIG. 2, the wearable display device 300 includes an augmented reality screen output unit 301, an image acquisition unit 303, a speaker 305, a gaze interface 307, a gesture interface 309, The voice interface 311, the communication interface 313, the storage unit 315, and the controller 317 are included.

As described with reference to FIG. 1, the AR screen output unit 301 augments the grid interface (200 of FIG. 1) on which the e-book is displayed on the live image. The AR screen output unit 301 may be various embodiments according to the display driving principle of the wearable display device 300.

According to an embodiment, the AR screen output unit 301 may be configured as a hologram lens using a holographic principle of displaying a stereoscopic image in the air by using interference of light. In this case, the AR screen output unit 301 projects a hologram image in which an e-book is displayed on the real world three-dimensional coordinate system.

According to another exemplary embodiment, the AR screen output unit 301 may be implemented as an optical display system of an optical display type by displaying light by projecting light onto a screen. The AR screen output unit 301 reflects the beam of the projector to the plate in the prism to form an image in the eyes of the user. In this case, the AR screen output unit 301 may output the e-book optical image through the projector.

However, since the wearable display device 300 is manufactured by various manufacturers and its types and technical principles vary, the wearable display device 300 is not limited to a specific technology including these two embodiments.

The image acquisition unit 303 photographs a real environment including at least one real object and acquires an image of the real environment, that is, a real image from the real environment.

According to an embodiment, the image acquisition unit 303 may be a color (RGB), stereo, depth, depth, or a combination of color and depth (RGBD). The image acquisition unit 303 may be a video camera capable of shooting a video.

In addition, the image acquisition unit 303 may obtain a user's hand image for gesture recognition based on user interaction.

The speaker 305 outputs a voice, an effect sound, and the like according to the operation of the controller 317.

The gaze interface 307 detects movement of the user's eyes. The gaze interface 307 may be a gaze tracking sensor that tracks a gaze direction of a user.

According to an embodiment of the present invention, the gaze interface 307 may be an optical gaze tracking sensor. In this case, the gaze interface 307 selectively scans the user with a light ray (for example, infrared ray) that is difficult to visually recognize and is harmless to the human body, and recognizes the ray reflected from the user to track the user's gaze.

According to another embodiment of the present invention, the gaze interface 307 may be a gaze tracking sensor of an image input method. In this case, the line of sight interface 307 receives image data of the user through a camera including a charge coupled device (CCD) element, and reads the image data to track the line of sight of the user.

However, the above-described method of tracking the eyes of the user is not limited to the above-described method, and may be variously modified and implemented by those skilled in the art, and the present invention includes all the above-described embodiments.

The gesture interface 309 detects a user's movement. According to an embodiment, the gesture interface 309 may include an infrared sensor or an ultrasonic sensor. For example, the gesture interface 309 can be gestured through an Infrared Ray Illuminator that emits multiple infrared points, an Infrared Ray Sensor that recognizes reflected infrared light, an auxiliary Infrared Sensor for large viewing angles, and the like. Can be recognized. The gesture may be detected through the light amount information recognized by the infrared sensor, and the light amount information for each gesture may be stored in advance, but is not limited thereto. In addition, since a gesture recognition technique using an infrared sensor or an ultrasonic sensor may use a technique known in the art, a detailed description thereof will be omitted.

According to another exemplary embodiment, the gesture interface 309 may extract a hand region from an image input from the image acquirer 303, and detect a gesture through image recognition of the extracted hand region. Such image recognition-based gesture recognition technology may use various techniques known in the art.

The voice interface 311 includes a configuration for receiving a voice from the outside and a configuration for recognizing the external voice. In the voice input configuration, a microphone array may be used. Since the speech recognition technology is well known, a detailed description thereof will be omitted.

The communication interface 313 is connected to an external server (not shown) through a wired or wireless communication network (not shown), and transmits and receives data related to an e-book service. For example, the e-book information is stored in an external server (not shown) and can be received. In addition, a gesture recognition result of the user, a gaze tracking result, and the like may be transmitted to an external server (not shown), and the processing result may be received.

Here, the external server (not shown) may perform a function of providing e-book information in response to a request of the wearable display device 300. For example, the external server may include a search engine (not shown) to retrieve e-book information corresponding to a request of the wearable display device 300 and browse the search results by a user of the wearable display device 300. Can be provided to do so.

The storage unit 315 includes a nonvolatile memory and functionally stores a program code to be loaded into the controller 317 and a data set used by the program code.

The storage unit 315 stores a program for providing an augmented reality-based electronic book. Such a program may include instructions for interpreting a user interaction through a gaze tracking result and a gesture recognition result, and implementing e-book information according to the interpretation in augmented reality.

The controller 317 controls each component to perform a method for providing augmented reality-based e-book according to the present invention, which may be a hardware processor or a software module executed in the hardware processor.

The controller 317 is the AR screen output unit 301, the image acquisition unit 303, the speaker 305, the gaze interface 307, the gesture interface 309, the voice interface 311, the communication interface 313 and storage In conjunction with section 315, it controls the data flow of these configurations.

The controller 317 tracks the gaze of the user wearing the wearable display device 300 through the gaze interface 307. The controller 317 obtains a live image located in the direction of the user's eyes from the image acquirer 303. The controller 317 augments the virtual image in which the e-book is displayed on the three-dimensional coordinate system of the live-action image. In this case, the controller 317 may project the hologram image on which the e-book is displayed. When the gesture of opening a book by the user's hand is recognized from the gesture interface 309, the controller 317 may augment an AR screen to augment a virtual image on which a plurality of book pages of an e-book is displayed on the real world 3D coordinate system according to the gesture recognition. The output unit 301 is controlled.

The controller 317 may identify the user device (100 of FIG. 1) from the actual reality image data indicated by the user's gaze direction, and output a grid interface (200 of FIG. 1) around the user device 100.

3, 4, and 5, the grid interface 200 may be a hologram image obtained by dividing a virtual space overlaid on a three-dimensional coordinate system of an actual image into a plurality of sections. Referring to FIG. 3, the grid interface 200 may be located on the background or the back of the real object. Referring to FIG. 4, the grid interface 200 may be located at the rear and left and right sides of the real object.

Referring to FIG. 5, the grid interface 200 may be divided into multiple layers. That is, the first layer 201, the second layer 203, and the third layer 205 may be divided into a predetermined number of sections, and the e-book information may be divided and displayed for each layer.

When the gesture of opening the book with the user's hand is recognized, the controller 317 lists the plurality of book pages of the e-book on the 3D coordinate system of the actual image according to the gesture recognition. In this case, the controller 317 may display a plurality of book pages in page order in a plurality of sections constituting the grid interface 200.

If the controller 317 recognizes a gesture of displaying each tag on the plurality of book pages and folding and clicking one of the respective tags, the controller 317 may match and store the tag with the recognized gesture. The controller 317 may store the grouping information in which at least one page corresponding to each tag in which the gesture is recognized is matched in the storage 315.

The controller 317 may overlay the book data on the grid interface 200 and output the book data based on the recognition result of the user voice received from the voice interface 311.

When the gesture of clicking the grid interface 200 is recognized, the controller 317 outputs a user interface for adjusting the angle and size of the grid interface 200, and the grid interface 200 according to the gesture recognized through the user interface. Can be changed.

Based on the configuration described so far, a series of processes for providing an augmented reality-based e-book will be described.

6 is a flowchart illustrating a method of providing an e-book according to an embodiment of the present invention, which will be described in connection with the configuration of FIG. 2.

Referring to FIG. 6, the controller 317 recognizes a user's gaze through the gaze interface 307 (S101). The controller 317 recognizes an object by photographing the front image indicated by the user's gaze through the image acquisition unit 303 (S103).

The controller 317 controls the AR screen output unit 301 to output a grid type e-book hologram image (S105) based on the recognized object (S103).

The controller 317 receives a user input signal from the gesture interface 309 or the voice interface 311 (S107), and determines whether the input user input signal is a gesture input or a voice input (S109).

If it is determined that the user input signal is a gesture input, the controller 317 recognizes the gesture with the user input signal input from the gesture interface 309 (S111).

The controller 317 determines that the gesture (S111) is to spread the page with a finger (Finger spread out) (S113), the AR screen output unit 301 to move the page in the direction of the finger to output (S115). To control. This operation allows the user to freely place pages in the desired space with finger gestures, making it easier to compare multiple books at the same time.

When the controller 317 determines that the recognized gesture (S111) is a function of folding and clicking the page tag by a finger (Singer Tag) (S117), the controller 317 stores the page corresponding to the tag with the tag (S119). This operation is to fold your finger on the tag button of the desired page and click on it to automatically sort the pages by tag, which can help to sort and save only the pages related to the topic.

If it is determined in operation S109 that the voice input operation is performed (Voice Find out), the controller 317 overlays the information according to the voice command recognized from the voice interface 311 on the e-book hologram image and outputs the electronic book hologram image in operation S123. This action ignites 'year' and 'writer' so that the year and author information for each book appears, making it easier to find the library information at any time, even if the screen is not changed in order to obtain the library information.

The embodiments of the present invention described above are not only implemented through the apparatus and the method, but may be implemented through a program for realizing a function corresponding to the configuration of the embodiments of the present invention or a recording medium on which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (10)

A wearable display device operated by at least one processor provides augmented reality-based e-book,
Tracking the eyes of the user wearing the wearable display device;
Acquiring a live-action image located in the user's gaze direction;
Augmenting a virtual image displaying an e-book on a three-dimensional coordinate system of the live-action image;
Recognizing a gesture of opening a book with a user's hand, and
Augmenting a virtual image on which the plurality of book pages of the e-book is displayed on the real world 3D coordinate system according to gesture recognition
Including, augmented reality based e-book providing method. In claim 1,
The outputting step,
Generating real reality image data by photographing the real world indicated by the user's gaze direction;
Determining a user device among the objects identified from the real reality image data, and
Outputting a grid interface around the user device,
The grid interface,
A hologram image obtained by dividing a virtual space overlaid on the real world three-dimensional coordinate system into a plurality of sections;
The steps listed above,
And displaying the plurality of book pages in the page order in the plurality of sections, respectively. In claim 2,
After outputting the grid interface,
Recognizing a gesture of clicking the grid interface, and
Adjusting and outputting the angle and size of the grid interface according to the gesture recognition
Further comprising, augmented reality-based e-book providing method. In claim 2,
The steps listed above,
A tag is displayed on each of the plurality of book pages,
After the steps listed above,
Recognizing a gesture of folding and then clicking on the tag; and
Matching and storing the tag with the corresponding page;
The tag is,
Augmented Reality-based e-book providing method used for page grouping. In claim 2,
After the steps listed above,
Recognizing a user's voice, and
Overlaying book data according to voice recognition on the grid interface and outputting the book data;
Further comprising, augmented reality-based e-book providing method. Image acquisition unit for taking an image,
A gaze interface for detecting the movement of the user's eyes,
A gesture interface for recognizing a gesture represented by movement of a user hand,
Augmented reality screen output unit for augmenting the virtual image in which the e-book is displayed on the real world three-dimensional coordinate system,
A storage unit for storing a program providing an augmented reality-based e-book, and
At least one processor for executing the program,
The program,
When the gesture of opening the book with the user's hand is recognized, the virtual image displaying the e-book is projected on the 3D coordinate system of the real world image which is located in the direction of the user's gaze detected by the gaze interface and acquired from the image acquisition. And instructions for arranging a plurality of book pages of the e-book on the real-world three-dimensional coordinate system according to gesture recognition. In claim 6,
The program,
Identify the user device from the actual real image data indicated by the user's gaze direction generated from the camera, project a grid interface around the user device, and page the plurality of book pages in a plurality of sections of the grid interface. Commands for controlling the augmented reality screen output unit to display each, as shown,
The grid interface,
A wearable display device, which is a holographic image obtained by dividing a virtual space overlaid on a real world three-dimensional coordinate system into a plurality of sections. In claim 7,
The program,
And displaying the respective tags on the plurality of book pages, and recognizing a gesture of folding and clicking one tag among the respective tags, and including a command for matching and storing the recognized tag with the corresponding page.
The storage unit,
The wearable display device storing grouping information in which at least one page corresponding to each tag in which a gesture is recognized is matched. In claim 7,
Further comprising a microphone for receiving a user voice,
The processor,
And a command for controlling the augmented reality screen output unit to recognize the user's voice, derive a command, and overlay the book data according to the command on the grid interface for output. In claim 7,
The processor,
Wearable, comprising: instructions for outputting a user interface for adjusting the angle and size of the grid interface when the gesture of clicking the grid interface is recognized and for changing the grid interface according to a gesture recognized through the user interface Display device.

Images