KR20230088048A - Learning method and system based on book image recognition - Google Patents

Abstract

The present invention relates to a learning method and device system capable of acquiring an image by photographing a page of a book through a camera of an electronic device and calling and executing digital content corresponding to data detected through image recognition. A learning method based on book image recognition includes: acquiring a photographed image of a book by a camera driving unit driving a camera; detecting data by an image recognition unit performing image recognition from the photographed book image; and a function executing unit selecting a current operation mode. determining, and executing, by the function execution unit, a specific function corresponding to the detected data according to the current operation mode, wherein the detected data detected through image recognition includes at least a book title and a current page. do.

Description

Learning method and system based on book image recognition

The present invention relates to the provision of learning services. More specifically, the present invention relates to a learning method and system capable of acquiring an image by photographing a page of a book through a camera of an electronic device and calling and executing digital content corresponding to data detected through image recognition.

Technologies for automatically recognizing book page content and providing related functions have been proposed and developed.

For example, according to Utility Model Registration No. 20-0245775 (Title: A device that automatically recognizes the pages of a book and sounds the contents of the book), a switch sensor capable of recognizing pages between the pages of a book is used. A device has been proposed that recognizes the page according to the opening and closing of the switch and outputs the contents of the corresponding page as sound whenever the book is opened and each page is turned.

However, since the switch sensor is attached to each page of the book, this prior art has disadvantages in that practicality is poor and page recognition is impossible when several pages are turned at once.

As another example, according to Utility Model Registration No. 20-0234062 (name: book for generating sound), holes corresponding to the page number are drilled on each page of the book, and light sensors are installed on the end page of the book. A technique of recognizing a page by detecting light induced through a hole and outputting the contents of the corresponding page as voice has been proposed.

However, this prior art has a disadvantage in that it is not practical because not only physical perforation is required on all pages of the book, but also light detection sensors must be installed as many as the number of pages.

As another example, according to Utility Model Registration No. 20-0262658 (title: book for generating voice with earphone), a sound generating device with a voice chip is installed on the back cover of the book, and the contents of the book embedded in the voice chip A device for listening to the back through an earphone has been proposed.

However, this prior art has a disadvantage in that it is not practical because it is impossible to designate a page and a sound generating device equipped with a voice chip must be installed in the book.

As another example, a book has been developed that listens to the content of the page by scanning a QR code printed on each page of the book. However, since the QR code is printed on a page-by-page basis, this prior art has a disadvantage in that content related to a specific row or specific word desired by the user cannot be provided within the page.

Registered Utility Model No. 20-0245775 (Announced on October 15, 2001) Registered Utility Model No. 20-0234062 (2001.10.08. Notice) Registered Utility Model No. 20-0262658 (Announced on March 18, 2002)

An object of the present invention is to implement a learning method and system capable of capturing an image of a page of a book through a camera of an electronic device, acquiring an image, and calling and executing digital content corresponding to data detected through image recognition.

The present invention provides a learning method and system based on book image recognition.

According to an embodiment of the present invention, a learning method based on book image recognition may include: obtaining a photographed image of a book by a camera driving unit driving a camera; detecting data by an image recognition unit by performing image recognition on the photographed image of the book; determining a current operation mode by a function execution unit; and executing, by the function execution unit, a specific function corresponding to the detected data according to the current operation mode, wherein the detected data detected through image recognition includes at least a book title and a current page.

According to an embodiment of the present invention, the current operating mode includes i) a first operating mode for calling and executing digital content, ii) a second operating mode for displaying the content of a book as a graphic element on the screen of an electronic device, and iii) It may be one of the third operation modes for scoring problem solving.

According to an embodiment of the present invention, the detection data detected through the image recognition is, in addition to the book title and the current page, a specific position within the current page according to the user's instructional operation and the user's gesture input operation. At least one of user gestures may be further included.

According to an embodiment of the present invention, when the current operation mode is the first operation mode and the detected data is the book title and the current page, a specific function executed corresponding to the detected data corresponds to the current page. It may be at least one of reproduction, creation, and storage of audio content to be played, and if the detection data is a specific location in the current page, a specific function executed in response to the detection data is to call digital content related to the specific location and If the detection data is the user gesture, the specific function executed in response to the detection data may be a menu call corresponding to the user gesture and reception of a user input related to the menu.

According to an embodiment of the present invention, when the current operation mode is the second operation mode, the detected data is the book title and the current page, and a specific function executed in response to the detected data corresponds to the current page. It may be to schematize the content and display it as a graphic element on the screen of the electronic device.

According to an embodiment of the present invention, when the current operation mode is the third operation mode, the detected data are the book title, the current page, and the user gesture, and a specific function executed in response to the detected data It may be to score the problem solving according to the user gesture for the problem corresponding to the current page.

According to an embodiment of the present invention, a learning system based on book image recognition may include: a camera driving unit driving a camera to acquire a photographed image of a book; an image recognizing unit detecting data by performing image recognition from the photographed image of the book; and a function execution unit that determines a current operation mode and executes a specific function corresponding to the detected data according to the current operation mode, wherein the detected data detected through image recognition includes at least a book title and a current page. do.

According to the present invention, a learning service with high practicality and improved user convenience is provided by capturing an image of a page of a book through a camera of an electronic device, obtaining an image, and calling and executing digital content corresponding to data detected through image recognition. can provide

1 is a schematic diagram showing the concept of a book image recognition based learning service according to an embodiment of the present invention.
2 is a block diagram illustrating a learning system based on book image recognition and an implementation environment thereof according to an embodiment of the present invention.
3 is a flowchart illustrating a learning method based on book image recognition according to an embodiment of the present invention.
4 is a diagram showing an example of an operation mode in which book contents are schematically displayed on a screen of an electronic device as graphic elements in a learning method based on book image recognition according to an embodiment of the present invention.
5 is a diagram illustrating another example of an operation mode in which book contents are schematically displayed on a screen of an electronic device as graphic elements in a learning method based on book image recognition according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In describing the embodiments, descriptions of technical contents that are well known in the technical field to which the present invention pertains and are not directly related to the present invention will be omitted. This is to more clearly convey the gist of the present invention without obscuring it by omitting unnecessary description.

Like reference numerals are assigned to like elements in the drawings. On the other hand, singular or plural expressions for components are appropriately selected according to the presented circumstances for convenience of description, and the present invention is not limited to singular or plural forms. Elements expressed in the singular form may be interpreted as plural elements, and vice versa.

Concept of book image recognition based learning service

1 is a schematic diagram showing the concept of a book image recognition based learning service according to an embodiment of the present invention.

Referring to FIG. 1 , a user's electronic device 10 and a book 20 are mounted on a cradle 30 or only the electronic device 10 is mounted. The cradle 30 has a predetermined inclined surface and the inclined angle is adjustable. The book 20 can be spread on the surface of the holder 30 and placed on the bottom surface facing the holder 30 . The electronic device 10 is, for example, a smart phone and is located above or next to the book 20 . The electronic device 10 includes a camera, and is fixed to the cradle 30 at an angle so that the entire page of the book 20 can be photographed with the camera.

When a page of the book 20 is photographed through the camera of the electronic device 10, the electronic device 10 may detect predetermined data through image recognition and execute a specific function corresponding to the detected data. Here, photographing includes, but is not limited to, an act of recognizing an image of the book 20 or an act of recognizing and then storing an image of the book 20 . For example, the current page of the book 20 may be recognized and corresponding audio content may be called and reproduced. In addition, when a user points a specific location in a page with a finger, various digital contents related to the indicated location can be called and executed. In this way, the present invention provides a learning service based on book image recognition, which will be described in detail later.

On the other hand, the electronic device 10 can not only audio-visually output digital content on its own, but also can work with one or more external devices 40 capable of visually and/or aurally outputting digital content. In addition, the electronic device 10 may access the external server 50 for book image recognition, digital content calling, and the like.

Book Image Recognition-Based Learning System and Implementation Environment

2 is a block diagram illustrating a learning system based on book image recognition and an implementation environment thereof according to an embodiment of the present invention.

A learning system 100 based on book image recognition according to an embodiment of the present invention includes a camera driving unit 110, an image recognizing unit 120, a function execution unit 130, and an external device linking unit 140.

According to an embodiment of the present invention, the learning system 100 may be installed in the form of an application or hardware/software module in the aforementioned electronic device (10 in FIG. 1). In some embodiments, the learning system 100 may be implemented as an independent device separate from the electronic device 10.

The learning system 100 may be associated with the camera 11, the storage unit 12, the output unit 13, and the communication unit 14, which are components of the electronic device 10. In addition, the learning system 100 may be connected to components of the electronic device 10 under the control of a controller (not shown) of the electronic device 10 . In addition, the learning system 100 may be connected to the external device 40 and/or the server 50 through the electronic device 10 or directly.

The camera driving unit 110 of the learning system 100 drives the camera 11 to acquire a photographic image of a book. The obtained book shooting image is transmitted to the image recognition unit 120 .

The image recognition unit 120 of the learning system 100 detects data by performing image recognition from a photographed image of a book. Image recognition may be performed through similar image search based on book files stored in the storage unit 12 or the server 50 or through deep learning-based image recognition. The detection data detected through image recognition is at least one of a book title, a current page, a specific position within the current page according to a user's instructional operation, and a user gesture according to a user's gesture input operation. This will be explained in detail later.

The function execution unit 130 of the learning system 100 determines the current operation mode. The current operating modes are i) a first operating mode that calls and executes digital content, ii) a second operating mode that diagrams the contents of a book and displays them as graphic elements on the screen of an electronic device, and iii) a third that scores problem solving. One of the modes of operation. This will be explained in detail later.

In addition, the function execution unit 130 of the learning system 100 executes a specific function corresponding to the detected data according to the current operation mode. For example, if the current operation mode is the first operation mode and the detected data is the title of the book and the current page, the specific function executed in response to the detected data is during reproduction, generation, and storage of audio content corresponding to the current page. at least one The current operation mode, detection data, and the function executed accordingly will be described in detail later.

The function execution unit 130 of the learning system 100 may interwork with the output unit 13 of the electronic device 10 when executing a specific function. That is, the function execution unit 130 may use the display and sound output of the output unit 13 . In addition, the function executing unit 130 may interwork with the external device 40 through the external device linking unit 140 and the communication unit 14 when executing a specific function.

Learning method based on book image recognition

Hereinafter, a learning method based on book image recognition according to an embodiment of the present invention will be described with reference to FIGS. 2 and 3 . 3 is a flowchart illustrating a learning method based on book image recognition according to an embodiment of the present invention. From the description below, the operations of the learning system 100 described above will become more apparent.

First, in step S1, the camera driving unit 110 of the learning system 100 drives the camera 11 to obtain a photographic image of a book. At this time, the photographed book image is an image obtained by capturing the currently opened page of the book (20 in Fig. 1) placed on the holder (30 in Fig. 1). In some cases, the book photographed image may be an image of a user's finger indicating a specific location of the book 20 together with the current page of the book 20, or a book (with the current page of the book 20) 20) It may be an image of a gesture taken by the user from above.

Next, in step S2, the image recognition unit 120 of the learning system 100 detects data by performing image recognition on the photographed image of the book. The detection data detected through image recognition is at least one of a book title, a current page, a specific position within the current page according to a user's instructional operation, and a user gesture according to a user's gesture input operation. In this case, the specific position in the current page may be a specific word, picture, table, etc. pointed by the user with one finger, and if the user points to a blank space in a specific row, the specific position may be the row indicated by the user. In addition, a user gesture is a predetermined user input operation other than an operation indicating a specific location. For example, the user gesture includes a pointing operation with two fingers, an operation of moving one finger in a specific direction, an operation of drawing a specific symbol with one finger, and the like.

Such image recognition may be performed through similar image search based on book files stored in the storage unit 12 or the server 50 or through deep learning-based image recognition. That is, the title of the book and the current page can be detected through object detection and classification in the photographed image of the book using image recognition technology, one of computer vision technologies, and a specific location and user gesture indicated by recognizing the user's finger and motion can be detected. can

Subsequently, in step S3, the function execution unit 130 of the learning system 100 determines the current operation mode. The current operating modes are i) a first operating mode that calls and executes digital content, ii) a second operating mode that diagrams the contents of a book and displays them as graphic elements on the screen of an electronic device, and iii) a third that scores problem solving. One of the modes of operation.

The current operation mode is designated by default or may be designated in advance by user selection. In addition, the current operation mode may be changed by user input in the process of providing the learning service. The current operation mode will be described in detail again in the following description of specific function execution.

Subsequently, in step S4, the function execution unit 130 of the learning system 100 executes a specific function corresponding to the detected data according to the current operation mode. That is, the function executor 130 executes a specific function corresponding to the detection data detected in step S2 based on the current operation mode determined in step S3.

(1) When the current operation mode is the first operation mode

When the current operation mode is the first operation mode for calling and executing digital content, a specific function to be executed varies according to detected data.

If the detected data is "book title and current page", a specific function executed in response to the detected data is "at least one of playback, creation, and storage of audio contents corresponding to the current page".

If the detection data is “a specific position in the current page”, a specific function executed corresponding to the detection data is “calling and executing digital contents related to a specific position in the current page”.

If the detected data is a “user gesture”, a specific function executed in response to the detected data is “calling a menu corresponding to the user gesture and receiving menu-related user input”.

In the case of the first operation mode, digital content may be repeatedly called through a menu call, and a speaker/explainer may be changed to view or listen. In addition, it is possible to control the start, pause, stop, speed control, volume control, and skip of digital content through a menu call. In addition, through a menu call, a voice similar to that of a preferred person (frequency, voice tone, demographic information, region, preference ranking, etc.) can be generated and listened to. In addition, the menu may include a function of recommending digital contents, a function of checking digital contents use records (by page, by row, by specific location), and by managing digital contents use (learning) goals and achievement rates. In addition, in the case of a book for which there is no recorded copy yet, or in case an error is found in a specific page or a specific location in the book, or if an update is required, a function to input a user request may also be included.

(2) When the current operation mode is the second operation mode

If the current operation mode is the second operation mode in which the contents of the book are schematized and displayed as graphic elements on the screen of an electronic device, the detected data is "book title and current page", and the specific function executed in response to the detected data is "current The content corresponding to the page is diagrammed and displayed as a graphic element on the screen of the electronic device”. This will be described with reference to FIGS. 4 and 5 .

FIG. 4 is a diagram showing an example of an operation mode in which book contents are schematized and displayed as graphic elements on the screen of an electronic device in a learning method based on book image recognition according to an embodiment of the present invention. FIG. It is a diagram showing another example of an operation mode in which the content of a book is schematized and displayed as a graphic element on a screen of an electronic device in the learning method based on book image recognition according to the example.

As shown in FIGS. 4 and 5 , when a book photographed image of a current page of the book 20 is acquired, contents corresponding to the current page are mapped through image recognition. Then, the schematized contents are displayed as graphic elements 60 and 70 on the screen of the output unit 13 of the electronic device 100. When the contents of the current page are schematized, only a part of the start word is exposed and/or Ad Hoc software It can be exposed on the screen as a button-like graphic element 60, 70. After that, the user can select a graphic element displayed on the screen of the electronic device to call/execute the corresponding digital content.

(3) When the current operation mode is the third operation mode

If the current operation mode is the third operation mode for scoring problem solving, the detection data is "book title, current page, and user gesture", and the specific function executed in response to the detection data is "the problem corresponding to the current page. It is to score the problem solving according to the user's gestures.

For example, if the book is a textbook including a workbook or problems, a user's action of marking a fingerprint of a multiple-choice question or writing an answer to a short-answer question may be recognized and a correct answer may be scored.

In addition, an incorrect answer note according to the scoring result may be created and provided/stored, or a user's weakness may be identified from the scoring result and customized learning materials for subsequent learning may be provided.

As described above, according to the present invention, by taking a picture of a page of a book through a camera of an electronic device to acquire an image and calling and executing digital content corresponding to data detected through image recognition, practicality is high and user convenience is high. This improved learning service can be provided.

Meanwhile, each block of the flowchart described above may be performed by computer program instructions. Since these computer program instructions may be loaded into a processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, the instructions executed by the processor of the computer or other programmable data processing equipment may function as described in the flowchart blocks. create a means to do them. These computer program instructions may also be stored in a computer usable or computer readable memory that can be directed to a computer or other programmable data processing equipment to implement functionality in a particular way, such that the computer usable or computer readable memory The instructions stored in are also capable of producing an article of manufacture containing instruction means that perform the functions described in the flowchart blocks. The computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to generate computer or other programmable data processing equipment. Instructions for performing processing equipment may also provide steps for performing the functions described in the flowchart blocks. Additionally, each block may represent a module, segment, or portion of code that includes executable instructions for executing specific logical functions. Also, in some embodiments, functions shown in blocks may be executed in a different order than shown. For example, two blocks shown in succession may be performed substantially concurrently or may be performed in reverse order in some cases.

Embodiments of the present invention disclosed in this specification and drawings are only presented as specific examples to easily explain the technical content of the present invention and help understanding of the present invention, and are not intended to limit the scope of the present invention. In addition to the embodiments disclosed herein, it is obvious to those skilled in the art that other modifications based on the technical idea of the present invention can be implemented.

10: electronic device 11: camera
12: storage unit 13: output unit
14: Department of Communication 20: Book
30: cradle 40: external device
50: server 100: learning system
110: camera driving unit 120: image recognition unit
130: function execution unit 140: external device interlocking unit

Claims (12)

In the learning method based on book image recognition,
Acquiring a photographic image of a book by a camera drive unit driving a camera;
detecting data by an image recognition unit by performing image recognition on the photographed image of the book;
determining a current operation mode by a function execution unit; and
executing a specific function corresponding to the detected data according to the current operation mode by the function executing unit;
including,
The learning method based on book image recognition, characterized in that the detection data detected through the image recognition includes at least a book title and a current page. According to claim 1,
The current operating mode is i) a first operating mode for calling and executing digital content, ii) a second operating mode for displaying the content of a book as a graphic element on the screen of an electronic device by schematizing, and iii) a problem solving score. A learning method based on book image recognition, characterized in that one of three operation modes. According to claim 2,
In addition to the book title and the current page, the detection data detected through the image recognition further includes at least one of a specific position within the current page according to the user's instructional operation and a user gesture according to the user's gesture input operation. A learning method based on book image recognition, characterized in that for doing. According to claim 3,
When the current operation mode is the first operation mode,
If the detection data is the book title and the current page, the specific function executed in response to the detection data is at least one of reproduction, generation, and storage of audio content corresponding to the current page;
If the detected data is a specific location within the current page, a specific function executed corresponding to the detected data is calling and executing digital content related to the specific location;
If the detection data is the user gesture, the specific function executed in response to the detection data is calling a menu corresponding to the user gesture and receiving a user input related to the menu. According to claim 2,
When the current operation mode is the second operation mode, the detected data is the book title and the current page, and the specific function executed in response to the detected data is a diagram of the contents corresponding to the current page, A learning method based on book image recognition, characterized in that it is displayed as a graphic element on the screen. According to claim 3,
When the current operation mode is the third operation mode, the detected data are the book title, the current page, and the user gesture, and a specific function executed in response to the detected data is related to a problem corresponding to the current page. A learning method based on book image recognition, characterized in that for scoring the problem solving according to the user gesture. In a learning system based on book image recognition,
A camera driving unit that drives a camera to obtain a photographic image of a book;
an image recognizing unit detecting data by performing image recognition from the photographed image of the book; and
a function execution unit that determines a current operation mode and executes a specific function corresponding to the detected data according to the current operation mode;
including,
The learning system based on book image recognition, characterized in that the detection data detected through the image recognition includes at least a book title and a current page. According to claim 7,
The current operating mode is i) a first operating mode for calling and executing digital content, ii) a second operating mode for displaying the content of a book as a graphic element on the screen of an electronic device by schematizing, and iii) a problem solving score. A learning system based on book image recognition, characterized in that one of three operation modes. According to claim 8,
In addition to the book title and the current page, the detection data detected through the image recognition further includes at least one of a specific position within the current page according to the user's instructional operation and a user gesture according to the user's gesture input operation. A learning system based on book image recognition. According to claim 9,
When the current operation mode is the first operation mode,
If the detection data is the book title and the current page, the specific function executed in response to the detection data is at least one of reproduction, generation, and storage of audio content corresponding to the current page;
If the detected data is a specific location within the current page, a specific function executed corresponding to the detected data is calling and executing digital content related to the specific location;
If the detection data is the user gesture, the specific function executed in response to the detection data is calling a menu corresponding to the user gesture and receiving a user input related to the menu. According to claim 8,
When the current operation mode is the second operation mode, the detected data is the book title and the current page, and the specific function executed in response to the detected data is a diagram of the contents corresponding to the current page, A learning system based on book image recognition, characterized in that it is displayed as a graphic element on the screen. According to claim 9,
When the current operation mode is the third operation mode, the detected data are the book title, the current page, and the user gesture, and a specific function executed in response to the detected data is related to a problem corresponding to the current page. A learning system based on book image recognition, characterized in that for scoring the problem solving according to the user gesture.

Images