KR20200072575A - Apparatus and method for providing e-book service - Google Patents

Abstract

The present invention relates to a technology for providing an e-book service, and more specifically, to a technology for providing an e-book service that provides image captions and page summary functions. According to an embodiment of the present invention, an image that is difficult for the visually-impaired to understand with only alternative text can be effectively transmitted to the visually-impaired.

Description

E-book service providing device and method{APPARATUS AND METHOD FOR PROVIDING E-BOOK SERVICE}

The present invention relates to an e-book service providing technology, and more particularly, to an e-book service providing technology that provides an image caption and a page summary function.

With the advent of various smart devices, the digitization of documents is progressing, and the use of e-books is rapidly increasing. In particular, due to the ability to read e-books using TTS, users can read books without looking through them. Due to this, even visually impaired people who lacked various information or materials can freely listen to e-books. Moreover, many e-book related technologies have been studied to provide convenience for the visually impaired.

The reading function is a function that outputs the voice from the first to the last text on the page. However, due to the nature of the read-out function that is output through voice, once the voice is played, it cannot be heard again, and there is a inconvenience in that it must be played again from the beginning of the page to listen again. In other words, if the visually impaired person wants to listen to the content of a page again, there is a problem in that he/she has to spend a lot of time re-entering the content of many pages until the page to be searched for is found.

Prior art for the present invention is Patent No. 10-1789057.

The present invention provides an apparatus and method for providing an e-book service that effectively delivers an image that is difficult for a visually impaired person to understand with only alternative text.

In addition, the present invention provides an apparatus and method for providing an e-book service that automatically improves access to each page of a user by automatically summarizing each page and providing a voice.

According to an aspect of the present invention, an apparatus for providing an e-book service is provided.

The apparatus for providing an e-book service according to an embodiment of the present invention accesses a page according to the user voice signal by using an input/output unit that receives a user voice signal, a memory including a voice recognition database, and the voice recognition database, and the It includes a control unit for generating one or more of the summary data and the image caption corresponding to the page, the control unit may output one or more of the summary data and the image caption through the input and output unit by voice.

The voice recognition database includes a voice command, and when the user voice signal corresponds to the voice command, the control unit may access the page corresponding to the voice command.

When the user voice signal does not correspond to the voice command, the control unit may search and access a page including a search word corresponding to the user voice signal.

When the image is included in the page, the controller may generate the image caption.

The controller generates a feature vector by inputting the image into a convolutional neural network (CNN), and generates the image caption by inputting the feature vector into a recurrent neural network (RNN)/long-short term memory (LSTM) network. Can.

The control unit may calculate a text rank for each sentence included in the text of the page, and generate a sentence with the highest text rank as the summary data.

According to another aspect of the present invention, there is provided a method for providing an e-book service in an e-book service providing apparatus and a computer-readable recording medium storing a computer program executing the same.

A method for providing an e-book service in a device for providing an e-book service according to an embodiment of the present invention and a computer-readable recording medium storing a computer program executing the same, receiving a user's voice signal, and the user using a voice recognition database The method may include accessing a page according to a voice signal, generating one or more of summary data and image captions corresponding to the page, and outputting one or more of the summary data and the image captions by voice.

The voice recognition database includes a voice command, and accessing a page according to the user voice signal using the voice recognition database includes: when the user voice signal corresponds to the voice command, corresponding to the voice command It may be a step of accessing the page.

The step of accessing the page according to the user's voice signal using the voice recognition database may include searching and accessing a page including a search word corresponding to the user voice signal when the user voice signal does not correspond to the voice command. It may be a step.

The generating of one or more of summary data and image captions corresponding to the page may be a step of generating the image caption when the page includes an image.

The generating of one or more of summary data and image captions corresponding to the page generates a feature vector by inputting the image into a convolutional neural network (CNN), and the feature vector is a Recurrent Neural Network (RNN)/LSTM (long). -short term memory) may be input to a network to generate the image caption.

Generating one or more of the summary data and the image caption corresponding to the page is a step of calculating a text rank for each sentence included in the text of the page, and generating the sentence with the highest text rank as the summary data Can.

As described above, according to an embodiment of the present invention, an image that is difficult for the visually impaired to understand with only alternative text can be effectively delivered to the visually impaired.

In addition, according to an embodiment of the present invention, by automatically summarizing each page and providing it by voice, access to each page of the user can be improved.

1 is a block diagram illustrating an e-book service providing apparatus according to an embodiment of the present invention.
2 is a flowchart briefly illustrating a process in which an e-book service providing apparatus according to an embodiment of the present invention provides an e-book service.
3 is a flowchart illustrating a process of generating summary data by an e-book service providing apparatus according to an embodiment of the present invention.
4 is a diagram conceptually illustrating a process in which the e-book service providing apparatus according to an embodiment of the present invention generates summary data.
5 is a flowchart illustrating a process in which an e-book service providing apparatus performs image captioning according to an embodiment of the present invention.
6 is a diagram conceptually illustrating a process in which an e-book service providing apparatus according to an embodiment of the present invention performs image captioning.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains may easily practice. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. In addition, when a part includes a certain component, this means that other components may be further included instead of excluding other components unless otherwise specified.

1 is a block diagram illustrating an e-book service providing apparatus according to an embodiment of the present invention, and FIG. 2 briefly illustrates a process in which the e-book service providing apparatus according to an embodiment of the present invention provides an e-book service It is a flowchart.

Referring to FIG. 1, an apparatus for providing an e-book service according to an embodiment of the present invention includes an input/output unit 110, a control unit 120, and a memory 130.

The input/output unit 110 receives a voice signal (hereinafter, referred to as a user voice signal) from a user, and outputs the text-converted voice through a speaker or a text through a display. At this time, the input/output unit 110 may be electrically connected to a device for inputting a voice signal such as a microphone, a device for outputting a voice such as a speaker, and a display.

When the user voice signal is received from the input/output unit 110, the controller 120 searches the voice recognition database to determine whether the user voice signal corresponds to a voice command or a search word. The voice recognition database is a database that stores voice commands. When the user's voice signal corresponds to a voice command as shown in Table 1 below, the controller () performs a system operation according to the voice command.

Voice commands System operation Remark Start, first Go to the first page End, end Go to the last page Previous, front Go to the previous page Next, back Go to the next page Yes, yes, yes Execute image caption function After informing the image, ask the user's intention and interpret "Yes" No no no Image caption function is not executed Other (Search term) Go to the search processing unit and search

The controller 120 accesses a specific page according to a voice command or a search word, generates summary data and an image script for a specific page, and outputs one or more of the summary data and the image script through the input/output unit 110. For example, when the voice command is the first time, the first page of the e-book may be accessed to output summary data of the first page. In addition, when a search word other than a command is input, the controller 120 searches and accesses a page including the search word.

At this time, the memory 130 includes a voice recognition database and a page information database. The page information database stores each text and image of the page, and additionally stores summary data and image captions generated by the controller 120.

Hereinafter, a process in which the e-book service providing apparatus provides the e-book service will be described in detail with reference to FIG. 2. Each process described below will be referred to collectively as an e-book service providing device for a concise and clear description of the process or the invention performed through each functional unit constituting the e-book service providing device.

In step S210, the e-book service providing apparatus receives a user voice signal through the input/output unit 110. At this time, the user voice signal may include a predetermined voice command or search word.

In step S220, the e-book service providing apparatus determines whether the user voice signal is a voice command. For example, the apparatus for providing an e-book service may convert a user voice signal into text using a speech to text (STT) function. The e-book service providing apparatus may recognize the user's voice signal as a voice command when the converted text corresponds to any one of the voice commands stored in the voice command list included in the voice recognition database. In addition, when the converted text is not a voice command stored in a voice command list, the e-book service providing apparatus recognizes the user voice signal as a search word.

If the user's voice signal is a search word in step S220, the e-book service providing apparatus searches and accesses a page including the search word in step S230. At this time, when the page including the search word is not searched, the e-book service providing apparatus may perform the process from step S210 again. In addition, the e-book service providing apparatus may search for a page corresponding to the summary data, if the summary data previously generated includes the search word.

If the user's voice signal is a voice command in step S220, the e-book service providing apparatus accesses the page according to the voice command in step S240.

In step S250, the e-book service providing apparatus determines whether summary data exists on the accessed page. For example, the apparatus for providing an e-book service may determine whether summary data for an accessed page exists in a page information database provided in the memory 130.

If the summary data does not exist in step S250, in step S260, the e-book service providing apparatus generates summary data for the full text of the accessed page. The process of generating the summary data will be described in detail later with reference to FIGS. 3 and 4.

If the summary data is present in step S250, in step S270, the e-book service providing apparatus determines whether an image exists in the page.

If an image exists in step S260, the e-book service providing apparatus performs image captioning to the user in step S280. At this time, the image captioning process will be described in detail later with reference to FIG. 5.

In step S290, the apparatus for providing an e-book service outputs one or more of summary data and an image caption for a page. For example, the apparatus for providing an e-book service outputs summary data and image captions when a user requests an image captioning process.

3 is a flowchart illustrating a process in which the e-book service providing apparatus according to an embodiment of the present invention generates summary data, and FIG. 4 is an e-book service providing apparatus according to an embodiment of the present invention generating summary data This is a conceptual illustration of the process. Each process described below is a process corresponding to step S260 of FIG. 2.

Referring to FIG. 3, in step S310, the e-book service providing apparatus extracts the text of the page.

In step S320, the apparatus for providing an e-book service separates the extracted text into sentence units as shown in FIG. 4 and processes the natural language to generate a TF-IDF model.

In step S330, the e-book service providing apparatus generates a correlation matrix according to the TF-IDF model, calculates a weight graph for each row and column, and calculates a text rank for each sentence according to the weight graph (See Figure 4). At this time, the text rank can be calculated according to Equation 1 below.

[Equation 1]

문장

에 대한 텍스트 랭크

sentence

Text rank for

r _ij : weight between sentences or words i and j

d: Probability of someone surfing the web at the page rank is not satisfied with the page and moves to another page

In step S340, the e-book service providing device selects the sentence with the highest text rank among the sentences as summary data of the page.

In step S350, the e-book service providing apparatus stores summary data in the page list of the page information database.

5 is a flowchart illustrating a process in which an e-book service providing apparatus according to an embodiment of the present invention performs image captioning, and FIG. 6 is an e-book service providing apparatus according to an embodiment of the present invention. It is a diagram conceptually illustrating the process of performing the. At this time, the process of FIG. 5 is a process corresponding to step S280 described in FIG. 2.

Referring to FIG. 5, in step S510, the e-book service providing apparatus requests the user to confirm the use of the image caption, and receives a user voice signal.

In step S520, the apparatus for providing an e-book service determines whether the user's voice signal is a voice command instructing to use an image caption.

In step S520, if the user's voice signal is not a voice command for instructing to use the image caption, the image captioning process is ended.

If the user's voice signal is a voice command instructing the use of the image caption in step S520, in step S530, the e-book service providing apparatus inputs the image into the convolutional neural network (CNN) to generate a feature vector. The convolutional neural network is a model capable of learning while maintaining spatial information of an image.

In step S540, the apparatus for providing an e-book service linearly changes the feature vector to be the same dimension as an input dimension of a Recurrent Neural Network (RNN)/Long-short Term Memory (LSTM) network.

In step S550, the e-book service providing apparatus inputs a feature vector into a recurrent neural network (RNN)/long-short term memory (LSTM) network to generate an image caption. At this time, the RNN/LSTM network is a neural network designed to enable long/short memory, and is a model that can grasp the meaning by synthesizing the elements of the front and back sentences. For example, the apparatus for providing an e-book service may generate an image caption through a language modeling process in which the RNN/LSTM network predicts a sequence sequentially based on a feature vector and converts it into text.

Accordingly, the apparatus for providing an e-book service according to an embodiment of the present invention can automatically generate summary data and an image caption for each page of an e-book and output it in voice, so that the contents of the page can be quickly understood through hearing. have.

The above-described embodiments of the present invention can be implemented through various means. Embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof. For implementation by hardware, methods according to embodiments of the present invention include one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), It can be implemented by field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and the like. In the case of implementation by firmware or software, the method according to embodiments of the present invention may be implemented in the form of a module, procedure, or function that performs the functions or operations described above. A computer program in which software code or the like is recorded may be stored in a computer-readable recording medium or memory unit and driven by a processor. The memory unit is located inside or outside the processor, and can exchange data with the processor by various means already known. Also, combinations of each block of the block diagram and each step of the flowchart attached to the present invention may be performed by computer program instructions. Since these computer program instructions may be mounted on an encoding processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, the instructions performed through the encoding processor of a computer or other programmable data processing equipment may be used in each block of the block diagram or In each step of the flowchart, means are created to perform the functions described. These computer program instructions can also be stored in computer readable or computer readable memory that can be oriented to a computer or other programmable data processing equipment to implement a function in a particular way, so that computer readable or computer readable memory The instructions stored in it are also possible to produce an article of manufacture containing instructions means for performing the functions described in each block or flowchart step of the block diagram. Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so a series of operational steps are performed on a computer or other programmable data processing equipment to create a process that is executed by the computer to generate a computer or other programmable data. It is also possible for instructions to perform processing equipment to provide steps for performing the functions described in each block of the block diagram and in each step of the flowchart. In addition, each block or each step can represent a module, segment, or portion of code that includes one or more executable instructions for executing a specified logical function. It should also be noted that in some alternative embodiments it is also possible that the functions mentioned in blocks or steps occur out of order. For example, two blocks or steps shown in succession may in fact be executed substantially simultaneously, or it is also possible that the blocks or steps are sometimes performed in reverse order depending on the corresponding function.

As such, those skilled in the art to which the present invention pertains will appreciate that the present invention may be implemented in other specific forms without changing its technical spirit or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be interpreted to be included in the scope of the present invention. .

Claims (13)

An input/output unit that receives a user voice signal;
A memory including a speech recognition database; And
A controller for accessing a page according to the user's voice signal using the speech recognition database, and generating one or more of summary data and image captions corresponding to the page,
The control unit is an electronic book service providing apparatus, characterized in that for outputting at least one of the summary data and the image caption through the input and output unit.
According to claim 1,
The voice recognition database includes voice commands,
When the user voice signal corresponds to the voice command, the controller accesses the page corresponding to the voice command.
According to claim 2,
When the user voice signal does not correspond to the voice command, the control unit searches for a page including a search word corresponding to the user voice signal and accesses the e-book service providing apparatus.
According to claim 1,
When the image is included in the page, the control unit generates the image caption.
According to claim 4,
The controller generates a feature vector by inputting the image into a convolutional neural network (CNN), and generates the image caption by inputting the feature vector into a Recurrent Neural Network (RNN)/long-short term memory (LSTM) network. E-book service providing device, characterized in that.
According to claim 1,
The control unit calculates a text rank for each sentence included in the text of the page, and generates a sentence with the highest text rank as the summary data.
A method for providing an e-book service by an e-book service providing device,
Receiving a user voice signal;
Accessing a page according to the user voice signal using a voice recognition database;
Generating one or more of summary data and image captions corresponding to the page; And
And outputting one or more of the summary data and the image caption by voice.
The method of claim 7,
The voice recognition database includes voice commands,
The step of accessing the page according to the user voice signal using the voice recognition database is when the user voice signal corresponds to the voice command, accessing the page corresponding to the voice command. How to provide e-book service.
The method of claim 8,
The step of accessing the page according to the user's voice signal using the voice recognition database may include searching and accessing a page including a search word corresponding to the user voice signal when the user voice signal does not correspond to the voice command. E-book service providing method characterized in that the step.
The method of claim 7,
The generating of one or more of the summary data and the image caption corresponding to the page is a method of providing an e-book service, when the page includes an image, generating the image caption.
The method of claim 10,
The generating of one or more of the summary data and the image caption corresponding to the page generates a feature vector by inputting the image into a convolutional neural network (CNN), and the feature vector is Recurrent Neural Network (RNN)/LSTM (long). -short term memory) A method of providing an e-book service, characterized in that the inputting to a network generates the image caption.
The method of claim 7,
Generating one or more of the summary data and the image caption corresponding to the page is a step of calculating a text rank for each sentence included in the text of the page and generating the sentence with the highest text rank as the summary data E-book service providing method, characterized in that.
A computer-readable recording medium storing a computer program executing any one of claims 7 to 12.

Images