KR102201153B1 - Apparatus and method for providing e-book service - Google Patents

Abstract

The present invention relates to an e-book service providing technology, and more particularly, to an e-book service providing technology that provides an image caption and a page summary function. According to an embodiment of the present invention, an image that is difficult for a visually impaired person to understand with only alternative text can be effectively transmitted to the visually impaired.

Description

Device and method for providing e-book service {APPARATUS AND METHOD FOR PROVIDING E-BOOK SERVICE}

The present invention relates to an e-book service providing technology, and more particularly, to an e-book service providing technology that provides an image caption and a page summary function.

With the advent of various smart devices and digitization of documents, the use of e-books is increasing rapidly. In particular, due to the reading function of e-books using TTS, users can read books without looking at them with their own eyes. Due to this, even visually impaired people who lacked various information or materials can freely listen to e-books. Moreover, many e-book related technologies have been studied to provide convenience for the visually impaired.

The read function is a function that outputs the text from the beginning to the end of the page by voice. However, due to the characteristics of the reading function that is output as a voice, once the voice is played, it cannot be heard again, and in order to listen again, there is an inconvenience of having to play again from the beginning of the page. In other words, when visually impaired people want to listen to the content of a certain page again, there is a problem in that they have to spend a lot of time re-entering the content of many pages until they find the page they want to find.

As a prior art for the present invention, there is Registration Patent No. 10-1789057.

The present invention provides an apparatus and method for providing an e-book service for effectively delivering an image that is difficult for the visually impaired to understand only with alternative texts to the visually impaired.

In addition, the present invention provides an apparatus and method for providing an e-book service for improving the accessibility of each page by a user by automatically summarizing each page and providing a voice.

According to an aspect of the present invention, an apparatus for providing an e-book service is provided.

The e-book service providing apparatus according to an embodiment of the present invention accesses a page according to the user voice signal using an input/output unit receiving a user voice signal, a memory including a voice recognition database, and the voice recognition database, and the And a control unit that generates at least one of summary data and image caption corresponding to a page, wherein the control unit may output at least one of the summary data and the image caption as a voice through the input/output unit.

The voice recognition database includes voice commands, and the controller may access the page corresponding to the voice command when the user voice signal corresponds to the voice command.

When the user voice signal does not correspond to the voice command, the controller may search for and access a page including a search word corresponding to the user voice signal.

When the page contains an image, the control unit may generate the image caption.

The control unit generates a feature vector by inputting the image to a convolutional neural network (CNN), and inputs the feature vector to a recurrent neural network (RNN)/long-short term memory (LSTM) network to generate the image caption. I can.

The control unit may calculate a text rank for each sentence included in the text of the page, and generate a sentence having the highest text rank as the summary data.

According to another aspect of the present invention, a method for providing an e-book service in an e-book service providing apparatus and a computer-readable recording medium storing a computer program for executing the same are provided.

In the apparatus for providing an e-book service according to an embodiment of the present invention, a method for providing an e-book service and a computer-readable recording medium storing a computer program that executes the same, receiving a user voice signal, and the user using a voice recognition database Accessing the page according to the audio signal, generating at least one of summary data and image caption corresponding to the page, and outputting at least one of the summary data and the image caption as a voice.

The voice recognition database includes a voice command, and the step of accessing a page according to the user voice signal using the voice recognition database comprises: when the user voice signal corresponds to the voice command, corresponding to the voice command It may be a step of accessing the page.

The step of accessing a page according to the user voice signal using the voice recognition database includes: when the user voice signal does not correspond to the voice command, a page containing a search word corresponding to the user voice signal is searched and accessed. It may be a step to do.

The generating of at least one of summary data and image caption corresponding to the page may be a step of generating the image caption when the page contains an image.

In the step of generating at least one of summary data and image caption corresponding to the page, a feature vector is generated by inputting the image into a convolutional neural network (CNN), and the feature vector is recurrent neural network (RNN)/long -short term memory) may be a step of generating the image caption by inputting it into a network.

The generating of at least one of summary data and image caption corresponding to the page is a step of calculating a text rank for each sentence included in the text of the page, and generating a sentence having the highest text rank as the summary data I can.

As described above, according to an embodiment of the present invention, an image that is difficult for a visually impaired person to understand only with alternative text can be effectively transmitted to the visually impaired.

In addition, according to an embodiment of the present invention, each page is automatically summarized and provided as a voice, thereby improving accessibility to each page of a user.

1 is a block diagram illustrating an e-book service providing apparatus according to an embodiment of the present invention.
2 is a flow chart briefly illustrating a process of providing an e-book service by an e-book service providing apparatus according to an embodiment of the present invention.
3 is a flow chart illustrating a process of generating summary data by an e-book service providing apparatus according to an embodiment of the present invention.
4 is a diagram conceptually illustrating a process of generating summary data by an e-book service providing apparatus according to an embodiment of the present invention.
5 is a flowchart illustrating a process of performing image captioning by an e-book service providing apparatus according to an embodiment of the present invention.
6 is a diagram conceptually illustrating a process of performing image captioning by an e-book service providing apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the present invention. However, the present invention may be implemented in various forms and is not limited to the embodiments described herein. In addition, when a certain part includes a certain component, this means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

1 is a block diagram illustrating a device for providing an e-book service according to an embodiment of the present invention, and FIG. 2 is a simplified illustration of a process in which an e-book service providing device according to an embodiment of the present invention provides an e-book service It is a flow chart.

Referring to FIG. 1, an apparatus for providing an e-book service according to an embodiment of the present invention includes an input/output unit 110, a control unit 120, and a memory 130.

The input/output unit 110 receives a voice signal (hereinafter, referred to as a user voice signal) from a user, and outputs the converted voice through a speaker or outputs the text through a display. In this case, the input/output unit 110 may be electrically connected to a device for inputting a voice signal such as a microphone, a device for outputting voice such as a speaker, and a display.

When receiving a user voice signal from the input/output unit 110, the controller 120 searches the voice recognition database and determines whether the user voice signal corresponds to a voice command or a search word. The voice recognition database is a database that stores voice commands. When the user voice signal corresponds to the voice command as shown in Table 1 below, the control unit () performs a system operation according to the voice command.

Voice command System operation Remark Beginning, beginning Go to the first page End, end Go to the last page Previous, front Go to the previous page Next, back Go to the next page Yes, yes, yes Execute the image caption function After notifying the image information, ask the user's intention and interpret it if "Yes" No, no Do not execute the image caption function Other (search terms) Go to the search processing unit and search

The controller 120 accesses a specific page according to a voice command or search word, generates summary data and image script for the specific page, and outputs at least one of the summary data and image script through the input/output unit 110. For example, when the voice command is the first, it is possible to access the first page of the e-book and output the summary data of the first page. In addition, when a search word other than a command is input, the controller 120 searches for and accesses a page including the search word.

In this case, the memory 130 includes a voice recognition database and a page information database. The page information database stores text and images of each page, and additionally stores summary data and image captions generated by the controller 120.

Hereinafter, a process of providing an e-book service by an e-book service providing device will be described in detail with reference to FIG. 2. Each of the processes to be described below will be collectively referred to as an e-book service providing device for a concise and clear description of the invention or a process performed by each functional unit constituting the e-book service providing device.

In step S210, the e-book service providing apparatus receives a user voice signal through the input/output unit 110. In this case, the user voice signal may include a predefined voice command or search word.

In step S220, the e-book service providing apparatus determines whether the user's voice signal is a voice command. For example, the e-book service providing device may convert a user's voice signal into text using a Speech to Text (STT) function. When the converted text corresponds to one of the voice commands stored in the voice command list included in the voice recognition database, the e-book service providing apparatus may recognize a user voice signal as a voice command. In addition, when the converted text is not a voice command stored in the voice command list, the e-book service providing apparatus recognizes a user voice signal as a search word.

When the user's voice signal is a search word in step S220, the e-book service providing apparatus searches for and accesses a page including the search word in step S230. In this case, when the page including the search word is not searched, the e-book service providing apparatus may perform the process from step S210 again. In addition, when there is a previously generated summary data including a search word, the e-book service providing device may search for a page corresponding to the summary data.

When the user voice signal is a voice command in step S220, the e-book service providing apparatus accesses a page according to the voice command in step S240.

In step S250, the e-book service providing apparatus determines whether summary data exists in the accessed page. For example, the e-book service providing apparatus may determine whether summary data on the accessed page exists in the page information database provided in the memory 130.

If the summary data does not exist in step S250, the e-book service providing device generates summary data for the full text of the accessed page in step S260. The process of generating the summary data will be described in detail later with reference to FIGS. 3 and 4.

If summary data exists in step S250, the e-book service providing apparatus determines whether an image exists on the page in step S270.

If the image exists in step S260, the e-book service providing device performs image captioning to the user in step S280. In this case, the image captioning process will be described in detail later with reference to FIG. 5.

In step S290, the e-book service providing apparatus outputs one or more of summary data and image captions for the page. For example, when a user requests an image captioning process, the e-book service providing apparatus outputs summary data and image captions.

3 is a flowchart illustrating a process of generating summary data by an e-book service providing device according to an embodiment of the present invention, and FIG. 4 is a flowchart illustrating a process of generating summary data by an e-book service providing device according to an embodiment of the present invention. It is a diagram conceptually illustrating the process of doing so. Each process described below is a process corresponding to step S260 of FIG. 2.

Referring to FIG. 3, in step S310, the apparatus for providing an e-book service extracts text of a page.

In step S320, the e-book service providing apparatus separates the extracted text into sentences as shown in FIG. 4 and processes natural language to generate a TF-IDF model.

In step S330, the e-book service providing apparatus generates a correlation matrix according to the TF-IDF model, calculates a weight graph for each row and column, and calculates a text rank for each sentence according to the weight graph. (See Fig. 4). In this case, the text rank can be calculated according to Equation 1 below.

[Equation 1]

문장

에 대한 텍스트 랭크

sentence

Text rank for

r _ij : weight between sentences or words i and j

d: Probability of a person surfing the web at page rank that is not satisfied with the page and goes to another page

In step S340, the e-book service providing apparatus selects a sentence with the highest text rank among each sentence as summary data of the page.

In step S350, the e-book service providing apparatus stores summary data in a page list of the page information database.

5 is a flowchart illustrating a process of performing image captioning by an e-book service providing device according to an embodiment of the present invention, and FIG. 6 is a flowchart illustrating a process of performing image captioning by an e-book service providing device according to an embodiment of the present invention. It is a diagram conceptually illustrating the process of performing. In this case, the process of FIG. 5 corresponds to step S280 described above in FIG. 2.

Referring to FIG. 5, in step S510, the apparatus for providing an e-book service requests a user to confirm use of an image caption, and receives a user voice signal.

In step S520, the e-book service providing apparatus determines whether the user's voice signal is a voice command instructing to use the image caption.

In step S520, if the user voice signal is not a voice command commanding the use of the image caption, the image captioning process is terminated.

In step S520, if the user's voice signal is a voice command instructing to use the image caption, in step S530, the e-book service providing apparatus inputs the image into a convolutional neural network (CNN) to generate a feature vector. A convolutional neural network is a model that can learn while maintaining spatial information of an image.

In step S540, the e-book service providing apparatus linearly changes the feature vector to have the same dimension as the input dimension of the recurrent neural network (RNN)/long-short term memory (LSTM) network.

In step S550, the e-book service providing apparatus generates an image caption by inputting a feature vector into a recurrent neural network (RNN)/long-short term memory (LSTM) network. In this case, the RNN/LSTM network is a neural network designed to enable long/short memory, and is a model capable of grasping the meaning by synthesizing elements of the front and back sentences. For example, the e-book service providing apparatus may generate an image caption through a language modeling process in which the RNN/LSTM network sequentially predicts sequences based on feature vectors and converts them to text as shown in FIG. 6.

Accordingly, the apparatus for providing an e-book service according to an embodiment of the present invention automatically generates summary data and image captions for each page of the e-book and outputs it as a voice, so that the contents of the page can be quickly recognized through hearing. have.

The above-described embodiments of the present invention can be implemented through various means. Embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, the method according to embodiments of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), It can be implemented by field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and the like. In the case of implementation by firmware or software, the method according to the embodiments of the present invention may be implemented in the form of a module, procedure, or function that performs the functions or operations described above. A computer program in which software codes and the like are recorded may be stored in a computer-readable recording medium or a memory unit and driven by a processor. The memory unit may be located inside or outside the processor, and may exchange data with the processor through various known means. In addition, combinations of each block of the block diagram attached to the present invention and each step of the flowchart may be performed by computer program instructions. Since these computer program instructions can be mounted on the encoding processor of a general-purpose computer, special purpose computer or other programmable data processing equipment, the instructions executed by the encoding processor of the computer or other programmable data processing equipment are each block of the block diagram or Each step of the flow chart will create a means to perform the functions described. These computer program instructions can also be stored in computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular way, so that the computer-usable or computer-readable memory It is also possible to produce an article of manufacture in which the instructions stored in the block diagram contain instruction means for performing the functions described in each block or flow chart. Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so that a series of operating steps are performed on a computer or other programmable data processing equipment to create a computer-executable process to create a computer or other programmable data processing equipment. It is also possible for the instructions to perform the processing equipment to provide steps for performing the functions described in each block of the block diagram and each step of the flowchart. In addition, each block or each step may represent a module, segment, or part of code including one or more executable instructions for executing a specified logical function. It should also be noted that, in some alternative embodiments, functions mentioned in blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially simultaneously, or the blocks or steps may sometimes be performed in the reverse order depending on the corresponding function.

As such, those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in other specific forms without changing the technical spirit or essential features. Therefore, the embodiments described above are illustrative in all respects and should be understood as non-limiting. The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. .

Claims (13)

An input/output unit for receiving a user's voice signal and outputting the converted text through a speaker or outputting the text through a display;
A memory including a speech recognition database including a list of speech commands and a page information database including page summary data; And
When receiving a user voice signal, search a voice recognition database to determine whether the user voice signal corresponds to a voice command or search word, and if the user voice signal corresponds to a voice command, perform a system operation according to the voice command, Access a page according to the user's voice signal using the speech recognition database, generate at least one of summary data and image caption corresponding to the page, and at least one of the summary data and the image caption through the input/output unit Including a control unit for outputting voice,
When the user voice signal corresponds to the voice command list, the controller accesses the page corresponding to the voice command,
When the user voice signal does not correspond to the voice command list, a page containing a search word corresponding to the user voice signal is searched and accessed,
When the page contains an image, a feature vector is generated by inputting the image to a convolutional neural network (CNN), and the feature vector is input to a recurrent neural network (RNN)/long-short term memory (LSTM) network. To generate the image caption,
It is determined whether summary data corresponding to the page exists in the page information data, and if the summary data does not exist, the text of the page is extracted,
The extracted text is separated into sentences, and natural language processing is performed to generate a TF-IDF model, a correlation matrix according to the TF-IDF model is generated, a weight graph for each row and column is calculated, and a weight graph The text rank for each sentence according to is calculated according to Equation 1 below,
[Equation 1]

TR(Vi): text rank for sentence Vi
rij: weight between sentences or words i and j,
d: Probability of a person surfing the web at page rank not satisfied with the page and navigate to another page,
An e-book service providing apparatus for calculating a text rank for each sentence included in the text of the page, generating a sentence having the highest text rank as the summary data, and storing page summary data of the page information database.
delete delete delete delete delete In the method for providing an e-book service by an e-book service providing device,
Receiving a user voice signal and outputting the converted text through a speaker or outputting the text through a display;
When receiving a user voice signal, searching a voice recognition database and determining whether the user voice signal corresponds to a voice command or a search word;
Accessing a page corresponding to the voice command when the user voice signal corresponds to a voice command list;
If the user voice signal does not correspond to the voice command list, searching for and accessing a page including a search word corresponding to the user voice signal;
When the page contains an image, a feature vector is generated by inputting the image to a convolutional neural network (CNN), and the feature vector is input to a recurrent neural network (RNN)/long-short term memory (LSTM) network. Generating an image caption;
Determining whether summary data corresponding to the page exists in page information data, and extracting text of a page if the summary data does not exist;
The extracted text is separated into sentences, and natural language processing is performed to generate a TF-IDF model, a correlation matrix according to the TF-IDF model is generated, a weight graph for each row and column is calculated, and a weight graph Calculating a text rank for each sentence according to Equation 1 below;
[Equation 1]

sentence

Text rank for,
rij: weight between sentences or words i and j,
d: Probability of a person surfing the web at page rank not satisfied with the page and navigate to another page,
Calculating a text rank for each sentence included in the text of the page, and generating a sentence having the highest text rank as the summary data; And
And storing the page summary data of the page information database.
delete delete delete delete delete A computer-readable recording medium in which a computer program for executing the method of providing an e-book service of claim 7 is stored in a computer.

Images