TW202009891A - E-book apparatus with audible narration and method using the same - Google Patents

Abstract

The invention relates to an automatic reading apparatus that receives a multi-media content including text content. The reading apparatus includes a display having a display region for the multi-media content; an input interface receiving an input signal associated with a location identification and/or a change of the multi-media content in the display region; and a reading and highlight unit generating audio content as well as one or more dynamic highlights associated with the text content, said dynamic highlight skip from a first portion of said text content to a second portion of said text content in response to the input signal.

Description

Electronic book voice reading device and method

The invention relates to an electronic book reading device and a method thereof, in particular to an electronic book reading device capable of voice reading and dynamically marking text content.

E-books have been developed for many years. Common e-book formats include PDF, EPUB, mobi, AZW, etc. According to the existing technology, the picture content and text content contained in the e-book can be completely visually presented, but the reading function of the e-book is relatively slow to develop, especially the automatic reading function for machine learning. The reason for this is that the difficulty of reading aloud by the machine is quite high, and it is necessary to overcome the monotonous machine pronunciation and the analysis of contextual semantics to read aloud smoothly. For example, the text contents "3/4 Opening Ceremony" and "3/4 Scope of Influence", although both contain "3/4", the former is read aloud as March 4th, and the latter is read aloud as four Thirds. However, these problems have been overcome with the development of AI technology, and the reading function of e-books will gradually become popular in the future.

Existing e-book reading devices can open the reading function, and some are accompanied by highlights of the text content to guide readers to read, so that readers can more easily enter the reading state through the cooperation of the text marking and the reading. However, the reading and marking functions of the existing e-books are only performed monotonously according to the sequence of the text content, and the targets of reading and marking are not allowed to be arbitrarily selected.

Accordingly, it is necessary to develop a reading device or method that allows the target of reading to be selectively changed according to the user's operation, and the marking of the text is also synchronized.

An object of the present invention is to provide an automatic reading device configured to receive and display a multimedia content, the multimedia content including at least text content, the reading device includes: a display with a display area to display a part of the multimedia content; an input Interface, receiving an input signal related to a position recognition in the display area and/or to the change of the portion of the multimedia content in the display area; and a reading and marking unit configured to generate Sound content and one or more dynamic tags associated with the text content, the dynamic tags jumping from a first part of the text content to a second part of the text content in response to the input signal, wherein the The first part of the text content is related to a first sound content, and the second part of the text content is related to the input signal and appears in the display area.

In a specific embodiment, the dynamic mark has a sentence mark. The dynamic mark has a single word mark. Or, the dynamic mark has a sentence mark and a word mark, and the sentence mark and the word mark overlap visually and distinguishably. The range of the sentence mark is defined by the two punctuation marks of the text content.

In a specific embodiment, the second part of the text content is related to a second sound content.

The present invention also provides a non-transitory computer readable medium, including a plurality of instructions, which can be executed by a processing unit to: analyze the text content contained in a multimedia content to identify a plurality of sentences and/or words; receive an input signal, the The input signal is related to a position recognition in a display area and/or to a change of a part of the multimedia content in the display area; generating one or more dynamic marks associated with the text content in response to the input signal, The dynamic mark is visible in the display area; and the dynamic mark jumps from a first part of the text content to a second part of the text content, wherein the second part of the text content and the The input signal is related and appears in the display area.

In a specific embodiment, the instructions are further executed: generating a corresponding sound content based on a recognition of the sentence and/or word of the text content, and the output of the sound content is synchronized with the dynamic mark.

In a specific embodiment, the generating one or more dynamic tags associated with the text content includes canceling an original dynamic tag.

In a specific embodiment, the change of a part of the multimedia content in the display area includes a scrolling operation or a page turning operation about the display area.

An automatic reading method is executed by a processing unit of an arithmetic device. The method includes: acquiring and displaying a part of a multimedia content on a display area of a display, wherein the multimedia content has text content; starting a mechanical reading method based on The text content outputs sound content; one or more dynamic marks are generated in the display area, the dynamic mark indicates a sentence and/or a word of the text content, the text content indicated by the dynamic mark and the Synchronization of text content associated with audio content; and receiving an input signal related to a position recognition in the display area and/or related to the change of the portion of the multimedia content in the display area, the dynamic mark The display and output of sound content jump from a first part of the text content to a second part in response to the input signal, wherein the first part of the text content is related to a first sound content, the The two parts are related to the input signal and a second sound content and appear in the display area.

In a specific embodiment, the generating one or more dynamic marks in the display area includes simultaneously generating a first dynamic mark indicating a sentence and a second dynamic mark indicating a word, the first dynamic mark and The second dynamic marker can visually overlap.

In a specific embodiment, the input signal is associated with a touch interface operation, an image recognition result, or a voice recognition result.

In a specific embodiment, the display of the dynamic mark jumps from the first part to the second part of the text content in response to the input signal, and the display containing the dynamic mark is displayed from the first part of the first part of the text content One sentence jumps to the second sentence of the second part of one of the text content.

In the following detailed description of a number of example specific embodiments, reference is made to the accompanying drawings, which form part of the present invention. It is shown by way of example description, and the described specific embodiments can be implemented by this example. Provide sufficient details to enable those skilled in the art to implement the described specific embodiments, and understand that other specific embodiments can be used and other changes can be made without departing from the spirit or scope thereof. In addition, although this may be the case, the reference to "a specific embodiment" does not need to belong to the same or singular specific embodiment. Therefore, the following detailed description does not have a limiting idea, and the scope of the specific embodiments of the description is only defined by the scope of the additional patent applications.

In the scope of the entire application and patent application, unless otherwise clearly stated in the context, the following word systems have a meaning clearly associated with this. When used herein, unless expressly stated otherwise, the term "or" is an inclusive usage of "or" and is equivalent to the term "and/or". Unless expressly stated otherwise in the context, the term "based on" is not exclusive and allows the basis of most other factors not stated. In addition, in the overall application, the meanings of "one", "one" and "the" include plural references. The meaning of "in" includes "in" and "on".

As used herein, the term "network connection" means a collection of links and/or software components that enables a computing device to communicate with another computing device through a network. One such network connection may be a transmission control protocol (TCP) connection. The transmission control protocol connection is a virtual connection between two network nodes, and is generally established through a transmission control protocol handshake communication protocol.

The following provides a brief summary of these innovative themes to provide a basic understanding of some aspects. This short narrative is not intended as a complete overview. This short description is not intended to identify major or key components, or to depict or limit the scope. Its purpose is only to present some concepts in a brief form as a prelude to the more detailed narrative presented later.

The first figure shows a system (100) provided by the present invention, which includes one or more servers (102) and multiple user terminal devices (104) connected to the network thereof, also known as user devices or user devices, In particular, the terminal device (104) is suitable as an e-book reader. For the user device (104), the server (102) is a remote server. The server (102) can be programmed to create a website or other forms that can be accessed by the browsing software of the user device, so that the user can download the data allowed by the server, such as an application, through the network (106) , Multimedia content data, software update data, etc. The server (102) may be further configured to perform specific operations and provide the results of the operations to the user device (104) via a network connection. In some embodiments, the server (102) may provide a download link for an e-book website and e-book reading software. The user can access the e-book through a given server, and the server's access can be restricted, such as limiting the number of people who access or data traffic.

Generally speaking, the server (102) includes one or more central processors (1020) and a memory (1022) for storing a plurality of operation instructions executable by the processor. Network interface (1024) The network is connected to one or more networks (106) and user terminal devices (104) to receive data, requests and commands from the network and send various forms of data to the user device, for example The multimedia content data stored in the digital storage unit (1026) includes picture data, text data and sound data. The processor (1020) can receive information and instructions from other computer systems or services via the network (106). For example, the processor (1020) may utilize a network interface (1024) to receive or provide various content items for e-book presentation. The processor (1020) may further utilize the network interface (1024) to receive or transmit synchronized information about content items. The processor (1020) can communicate with the memory (1022) to access various operation instructions therein. In some embodiments, the role played by the central processor (1020) and the operations it performs may be shared or replaced by the processor inherent in the user device (104).

The memory (1024) includes a plurality of computer-executable instructions that can be executed by the central processor (1020) to implement various operations disclosed in the present invention. Memory (1024) can include any combination of transient or non-transitory memory, including RAM, ROM, hard disk, solid state drive, flash memory, etc. The memory (1024) can store an operating system, which provides multiple computer program instructions to be used by the processor (1020) for general content management and operation. In other embodiments, the digital data storage unit (1026) may be included in the configuration of the memory (1024) and store various content items for presentation on the user device, such as picture content, text content, and sound content. The digital data storage unit (1026) may contain other information about the content item, such as synchronization mapping information about the content item, metadata of the content item, etc. In other embodiments, the server (102) may be network connected to another external digital data storage unit (not shown) to obtain content items for presentation on the user device or stored in memory (1024).

The user terminal device or user device (104) may be a personal computer, a tablet computer, a digital assistant, a mobile device, or any suitable form. In this embodiment, the user device (104) includes a display, an input unit (such as a physical keyboard, a touch screen, a mouse, a microphone, or an imaging unit), a processing unit, a memory, and other devices used to execute the present invention. The configuration of the embodiment.

The second figure shows an embodiment (200) of the user device (104) of the first figure, which includes a processing unit (202), a computer readable medium (210), a network interface (220), and a memory (230), an output/input interface (240) and a reading and marking unit (250). Similarly, the processing unit (202) can receive various information, commands, and multimedia content from the network. The processing unit (202) can utilize the web interface (220) to receive various content items for presenting the e-book. The processing unit (202) further uses the network interface (220) to transmit or receive other information or instructions for performing various embodiments of the present invention, such as synchronous mapping information about multiple content items. The processing unit (202) can access the computer-executable programs contained in the memory (230), such as the operating system (231), content playback module (232), and content data (233) of the client device, and input/ The output interface (240) outputs to the output unit (260), which may include one or more output devices for presenting various content items to the user, such as a display and a speaker. The input/output interface (240) can receive input from an input unit (270). The input unit (270) can include one or more input devices, such as a touch screen (combination of output device and input device), a mouse, Microphone and imaging device. In terms of the touch screen, a touch event generates a corresponding input signal, and the processing unit (202) can determine one or more coordinates about the touch event according to the input signal, and further according to the coordinates of the touch event The analysis processing unit (202) can recognize one or more pixels of the display and corresponding touch behaviors. According to this, the processing unit (202) can determine the relevant pixel output according to the coordinates.

The memory (230) may include any combination of transient and non-transient, such as RAM, ROM, hard disk, solid state drive, and flash memory. The operating system (231) provides computer programming instructions for general management and operation of the user device, which varies according to the type of the user device and is well known in the art, so it will not be repeated here. The content playback module (232) may be configured to perform presentations on various content items and provide a user interactive interface for controlling content playback. The content data (233) contains one or more content items, such as text content, picture content, and sound content, which can be played through the content playing module (232). The content data (233) may further include other information related to each content item, such as synchronization mapping information between two different content items, and metadata of each content item. The content data (233) can be updated according to other content data received from the network interface (220) or the output/input interface (240). The user device (200) can obtain other external content items and store them in the content material (233) to realize real-time streaming playback or play at any time. For e-books, the content playback module (232) can process text content, picture content, and sound content. The content playback module (232) can also provide user operation elements, such as page turning buttons or scrolling components.

The reading and marking unit (250) is configured to perform voice reading and marking actions corresponding to the text content. In other embodiments, the reading and marking unit (250) may be split into a reading unit and a marking unit that are independent of each other. In some embodiments, the reading and marking unit (250) may be part of the content playback module (232) or a related extension. Alternatively, part of the work of the reading and marking unit (250) may be performed on the server (100) side. The third figure shows an embodiment (300) of the reading and marking unit of the present invention, including a text generation engine (301), a word processing engine (302), a semantic analysis engine (303), an audio matching engine (304), and a text marking engine (305) and synchronization generation engine (306).

The text generation engine (301) is configured to recognize text content from one or more content items stored in the server (100) or content data (233) and generate visual text and punctuation marks that can be displayed, and according to the content items Other information in determines the visual effects of text layout, fonts, and fonts. The typesetting may include visual presentation of pictures and text. The content items can be defined by various e-book specific formats, such as PDF, EPUB and AZW. The visual text may be covered in a single page of the e-book or a space of multiple pages respectively. In other possible embodiments of the present invention, as in an audio book application, the text generation engine (301) may be configured to use known speech recognition means to generate corresponding text content based on the received sound content.

The word processing engine (302) is configured to recognize one or more sentences in the text content. For example, according to known rules, a sentence may be a text between two adjacent periods, or any two adjacent punctuation marks (comma and period). The composition of one or more words covered by the parentheses can also be regarded as sentence recognition. In other embodiments, the sentence recognition can be further optimized according to the technology based on machine learning, which can solve the unrecognition that may be caused by punctuation error. In one embodiment, the text content recognized as a sentence may be given an identifier or label and stored in the memory together with the corresponding text content, that is, each sentence has its own identifier or label. For example, specific identifiers can be given to these sentences so that the sentences can be identified and the relationship between the sentences and the sentences can be clearly defined, such as the sequential relationship between the sentences and the sentences, and the number of paragraphs or lines in the sentences.

The semantic analysis engine (303) is configured to determine the semantic features associated with the one or more sentences based on the recognized one or more sentences, which may be stored in memory along with the text content and the identifier or tag. The semantic features mentioned here refer to the statistics or measurement related to the grammar, semantic meaning and/or word composition of the sentence. In one embodiment, the semantic analysis engine (303) may divide the text of each sentence into multiple segments and determine the corresponding semantic features for each segment. The semantic analysis engine (303) can be implemented by known machine learning means (such as ??), and the establishment of the semantic analysis engine (303) can be completed on a remote server and downloaded to the user terminal device. Alternatively, the semantic analysis engine (303) may not be executed in the user device, but executed in the remote server and store the analysis result in the remote server. The semantic analysis engine (303) can continuously optimize the accuracy of semantic analysis through continuous training feedback, and even detect errors in sentences.

The audio matching engine (304) is configured to recognize one or more sound contents corresponding to a sentence according to the semantic features associated with the sentence, thereby completing the matching of text content and sound content. The sound content may be composed of one or more files and may be converted into sound signals and output through a speaker. In one embodiment, the audio matching engine (304) can access an audio sample database (not shown), which can store candidate sound contents corresponding to various words. In an embodiment, in the audio sample database, each sound content item corresponding to a word or word may be associated with one or more semantic features, and the matching is based at least on words, words and/or sentences Identification of the semantic features associated with the sound content and the semantic features of the sound content. The matching associates the textual content (words, words, sentences) with one or more sound content. The audio matching engine (304) can be implemented by known means, such as an automatic reading application. In other embodiments, if there is an audio book application, the voice content synthesized by the audio matching engine can be read by pre-recorded vocals, which means that the vocal content of the vocal reading can be processed and associated with the corresponding text content for playback.

The text tagging engine (305) is configured to generate one or more dynamic tags based on a start signal or an input signal in sentences and/or text associated with the start signal or input signal. The dynamic mark can be visually presented to the user via the display. There are any possible forms of dynamic markup sentences, such as fluorescent marks on the text, underline below the text, color/type/font of the text, etc. The dynamic here refers to the action that the mark will appear and jump between sentences and/or words as the reading goal advances during automatic reading. When the automatic reading stops, the mark will stop at the text or disappear. The start signal indicates the start of an automatic reading. The text marking engine (305) marks the first sentence or first word recognized in the text content in response to the starting signal. Alternatively, the start signal may further indicate the start of the automatic reading operation after the pause. The jumping frequency of the mark is related to the length of the sentence or word and the speed of automatic reading. The input signal is generated by the input unit (270), where the input signal indicates position information on a display area of the display. In one embodiment, the input signal is generated based on the recognition of a coordinate or a set of coordinates in a display area of the display (such as the user tapping the touch screen), wherein the coordinate is associated with one or more Pixel location. In another embodiment, the input signal is generated based on the selection of a part of the multimedia content (eg, the user clicks on a part of the display content in the display area). Alternatively, the input signal indicates location information of multimedia content that is not displayed (eg, the user clicks on Chapter 3 on the displayed directory). The text tagging engine (305) can associate a tag to undisplayed text content in response to the input signal. It is worth noting that although the multimedia content is not displayed, the location information of each content item in the multimedia content can be determined according to the applied typographic rules (for example, the seventh sentence of the third paragraph of the text content is on the first page of the ninth page) To five lines). The foregoing content playback module (232) can provide an input field allowing input of navigation information of the e-book, such as chapters, pages, and lines.

Taking an e-book as an example, the fourth diagram illustrates a display screen (400) of a display, in which a display area (401) displays a part of text content, while other parts are not displayed or covered by a window (402). Alternatively, the undisplayed content may appear through a scrolling operation or a page turning operation. For example, the input signal corresponding to the scrolling operation (403) indicates that a set of coordinates on the display area (401) changes along a longitudinal direction, and accordingly the undisplayed text content can be above the display area (401) Or the loading screen below. The input signal corresponding to the page turning operation (404) indicates that a set of coordinates on the display area (401) changes along a horizontal direction, according to which the undisplayed content can be loaded into the screen by the left and right sides of the display area (401) . The dynamic mark may not respond to the loading of the undisplayed content, or the dynamic mark may remain in the display area (401) in response to the loading of the undisplayed content. A selection operation can be associated with the text content in the display area (401). As shown in the figure, the input signal corresponding to a first selection operation (405) indicates a mark or a set of marks corresponding to the text content "I know", and the text marking engine can accordingly mark the sentence or word corresponding to the text content . An input signal corresponding to a second selection operation (406) indicates a mark or a set of marks corresponding to a page margin, and the text marking engine can mark the sentence or the text content closest to the margin according to the margin Word. An input signal corresponding to a third selection operation (407) indicates a mark or a set of marks corresponding to the connection of two sentences, and the text marking engine can select and mark one of the two sentences accordingly.

Various forms of input units can implement the aforementioned operations and generate corresponding input signals. The touch screen is a universal input unit and can also provide intuitive operation. Alternatively, the input unit may be an imaging device used to capture the reader's eyeball image or gesture image, and cooperate with the image recognition to generate an input signal corresponding to the operation. Known image recognition techniques can determine the position or saccade of the reader's eye in the display area (401) to identify the aforementioned operation. For example, when the eyeball is staring at a position in the display area (401) for a period of time or with an eye-catching motion, the selection operation can be recognized. When the reader moves away from the display and gives a waving gesture or pointing gesture, the page turning operation or the selection operation can be recognized. Alternatively, the input unit may be a microphone for capturing human voice. The known speech recognition technology can judge the keywords given by the reader and correspondingly generate the input signal of the associated selection operation. Further, in conjunction with the search technology of the known search technology, the text tagging engine can tag all selected keywords in the text content. The selectivity of these input units is friendly to people with disabilities, on the other hand, it is also helpful in the field of teaching, not just limited to the use of known e-books.

Optionally, one or more dynamic marks can be displayed in the display area. The fifth figures A to B show schematic diagrams of various embodiments of the present invention. Figure 5A shows the dynamic tagging for a single sentence (501). In the embodiment of the scrolling animation screen, the dynamic mark (501) can be maintained at a horizontal height or a range of the display area during the automatic reading, so the screen is dynamically scrolled automatically as the automatic reading progresses. In the embodiment of the page turning screen, when the dynamic mark appears at the bottom content of the current page, then the loading of the next screen will dynamically mark the top content of the screen. Figure 5B shows the dynamic markup (502) for a single text. However, when encountering lengthy sentences or reading too fast, the simple use of sentence markers (501) or word markers (502) has its disadvantages. Therefore, combining the two can make up for each other's shortcomings. As shown in the fifth image C, two kinds of marks are displayed at the same time, in which the single-word mark (502) is included in the single-sentence mark (501), and the two can give appropriate visual distinction, such as the treatment of color or transparency. The fifth image D further shows the paragraph mark (503), which is suitable for the reduced content based on the paragraph.

Referring back to the third figure, the synchronization generation engine (306) is configured to synchronize the recognized sound content with the corresponding one or more dynamic tags. The synchronization generation engine (306) can output the sound content about a sentence or a word and the dynamic mark to the output unit (270), such as a speaker and a display, according to the stored synchronization information or identification information (that is, identifier or label). In an embodiment, the synchronization generation engine (306) may use a known identifier or label to associate a part of the text content and its corresponding dynamic tag with a part of the sound content, for example, through a known linking method , Where the identifier or label is used to identify a sentence or word in the text content. In some embodiments, the synchronization generation engine (306) may continuously execute the linking method until all the text content and sound content are synchronized. The execution of the synchronization generation engine (306) can be completed on the remote server and the synchronization result is stored in the cloud, which can be downloaded to the user device along with the multimedia content. Further, the synchronization generation engine (306) may determine the display time of the synchronized dynamic mark based on the playing time of the sound content item, and record it in the synchronization result.

The sixth figure shows a flowchart of interaction between the user and the automatic reading device (such as the user device 104 in the first figure), including steps S600 to S640. In step S600, the user turns on the e-book reader installed on the user device and opens the e-book file through the reader. The e-book reader can be downloaded from a website or link provided by a remote server, or can be built into a user device. The reader may include a control interface to allow the user to selectively navigate the content of the article. The reader can also contain options for other additional functions, such as automatic reading and the aid of text marking. The reader can provide a reading window displayed on a display area of the display, which displays a part of the e-book content, including text content and picture content, and even clickable links. The navigation mode of the reader can be scrolling navigation mode or page turning navigation mode according to the user setting or the type of e-book file loaded. After the reader is turned on, part of the content of the article appears in the display area, and step S600 ends

In step S610, the user activates the automatic reading and marking functions through the control interface of the reader (such as virtual or physical keys). In one embodiment, the user can click on the virtual button in the fourth window (402) to call the display of a selection interface. Alternatively, the user can pull a selection interface listing a plurality of selectable functions in a sliding gesture from an edge of the display area. The reading and marking functions can be independent. Preferably, when the two are determined to be active, the results of reading aloud and marking should be synchronized. When the reading and marking functions are active, after the user presses the play button, the audio signal and the dynamic marking of the reading are received by the user at the same time. The dynamic mark appears in the display area in units of words, words, or sentences, and the text associated with the dynamic mark completely or partially matches the text associated with the sound signal. Dynamic marks and audio signals are generated at an appropriate speed and in a predetermined order of text content. As shown in the display picture illustrated in the fourth figure, the dynamic mark and the reading voice signal are automatically synchronized from the first sentence or the first word in the first line of the display area to the last sentence or the last word in the display area. In the scrolling navigation mode, when reading to the last sentence or word in the display area, the reader can automatically scroll the animated page to replace the original part of the undisplayed text content, and continue reading and marking from the updated part. In the page navigation mode, when reading to the last sentence or word in the display area, the reader can automatically load the undisplayed text content to replace the original text, and continue reading and marking from the updated part. Unless the user instructs the reader to stop the action, the voice signal and the dynamic mark read aloud will continue to play in the order of the article until the end of the article, and step S610 ends.

In step S620, the reader determines whether the user has an instruction to skip or skip part of the article content. The skipping or skipping refers to a new tag and reading target content in the article specified by the user, which is not included in the currently marked and/or reading target content. The jump or skip action may represent that the user wishes to switch from the current reading, marking and/or displaying target content to another unreading, marking and/or non-displaying target content, ie the user wishes to change the current reading and reading /Or marked text content. For example, a user can decide and change aloud and dynamically marked sentences via a reader. The reader continuously detects whether there is any input signal about the user's instruction to skip the article, and ends step S620.

If the reader does not receive the relevant instructions, the reader will read and mark the next sentence or word in sequence according to the predetermined article sequence until the end of the article (step S630).

In step S640, the user instructs the reader to switch the target content of automatic reading and dynamic tagging. For example, the user can find the target content through the navigation and select a starting position for automatic reading and dynamic marking. In a feasible embodiment, during the navigation process (whether scrolling or turning pages), the actions of automatic reading and dynamic marking can automatically recognize a starting position as the text content in the display area changes. For example, when the process of scrolling the animation surface causes the text in the current reading and marking to disappear in the display area, the reader can be configured to automatically recognize a position in the current display area, which allows new reading and marking to continue from then on. In other embodiments, the actions of reading and marking will not change with the change of the display area, which means that even if the process of scrolling the animation surface causes the current reading and the text in the mark to disappear in the display area, the position and order of reading Unchanged. This applies to users who only navigate, but do not intend to change the current reading and marking goals.

The seventh figure shows the flow of steps for dynamically marking text content of the present invention, including steps S700 to S730. These steps can be implemented by multiple executable instructions stored in one or more memories (such as the first image server or user device). The execution of these steps may be performed on the user terminal device, or part of these steps may be performed on the remote server, or part of these steps may be performed on both the terminal device and the remote server.

In step S700, an instruction may be configured to perform analysis of a multimedia content (such as an e-book file) to recognize a plurality of sentences and/or words of the text content included in the multimedia content. The recognition can be achieved based on known semantic recognition and machine learning. In one embodiment, the recognition of the sentence is based on the correlation between punctuation marks. In other embodiments, the identification may include attempting to cut out a series of text content for analysis. In some embodiments, the recognition may include analyzing the semantics of a series of text content to determine the scope of a sentence. Recognized sentences, words or paragraphs can be given corresponding recognition information, such as identifiers or tags, which can specifically indicate the contextual relationship or position of the sentence in an article. The identification information can be stored and transmitted along with the multimedia content, and step S700 ends.

In step S705, an instruction is configured to perform sentence and/or word recognition based on the aforementioned text content to generate corresponding sound content, which can be achieved by known mechanical reading methods. Known means can output or synthesize corresponding sound content for each word, word and sentence of the text content. These sound contents can be generated and stored in an appropriate memory before reading aloud, or these sound contents can be generated and output in a timely manner during the execution of the reading aloud.

In step S710, an instruction is configured to cause a processing unit to process an input signal, in particular the input signal and a position recognition in a display area of a display and/or changes in a portion of the multimedia content in the display area related. The input signal is received by the output/input interface (240) of the client device as shown in the second figure. Different input units (260) can communicate with the output/input interface (240), such as touch panels, optical lenses, or microphones. The input signals generated by these input units are as described above. As shown in the fourth figure, the user can generate a navigation operation and a selection operation relative to the display area through these input units, where the navigation operation (403, 404) will cause a part of the multimedia content to change in the display area ( Such as scrolling, page turning, switching), and the selection operation (405, 406, 407) is to cause a position selection in the display area. In one operation, the user first performs a navigation operation to find what he wants to read, and then performs a selection operation to decide a reading item, such as a sentence or a word. According to this, the processing unit obtains at least one or more pieces of position information in the display area, and step S710 ends.

In step S720, an instruction is configured to cause the processing unit (or marking unit) to generate one or more dynamic marks associated with the text content, the dynamic marks being visible in the display area. In an embodiment, the processing unit automatically generates the text of the dynamic mark in the display area. In another embodiment, according to the position recognition information in the display area obtained above, the processing unit finds the sentence or word corresponding to the position recognition and generates the dynamic mark in the corresponding range in the display area. As shown in the fourth figure, regardless of whether the position associated with the user's selection operation (405, 406, 407) directly points to a sentence or word of the article, the sentence most relevant to that position should be recognized and marked with priority. In an embodiment, the processing unit may further generate the text content where the dynamic mark finally stays in the display area according to the navigation operation. For example, according to a page turning operation, a new dynamic mark may be generated in the first sentence of a new page. One or more visible dynamic markers can appear in the display area. Figures 5A to 5D show single sentence marks, single word marks and their combinations.

In step S730, the processing unit causes the dynamic mark to jump from a first part of the text content to a second part of the text content in response to the input signal, wherein the second part of the text content appears on the display In the area. In a possible situation, a first sentence in the current display area has seen a dynamic mark, and after the user selects a second sentence in the current display area (that is, an input signal is generated), the original first sentence's The dynamic marker jumps to the second sentence selected by the user. The jump described herein does not refer to a specific jump action, but should be understood as a visual effect similar to jump or switching. The visual effect of the jump can be seen that the dynamic mark ignores the stay of other sentences between the first sentence and the second sentence. In another possible situation, a first sentence with a dynamic mark disappears from the current display area due to the navigation operation, and the user selects a second sentence in the current display area (that is, generates an input signal), The dynamic marker returns to the display area and marks the second sentence. Although the dynamic mark may disappear from the display area due to the operation, it may be assumed that the dynamic mark jumps from the undisplayed portion to the content being displayed. Step S730 may be performed simultaneously with step S720 or be part of step S720.

The eighth figure shows the step flow of the reading method of the present invention, including steps S800 to S830. These steps can be performed individually or jointly by one or more computing devices (such as the first image server 102 and the user device 104). For example, when the user device communicates with the server, some of these steps can be performed by both. Alternatively, in the free state, the user device can independently perform these steps.

In step S800, a multimedia content is obtained through a remote server or a user device, and a part of the multimedia content is displayed on the display area of a display. The multimedia content may be a combination of various content items, such as text content, picture content, audio content, and video content, which may also be integrated in the streaming content and transmitted from the remote server to the user device. The display may be included in the user device or an external device independent of the user device and communicatively connected thereto. The display area (such as the fourth image, 401) displays a part of multimedia content. For e-books, the display area is mainly based on text content, and pictures or advertisement boards can be interspersed between texts. Step S800 is ended.

In step S810, a mechanical reading method is initiated through the user device to output an audio signal based on the text content. The user device can be configured to have the ability to read text aloud. For example, the user device may include an e-book reader, an audio database, a semantic recognition engine or module, and a speaker. The sound data library contains sound data corresponding to each word, word or sentence. These data can be matched with the results of semantic recognition to output corresponding sound signals. The audio signal described in this article can be in digital or analog form, and is not limited to the audible signal at the circuit transmission stage or the final output. In the case that the user does not specify, the mechanical reading can start from anywhere in the text content, such as the first word of the text content, or the first word of the current text content in the display area, or the position where the previous reading ends. The reading continues until the end of the article or the user stops actively, and step S810 ends.

In step S820, one or more dynamic marks are generated in the display area via the user device, the dynamic marks indicate a sentence and/or a word that the reader wishes to watch. The text content indicated by the dynamic mark is synchronized with the text content associated with the sound signal. Synchronization described in this article means that the scope of dynamic mark generation is the same as the currently spoken word or the sentence to which the currently spoken word belongs. It is not limited to the same time when the relevant signal occurs. In an embodiment, the dynamic mark is a sentence mark, which appears at a position where a sentence is displayed so that the sentence can be visually distinguished from other words (as shown in Fig. 5A). In another embodiment, the dynamic mark is a word mark, which appears at a position where a word is displayed so that the word can be visually distinguished from other characters (as shown in the fifth image B). The word mark can actually jump to the next word following the speed of reading aloud. In other embodiments, the dynamic mark is a combination of a sentence mark and a word mark, which appears at the same time as the position of a sentence and the position of a word in the sentence, and the two are visually distinguishable (as shown in Figure 5C) ). For example, the sentence mark and the word mark may have different colors, or one of them may be a text underline. The dynamic mark will continue to jump towards the end of the article until the reading stops, and step S830 is ended. Preferably, step S810 and step S820 are performed together. In step S830, before or after the reading and the dynamic mark have not been stopped, an input signal is received through the user device, the input signal is identified with a position in the display area and/or the portion of the multimedia content is displayed on the display The changes in the area are related. The user device may include or communicatively connect an input unit, such as a touch panel, optical lens, or microphone, which allows the user to operate to indicate position information on the display area and allows the user to navigate all multimedia content in the display area. The location information includes a landmark or a collection of landmarks on the display area, which can instruct one or more user operations, such as the aforementioned selection operation or navigation operation. The display of the dynamic mark and the output of the audio signal jump from a first part of the text content to a second part in response to the input signal, wherein the second part of the text content appears in the display area. When a selection operation is recognized by the user device, a sentence or a word (most related to the location information) is then recognized in response to the selection operation. The sentence or list recognized based on the input signal becomes a new target for reading and marking, and is immediately read and marked. The target of dynamic marking jumps from the original sentence (first sentence) to the recognized sentence (second sentence). The first sentence and the second sentence are different sentences, and are not limited to the first sentence before the second sentence. If the input signal indicating a navigation operation causes the content of the display screen to change, the dynamic mark may change the position that appears on the display screen or disappears with the change. In an embodiment, when the dynamic mark disappears, the user device may generate a new dynamic mark in the changed display area to mark the current content, and simultaneously read the target and synchronize to the new target content. In some embodiments, such as an e-book directory containing chapter links or a shortcut key to return to the home page, an input signal indicating a selection operation or a navigation operation may cause the user device to direct the reader to the page associated with the selected chapter link, while The reading and marking of the target are also synchronized to this page. Based on the input signal, the dynamically marked target and/or the reading target jump to the new target content, and step S830 ends.

To understand the step diagrams and combinations in such flow charts, it can be implemented as computer program instructions. These program instructions can be provided to a processor to make a machine, so when the instructions are executed on the processor, a method for implementing the actions specified in the flowchart block or blocks is generated. The computer program instructions can be executed by a processor to perform a series of operation steps by the processor to form a computer-implemented program, so the instructions are executed on the processor to provide for implementation in the flowchart The step of the action specified in the block or multiple blocks. These program instructions can be stored on a computer-readable medium or machine-readable medium, such as on a computer-readable storage medium.

Accordingly, the descriptions support a combination of means to perform the specific actions, a combination of the majority supporting the execution of the specific actions, and a program instruction method that supports the execution of the specific actions. It will also be understood that each block depicted in the flowchart and the combination of blocks depicted in the flowchart can be implemented by a module, like a system based on special purpose hardware that performs these specific actions Steps, or a combination of special purpose hardware and computer instructions.

The foregoing provides a complete description of the manufacture and use of combinations of these specific embodiments. Since many specific embodiments can be produced without departing from the spirit and scope of this description, these specific embodiments will exist within the scope of the patent applications appended below.

100‧‧‧System 102‧‧‧Server 1020‧‧‧CPU 1022‧‧‧Memory 1024‧‧‧Network interface 1026‧‧‧Digital storage unit 104‧‧‧User terminal device, user device 106 ‧‧‧Network 200‧‧‧User device 202‧‧‧Processing unit 210‧‧‧Computer-readable medium 220‧‧‧Network interface 230‧‧‧Memory 231‧‧‧Operating system 232‧‧‧Content Play module 233‧‧‧Content data 240‧‧‧Output/input interface 250‧‧‧Reading and marking unit 260‧‧‧ Output unit 270‧‧‧Input unit 300‧‧‧Reading and marking unit 301‧‧‧Text Generating engine 302‧‧‧Text processing engine 303‧‧‧Semantic analysis engine 304‧‧‧ Audio matching engine 305‧‧‧Text marking engine 306‧‧‧ Synchronous generating engine 400‧‧‧Display screen 401‧‧‧Display area 402 ‧‧‧Window 403‧‧‧scrolling operation 404‧‧‧page turning operation 405‧‧‧first selection operation 406‧‧‧second selection operation 407‧‧‧third selection operation 501‧‧‧dynamic marker 502‧ ‧‧Dynamic Mark 503‧‧‧ Paragraph Mark S600-S640‧‧‧Step S700-S730‧‧‧Step S800-S830‧‧‧Step

The first figure shows a system provided by the present invention.

The second figure shows an embodiment of the user device of the first figure.

The third figure shows an embodiment of the reading and marking unit of the present invention.

The fourth figure illustrates a display screen of a display.

The fifth figures A to B show schematic diagrams of various embodiments of the present invention.

The sixth figure shows the interaction process between the user and the automatic reading device.

The seventh figure shows the flow of steps for dynamically marking text content of the present invention.

The eighth figure shows the steps of the reading method of the present invention.

S600-S640‧‧‧Step

Claims (14)

An automatic reading device is configured to receive and display a multimedia content, the multimedia content includes at least text content, the reading device includes: a display with a display area to display a part of the multimedia content; an input interface to receive an input A signal, the input signal is associated with a position recognition in the display area and/or with the change of the portion of the multimedia content in the display area; and a reading and marking unit configured to generate the text associated with the text Sound content of the content and one or more dynamic tags that jump from a first part of the text content to a second part of the text content in response to the input signal, wherein the first part of the text content Related to a first sound content, the second part of the text content is related to the input signal and appears in the display area. The automatic reading device as described in item 1 of the patent application scope, wherein the dynamic mark has a sentence mark. The automatic reading device as described in item 1 of the patent application scope, wherein the dynamic mark has a single-word mark. The automatic reading device as described in item 1 of the patent application scope, wherein the dynamic mark has a sentence mark and a word mark, and the sentence mark and the word mark overlap visually and distinguishably. The automatic reading device as described in one of items 2 to 4 of the patent application scope, wherein the range of the sentence mark is defined by two punctuation marks of the text content. The automatic reading device as described in item 1 of the patent application scope, wherein the second part of the text content is related to a second sound content. A non-transitory computer readable medium, including a plurality of instructions, which can be executed by a processing unit to: analyze the text content contained in a multimedia content to identify a plurality of sentences and/or words; receive an input signal, the input signal and the A position recognition in a display area and/or related to a change of a part of the multimedia content in the display area; generating one or more dynamic marks associated with the text content in response to the input signal, the dynamic marks Is visible in the display area; and causes the dynamic mark to jump from a first part of the text content to a second part of the text content, wherein the second part of the text content is related to the input signal and Appears in the display area. The non-transitory computer-readable medium as described in item 7 of the patent application scope, in which the instructions are further executed: generating a corresponding sound content based on a recognition of sentences and/or words of the text content, the sound content Is synchronized with the dynamic tag. The non-transitory computer readable medium as described in item 7 of the patent application scope, wherein the generating one or more dynamic marks associated with the text content includes canceling an original dynamic mark. The non-transitory computer readable medium as described in item 7 of the patent application scope, wherein the change of a part of the multimedia content in the display area includes a scrolling operation or a page turning operation on the display area . An automatic reading method is executed by a processing unit of an arithmetic device. The method includes: acquiring and displaying a part of a multimedia content on a display area of a display, wherein the multimedia content has text content; starting a mechanical reading method based on The text content outputs sound content; one or more dynamic marks are generated in the display area, the dynamic mark indicates a sentence and/or a word of the text content, the text content indicated by the dynamic mark and the Synchronization of text content associated with audio content; and receiving an input signal related to a position recognition in the display area and/or related to the change of the portion of the multimedia content in the display area, the dynamic mark The display and output of sound content jump from a first part of the text content to a second part in response to the input signal, wherein the first part of the text content is related to a first sound content, the The two parts are related to the input signal and a second sound content and appear in the display area. The automatic reading method as described in item 12 of the patent application scope, wherein the generating one or more dynamic marks in the display area includes simultaneously generating a first dynamic mark indicating a sentence and a second dynamic indicating a word Mark, the first dynamic mark and the second dynamic mark overlap visually and distinguishably. The automatic reading method described in item 12 of the patent application scope, wherein the input signal is associated with a touch interface operation, an image recognition result, or a voice recognition result. The automatic reading method as described in item 12 of the patent application scope, wherein the display of the dynamic mark jumps from the first part of the text content to the second part in response to the input signal, and the display containing the dynamic mark is read from the The first sentence of the first part of one of the text content jumps to the second sentence of the second part of one of the text content.

Images