JP2013126184A - Content playback device, control method therefor, control program, and content display control device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform output while appropriately associating playback of video and/or audio with playback of a book.SOLUTION: A display control unit 34 controls an AV playback unit 32 to play, from book data, a character string assigned with a character string index corresponding to an elapsed time in playback of AV stream data, or to play the AV stream data from an elapsed time corresponding to a character string index of a character string played from the book data.

Description

  The present invention relates to a content reproduction apparatus that reproduces content including stream data that is at least one of video and audio data and book data, a control method and control program therefor, and a content display control apparatus. In particular, the present invention relates to a content playback device used for media players, electronic books, e-learning, digital signage, and the like.

  In recent years, books that include video learning materials for language learning, video content such as movies and dramas, hobby-related texts and explanation books such as railways and automobiles, and video media such as DVDs (Digital Versatile Disk) have been sold. These books are created on the premise of learning by reading books while viewing video content and browsing and enjoying related information.

  For books to which video content is attached, it is also assumed that the content is created in the form of an electronic book. Some e-book formats can embed video data. Video content stored on a DVD is converted to a format that can be played back by the e-book viewer. Distribution as data is also assumed.

  Also, in the field of e-learning, the lecturer's video and handouts are distributed to the student's PC (Personal Computer) and other terminals via the Internet, and the instructor's web browser or dedicated application runs on the terminal. A system has been developed that displays each page of a handout in synchronization with the explanation.

  In BD-J, which is a standard for interactive content of Blu-ray DISC, a technique for simultaneously displaying information related to moving image content in a picture-in-picture or the like is defined. Patent Document 1 discloses a technique for acquiring information related to a moving image from a server on the Internet and displaying the information in synchronization with reproduction of the moving image.

  By applying these techniques, it is possible to associate and display specific pages of the electronic book at the same time in synchronization with a specific scene in the moving image embedded in the electronic book content. Specifically, it is assumed that the reproduction of the moving image and the display of the book are performed in synchronization with reference to the correspondence table between the time stamp of the video content and the book page.

Japanese Patent Laying-Open No. 2006-33873 (released February 02, 2006)

  However, in an electronic book mainly composed of general character information, it is possible to change the character size and the number of displayed characters per page according to the resolution of the content viewer and user settings. Accordingly, the position of the page break and the total number of pages change. Therefore, the method of associating a video scene with a specific page as in the above-described prior art cannot cope with changes in the page according to the viewer resolution or user settings.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a content playback apparatus and the like that can output video and / or audio playback and book playback in an appropriate manner. is there.

  A content reproduction apparatus according to the present invention is a content reproduction apparatus that reproduces content including stream data, which is at least one of video and audio data, and book data. Content acquisition means for acquiring, information on elapsed time of reproduction of the stream data, and correspondence information acquisition means for acquiring correspondence information in which the character string index assigned to each character string in the book data is associated; Reproduction means for reproducing the stream data and the book data included in the content acquired by the content acquisition means based on the correspondence information, reproducing the stream data and corresponding to the elapsed time of the reproduction The character string assigned the character string index to be Or reproduced from data, or reproduces the book data, and the elapsed time corresponding to the string index string to be reproduced; and a reproducing means for reproducing the stream data.

  A content playback apparatus control method according to the present invention is a content playback apparatus control method for playing back content including stream data, which is at least one of video and audio data, and book data. In order to solve the problem, the content acquisition step for acquiring the content, the information on the elapsed time of reproduction of the stream data, and the correspondence information that associates the character string index assigned to each character string in the book data are acquired. And a reproduction step of reproducing the stream data and the book data included in the content acquired in the content acquisition step based on the correspondence information, wherein the stream data is reproduced. And a character string icon corresponding to the elapsed time of the reproduction. A reproduction step of reproducing the character string to which the index is assigned from the book data, or reproducing the book data and reproducing the stream data from the elapsed time corresponding to the character string index of the character string to be reproduced; It is characterized by including.

  According to the above configuration and method, when stream data and book data included in the acquired content are played back, a character string assigned a character string index corresponding to the elapsed time of playback of the stream data being played back is supported. Played based on information. Alternatively, based on the correspondence information, the stream data is reproduced from the elapsed time corresponding to the character string index of the character string reproduced from the book data.

  Thus, in the present invention, the elapsed time of video and / or audio playback is associated with the character string index assigned to each character string of the book, not the page number of the book. Therefore, by changing the size of characters and the number of displayed characters per page, the correspondence between the elapsed time of reproduction and the character string index does not change even if the total number of pages of the book changes. It is possible to output the audio reproduction and book reproduction in an appropriate manner.

  The correspondence information may be acquired from the outside or may be acquired from an internal storage device. Further, when the correspondence information is included in the acquired content, the correspondence information can be acquired from the content.

  As the character string index, the sentence is divided into at least one of a character unit, a word unit, a sentence unit, a paragraph unit, a section unit, and a chapter unit, or any unit. Information that specifies the combination.

  In the content reproduction apparatus according to the present invention, the book data includes at least one of video and audio media data, and the correspondence information includes information on an elapsed time of reproduction of the stream data and the book data. It is preferable that the character string index assigned for each character string is associated with the media data information. In this case, the book data and the media data can be appropriately reproduced in correspondence with the reproduction of the stream data, or the stream data and the media data can be appropriately reproduced in correspondence with the reproduction of the book data. .

  By the way, there are cases where video and / or audio and a book are to be reproduced in synchronism with each other, and others are desired to be reproduced without being synchronized. Therefore, in the content reproduction apparatus according to the present invention, the reproduction means has a synchronous mode for reproducing the stream data and the book data based on the correspondence information, and an asynchronous mode for reproducing regardless of the correspondence information. It is preferable. In this case, user convenience is improved.

  By the way, when one of the stream data and the book data is reproduced in advance, the other content may be revealed. Therefore, in the content reproduction apparatus according to the present invention, it is preferable that the reproduction unit can limit a reproduction range of one of the stream data and the book data in the asynchronous mode. In this case, it is possible to restrict the reproduction of one of the stream data and the book data in advance, and to prevent the other content from being exposed.

  By the way, in the correspondence information, there may be a case where the reproduction order of video and / or audio does not match the order in which books are written. Therefore, in the content reproduction apparatus according to the present invention, in the asynchronous mode, the reproduction unit reproduces the book data in the order of the character string index in the correspondence information, or is added to the book data in order. It is preferable that it is possible to select whether to reproduce the book data in the order of the character string index. In this case, since it can be reproduced in the order in which the books are written, the convenience for the user is improved.

  Incidentally, the content may include a plurality of at least one of the stream data and the book data. Therefore, in the content reproduction apparatus according to the present invention, it is preferable that the reproduction unit reproduces the selected stream data and book data.

  It should be noted that the content playback device configured as described above that plays back content including stream data that is at least one of video and audio data and book data, and an external display so as to display information played back by the content playback device If it is a content display control apparatus provided with the display control means which controls an apparatus, there can exist an effect similar to the above-mentioned.

  In the content display control device according to the present invention, when the display control unit divides and displays the character string reproduced by the content reproduction device, the display unit displays the number of characters or the number of words of each divided character string. It is preferable. Further, when the character string reproduced by the content reproduction device is divided and displayed, the display control means preferably displays the divided character strings with the same display interval. In these cases, since the character strings are displayed uniformly, the visibility of the user is improved.

  In the content display control device according to the present invention, the display control means adjusts a speed at which the stream data reproduced by the content reproduction device is displayed based on the number of characters or the number of words of the character string reproduced by the content reproduction device. May be. In this case, the user can easily understand the sentence.

  In the content display control device according to the present invention, the correspondence information of the content reproduction device further includes a correspondence between a chapter image of the stream data and a table of contents in the book data, and the display control means includes the correspondence information Based on the information, the chapter image and the table of contents are displayed, and by the selection operation of the chapter image or the table of contents, the playback unit of the content playback device specifies the selected elapsed time, and the specified elapsed time The stream data and the book data may be reproduced. In this case, since the reproduction of the desired content can be started from the table of contents, the convenience for the user is improved.

  The display control means may transmit the stream data and book data reproduced by the content reproduction device to separate display devices for display. In this case, it is preferable that the display control means controls display timing on the separate display devices. Thereby, it is possible to display the video and / or audio and the book in synchronism with different display devices.

  Each unit of the content reproduction apparatus can be executed on a computer by a control program. Furthermore, by storing the control program in a computer-readable recording medium, the control program can be executed on an arbitrary computer.

  As described above, the content reproduction apparatus according to the present invention reproduces a character string to which a character string index corresponding to the elapsed time of reproduction of stream data being reproduced is reproduced based on correspondence information, or book data Since the stream data is played back from the elapsed time corresponding to the character string index of the text played back from, even if the total page of the book changes by changing the character size or the number of display characters per page The correspondence relationship between the elapsed time of the reproduction and the character string index does not change, and as a result, the reproduction of the video and / or audio and the reproduction of the book can be output in an appropriate manner.

It is a block diagram which shows schematic structure of the terminal in the communication system which concerns on this embodiment. It is a block diagram which shows schematic structure of the said communication system. It is a figure which shows the structural example of the display screen of the display in the said terminal. It is a figure which shows the format of the content file which the said terminal acquires. It is a figure which shows an example of the layout information contained in the said content file in a table format. It is a figure which shows an example of the moving image and book correlation table contained in the said content file in a table format. It is a flowchart which shows the flow of the process in the display control part of the said terminal. It is a figure which shows the example of a display before and behind changing the display form of a book. It is a figure which shows an example of the display layout based on the layout information of a table of contents. It is a figure which shows the transition of the said display screen in synchronous mode and asynchronous mode. It is a figure which shows an example of the correspondence of the time index of a moving image, and the page number of a book in a table format. It is a figure which shows an example of the selection screen for a user to select from the several content file memorize | stored in the content memory | storage part of the said terminal. It is a figure which shows the structural example of the display screen of the display in the terminal which is another embodiment of this invention. It is a figure which shows the format of the content file which the said terminal acquires. It is a figure which shows an example of the moving image and book correlation table contained in the said content file in a table format. It is a figure which shows the structural example of the display screen of the display in the terminal which is another embodiment of this invention. It is a figure which shows the format of the content file which the said terminal acquires. It is a figure which shows an example of the 2nd and 3rd moving image and book correlation table added to the said content file in a table format, respectively. It is a figure which shows the structural example of the display screen of the display in the terminal which is another embodiment of this invention. It is a figure which shows the outline | summary of the terminal system which is another embodiment of this invention. It is a block diagram which shows schematic structure of the said portable terminal. It is a block diagram which shows schematic structure of the terminal system which is other embodiment of this invention.

[Embodiment 1]
An embodiment of the present invention will be described with reference to FIGS.

  FIG. 2 is a block diagram showing a schematic configuration of the communication system according to the present embodiment. As illustrated, the communication system 10 has a configuration in which a plurality of terminals 11 and a server 12 are communicably connected via a communication network 13.

  The terminal 11 communicates with the server 12 via the communication network 13, and examples thereof include a PC, a mobile phone, and a PDA (Personal Digital Assistant). The server 12 provides various information to the terminal 11 via the communication network 13. The server 12 has the same configuration as a normal computer. As the communication network 13, a known communication network such as a mobile communication network, a telephone line network, the Internet, or a LAN (Local Area Network) can be used.

  In the present embodiment, the terminal 11 downloads and stores a content file including AV (audio-visual) stream data and book data from the server 12 via the communication network 13. Then, the terminal 11 reproduces the AV stream data and the book data in the accumulated content file, displays and outputs the moving image and the book, and outputs the sound.

  The AV stream data is stream data including moving images (video and audio) reproduced along the time axis. However, the present invention is not limited to this, and may be stream data including one of video and audio. The book data includes a plurality of character string data constituting the text of the book. Note that the book data may further include at least one of image and sound data (media data). In this case, it is also possible to reproduce and output an image and / or sound in the book in synchronization with the text of the book.

  FIG. 1 is a block diagram illustrating a schematic configuration of the terminal 11. As illustrated, the terminal 11 includes a network I / F (interface) 21, a content storage unit 22, a content reproduction unit (content reproduction device, content display control device) 23, a touch panel 24, a display (display device) 25, and a speaker. 26.

  The network I / F 21 converts data to be transmitted to the communication network 13 into a format suitable for the communication network 13, transmits the data to the communication network 13, and processes the data from the communication network 13 in the terminal 11. Is converted to a format suitable for the terminal 11 and transmitted to the block in the terminal 11. Specifically, data modulation / demodulation processing and the like are performed in the network I / F 21. The network I / F 21 may be for wireless communication such as WiFi (Wireless Fidelity (registered trademark)) or 3G (3rd Generation), or may be for wired communication such as Ethernet (registered trademark).

  The content storage unit 22 stores the content file. Examples of the storage unit 22 include flash memory (registered trademark), EEPROM (Electrically Erasable and Programmable ROM), HDD (Hard Disk Drive), and the like. In the present embodiment, the content file received from the server 12 via the communication network 13 is converted into an appropriate format by the network I / F 21 and then stored in the content storage unit 22.

  The content reproduction unit 23 reproduces content (AV stream data and book data) included in the content file read from the content storage unit 22. The content reproduction unit 23 generates image data based on the reproduced content and transmits it to the display 25, and also generates audio data and transmits it to the speaker 26. Details of the content reproduction unit 23 will be described later.

  The touch panel 24 is a position input device that is provided on the display screen of the display 25 and detects a position on the display screen instructed by a user's finger, pen, or the like, and transmits the detected position information to the content reproduction unit 23. . The user can give various instructions to the terminal 11 by operating the touch panel 24. An input device other than the touch panel 24, such as a keyboard and button switches to which various functions are assigned, can also be used.

  The display 25 receives image data from the content reproduction unit 23 and displays an image based on the received image data. Specifically, the display 25 includes a display element such as an LCD (Liquid Crystal Display), an EL (Electroluminescence) display, and a PDP (Plasma Display Panel), and a driver circuit that drives the display element based on received image data. It is the composition provided. The image includes characters in addition to pictures and figures, and also includes still images and moving images.

  The speaker 26 converts audio data from the content reproduction unit 23 into sound waves and outputs the sound waves to the outside. An audio output device other than the speaker 26, such as an earphone or an audio output connector, can also be used.

  FIG. 3 is a diagram illustrating a configuration example of a display screen of the display 25 in the terminal 11 of the present embodiment. In the illustrated example, the display screen 201 includes a moving image area 202 at the top, a book area 203 at the bottom, and an operation panel area 204 at the bottom.

  A moving image (video) is displayed in the moving image area 202 based on the AV stream data included in the content file, and a book is displayed in the book area 203 based on the book data included in the content file. In the present embodiment, the book is reproduced in synchronization with the elapsed time from the beginning related to the reproduction of the moving image. Therefore, the display of the book can be changed in synchronization with the progress of the moving image.

  In the operation panel area 204, various buttons for operating the content and the playback state are displayed. In the example of FIG. 3, a playback position bar 205, a volume bar 206, a target switching button 207, a synchronous / asynchronous switching button 208, a fast reverse button 209, a stop button 210, a pause button 211, a playback button 212, and a fast forward button 213 are provided. It is displayed.

  The reproduction position bar 205 is a slide bar that indicates a relative reproduction position with respect to the entire moving image or book. The slider of the slide bar moves from left to right in synchronization with the elapsed time of the moving image being displayed or the page number of the book being displayed. When the user moves the slider, a moving image of the elapsed time corresponding to the moved position is displayed, or a book having a page number corresponding to the moved position is displayed.

  The volume bar 206 is a slide bar indicating the volume of audio output based on the AV stream data. The volume is adjusted by moving the slide of the slide bar up and down. When audio data is embedded in the book or when the terminal 11 has a text reading function, it is desirable that the volume can be adjusted by switching between audio based on AV stream data and audio based on the book.

  The target switching button 207 is used to switch the operation target in the operation panel area 204 from one of the moving image and the book to the other by a toggle method. In the example of FIG. 3, an icon imitating a book is displayed on the object switching button 207, indicating that the book is an operation target.

  The synchronous / asynchronous switching button 208 is for switching between the synchronous mode and the asynchronous mode in a toggle manner. Thus, in this embodiment, the terminal 11 has a synchronous mode in which a moving image and a book are displayed in synchronization and an asynchronous mode in which the image is displayed asynchronously. In the asynchronous mode, the user can perform operations such as page turning of books regardless of the elapsed time of the moving image. In the example of FIG. 3, an icon indicating separation is displayed on the synchronous / asynchronous switching button 208 to indicate that the mode is asynchronous mode.

  A fast reverse button 209, a stop button 210, a pause button 211, a playback button 212, and a fast forward button 213 are for controlling the playback and display of a moving image. In the synchronous mode, the reproduction and display of moving images are controlled based on the operation of these buttons 209 to 213, and the reproduction and display of books are also controlled. Note that when an operation (trick play operation) such as the fast reverse button 209 and the fast forward button 213 is performed, the terminal 11 may shift from the synchronous mode to the asynchronous mode, and only playback and display of the moving image may be controlled.

  By the way, the display page of the book can be changed by operating the slider of the reproduction position bar 205 when the operation target is set to the book by the object switching button 207. Further, in the area of the touch panel 24 corresponding to the book area 203 of the display screen 201, by performing a gesture operation of sliding the user's finger in the left-right direction, the operation for feeding (turning) to the next page and the previous page are performed. Can be performed.

  It is desirable that these pages can be operated even in the synchronous mode. In this case, the user can temporarily browse the previous page or review the previous page, while returning to the synchronized display state at the page switching timing synchronized with the elapsed time of the video. Become. When a page operation is performed by the gesture operation, the terminal 11 may change from the synchronous mode to the asynchronous mode and change the operation target of the operation panel area 204 to a book. In this case, it is possible to return from the asynchronous mode to the synchronous mode by pressing the synchronous / asynchronous switching button 208. Note that, when no operation is performed for a predetermined time, the asynchronous mode may be returned to the synchronous mode.

  Next, details of the content file will be described. FIG. 4 is a diagram illustrating a format of a content file acquired by the terminal 11. The content file 301 is composed of content in one file format, and includes a file header 302, layout information 303, a moving image / book association table 304, book data 305, and AV stream data 308 as shown in the figure. Yes.

  The file header 302 is header information of the content file, an identifier indicating the type of file, the size of the entire file, information on media such as moving images, audio, and books included in the file, and access to each media data The offset value from the beginning of the file is included. By the way, even if a file is encrypted for copyright protection, it is conceivable that only header information is not encrypted. In this case, it is desirable that the file header 302 includes information related to the encryption method.

  A plurality of media data included in the content file 301 is assigned a media identifier so that each can be distinguished. In the example of FIG. 4, a media identifier “book01” is assigned to the book data 305, and a media identifier “movie01” is assigned to the AV stream data 308. These media identifiers are used in the layout information 303 and the moving image / book association table 304.

  The layout information 303 is information indicating the layout of the display screen. FIG. 5 is a diagram illustrating an example of the layout information 303 in a table format. The layout information 303 of the layout as shown in FIG. 3 is information as shown in (a) of FIG.

  As shown in FIG. 5A, the layout information 303 includes the media type and media identifier for each media included in the content file 301. Furthermore, the layout information 303 includes, for each medium, the x and y coordinates of the upper left corner of the display area, the width and the height, indicating the position and size of the display area of the medium. . The x and y coordinates are coordinates when the upper left corner of the display screen 201 is the origin, the right direction is the x direction, and the lower direction is the y direction.

  A layout in which display areas of a plurality of media overlap, such as when displaying in picture-in-picture format, is also conceivable. Therefore, as shown in FIG. 5B, the layout information 303 preferably includes a priority order indicating which medium is displayed with priority in the overlapping area. In the illustrated example, the higher the number, the higher the priority, so the book has a higher priority than the video. Therefore, in the illustrated example, a book display area exists in the moving picture display area, and the book is displayed in the display area of the book without displaying the moving picture.

  The moving image / book association table 304 is a table for associating the time stamp of the moving image in the AV stream data 308 with the character string index assigned to each character string in the book data 305. In the present embodiment, the content reproduction unit 23 refers to the moving image / book association table 304 to realize synchronous display of moving images and books.

  FIG. 6 is a diagram illustrating an example of the moving image / book association table 304 in a table format. The moving image / book associating table 304 in FIG. 10A is based on the moving image reproduced based on the AV stream data 308 whose media identifier is “movie01” and the book data 305 whose media identifier is “book01”. It associates with the book being played.

  As shown in FIG. 6A, the moving image start position and end position are associated with the book start position and end position for each of the time indexes 0 to 130. The start position and end position of the moving image are indicated by a time stamp indicating the reproduction timing of the moving image, and in the illustrated example, the format is “hour: minute: second”.

  On the other hand, the start position and end position of the book are indicated by the above-described character string index. The character string index includes chapters, sections, and paragraphs that divide the book, and the number of words in each paragraph. In the example of FIG. 6A, each of the chapter, section, and paragraph is indicated by a two-digit number, and the order number is indicated by a five-digit number. , And the order number are separated by “.”.

Note that the character string index may be assigned to each character string in a certain range regardless of the division of chapters, sections, paragraphs, and the like. For example, the range of character strings is delimited by tags in XML (Extensible Markup Language), such as XMDF (ever-eXtending Mobile Document Format), which is one of the formats of electronic books, and a specific character string index is assigned to each tag. The format may be assigned. In the case of XMDF, for example, it is described as follows.
<char_id char_id = ”CR0001”>Today's weather was sunny, </ char_id>
<char_id char_id = ”CR0002”> It will rain tomorrow. </ char_id>
In this case, “CR0001” and “CR0002” correspond to the character string index.

  In the moving image / book association table 304 shown in FIG. 6A, a time index including the time stamp of the moving image to be displayed between the start position and the end position of the moving image is searched, and the time index corresponding to the searched time index. The sentences at the start position and the end position of the book are displayed. Therefore, taking the case where the time index is 0 as an example, the first to first words in the first chapter, first section, and first paragraph of the book are displayed for 30 seconds from the beginning of the moving image. The movie / book association table 304 shown in FIG.

  The book data 305 is media data related to books, and is a general electronic book data format and includes various information related to the books. In the example of FIG. 4, the book data 305 includes a header 306 and a plurality of main body data 307, but other data may be included.

  The header 306 includes book configuration information, book layout information, and the like. The plurality of main body data 307 are books divided into pages, paragraphs, and the like, and the character string index is assigned to each.

  The AV stream data 308 is media data related to video and audio, and the video and audio are in a known moving image format such as MPEG2 (Moving Picture Experts Group phase 2) -TS (Transport Stream), MP4 (MPEG-4 Part14). Various information about is included. In the example of FIG. 4, the AV stream data 308 includes a header 309, a plurality of video data 310, and a plurality of audio data 311. However, other data may be included.

  The header 309 includes moving image (video and audio) configuration information, information necessary for decoding such as an encoding format, and the like. The plurality of video data 310 is obtained by dividing a moving image, and a time stamp indicating a reproduction timing is assigned to each of the video data 310. Similarly, the plurality of audio data 311 is obtained by dividing audio, and the time stamp is assigned to each.

  Next, details of the content reproduction unit 23 will be described. As shown in FIG. 1, the content reproduction unit 23 includes a content analysis unit (content acquisition unit, correspondence information acquisition unit) 31, an AV reproduction unit (reproduction unit) 32, a book reproduction unit (reproduction unit) 33, a display control unit ( Display control means) 34 and an audio output control unit 35.

  The content analysis unit 31 analyzes (analyzes) the content file 301 read from the content storage unit 22. Specifically, the content analysis unit 31 refers to the file header 302 of the content file 301 and the like, from the content file 301, layout information 303, video / book association table 304, book data 305, AV stream data 308, etc. Extract various information. Of the extracted information, the layout information 303 is displayed on the display control unit 34, the video / book association table 304 is displayed on the book playback unit 33 and the display control unit 34, the AV stream data 308 is stored on the AV playback unit 32, and the book data 305 is stored on the display control unit 34. Each is transmitted to the book reproducing unit 33.

  The AV playback unit 32 plays back the AV stream data 308 from the content analysis unit 31 based on an instruction from the display control unit 34. The AV reproducing unit 32 transmits the reproduced video data 310 to the display control unit 34 and transmits the reproduced audio data 311 to the audio output control unit 35.

  The book reproduction unit 33 reproduces the book data 305 from the content analysis unit 31 based on an instruction from the display control unit 34. The book reproducing unit 33 transmits the reproduced main body data 307 to the display control unit 34.

  The display control unit 34 controls the display on the display 25. In the present embodiment, the display control unit 34 specifies the AV stream data 308 based on the layout information 303 from the content analysis unit 31 when a moving image playback instruction is acquired from the user via the touch panel 24. Next, the display control unit 34 instructs the AV playback unit 32 to play back the specified AV stream data 308, and sequentially acquires the played video data and the time stamp from the AV playback unit 32.

  Next, the display control unit 34 refers to the moving image / book association table 304 from the content analysis unit 31 and identifies the character string index of the book associated with the received time stamp. Next, the display control unit 34 instructs the book reproduction unit 33 to reproduce the text of the specified character string index from the book data 305 and acquires the reproduced character string data from the book reproduction unit 33.

  Next, based on the layout information 303 from the content analysis unit 31, the display control unit 34 includes the video data video from the AV playback unit 32 in the video area 202, and the text string data from the book playback unit 33. Is displayed in the book area 203 and the operation panel area 204 includes the operation panel. Then, the display control unit 34 transmits the generated display image data to the display 25. As a result, the display image is displayed on the display screen 201 of the display 25.

  The audio output control unit 35 controls the timing at which the audio data from the AV playback unit 32 is transmitted to the speaker 26. Note that the audio output control unit 35 may immediately transmit the audio data received from the AV reproduction unit 32 to the speaker 26.

  Next, the flow of processing in the terminal 11 having the above configuration will be described. First, when the content file 301 from the server 12 is downloaded to the terminal 11 via the communication network 13 and stored in the content storage unit 22 of the terminal 11, the content analysis unit 31 reads the layout information 303 from the content file 301. The video / book association table 304, the book data 305, and the AV stream data 308 are extracted.

  The timing at which the content analysis unit 31 starts processing may be when the user gives an instruction via the touch panel 24 in addition to when the content is downloaded from the server 12 as described above. A case where a plurality of content files 301 are stored in the content storage unit 22 is conceivable. In this case, the content analysis unit 31 displays a list of content files 301 on the display 25, and when the user selects via the touch panel 24, the selected content file 301 is read from the content storage unit 22 and processed. It is desirable to start.

  Next, when the user presses the play button 212 shown in FIG. 3, the display control unit 34 refers to the layout information 303 (FIG. 5) extracted by the content analysis unit 31, and the moving image to be displayed on the display screen 201. Is specified (in the example of FIG. 5, “movie01”). Next, the display control unit 34 instructs the AV playback unit 32 to play back the AV stream data 308 having the specified media identifier.

  Thereby, the AV playback unit 32 starts the decoding process of the AV stream data 308 instructed by the display control unit 34. The AV stream data 308 is separated into video data 310 and audio data 311, and the video data 310 is decoded into a format that can be displayed in units of frames. The decoded video frame is transmitted to the display control unit 34 together with a time stamp indicating the display timing. The decoded audio data is transmitted to the audio output control unit 35.

  FIG. 7 is a flowchart showing the flow of processing in the display control unit 34. As illustrated, the display control unit 34 confirms the input of the video frame from the AV playback unit 32 (S601), and when the input is performed, transfers the acquired video frame to the display 25 (S602). In parallel with this, the audio output control unit 35 transfers the audio data from the AV reproduction unit 32 to the speaker 26. Thereby, a moving image is displayed on the display 25 and sound is output from the speaker 26. A series of processes from the decoding of the AV stream data 308 to the transfer of video frames and audio data is repeated until the user presses the stop button 210 shown in FIG. 3 or the end of the AV stream data 308 is reached. And it is executed continuously.

  Next, the display control unit 34 refers to the moving image / book association table 304 ((a) of FIG. 6) extracted by the content analysis unit 31 and identifies the book data corresponding to the time stamp of the video frame. Specifically, the display control unit 34 checks the character string index of the book data corresponding to the time stamp of the video frame (S603), and if there is a corresponding character string index, the corresponding book data The book reproducing unit 33 is notified of the media identifier 305 and the range of the character string index of the book data 305 (S604). At this time, the book reproducing unit 33 reproduces the book data corresponding to the notified media identifier and character string index, and transmits the reproduced book data to the display control unit 34.

  Next, the display control unit 34 confirms the input of the reproduced book data from the book reproducing unit 33 (S605), and when the input is performed, the book shown in FIG. 3 based on the layout information 303. The reproduced book data is arranged in the area 203 and transferred to the display 25 (S606). Then, it returns to step S601 and repeats the said operation | movement. For the moving image area 202 shown in FIG. 3, it is assumed that a transparent color for the video window displaying the moving image is set so that the moving image area 202 is not overwritten.

  Note that the book reproducing unit 33 may notify the display control unit 34 of a time stamp indicating a desired display time together with the book data. Further, the display control unit 34 may transfer the video data to the display 25 in synchronization with the time stamp of the book data and the time stamp of the video frame.

  In addition, when the book data includes audio data, the book reproduction unit 33 preferably transfers the audio data to the audio output control unit 35. In this case, the audio output control unit 35 synthesizes the audio data transferred from the AV reproduction unit 32 and the audio data transferred from the book reproduction unit 33 and transfers them to the speaker 26.

  As described above, the terminal 11 according to the present embodiment can continue the reproduction of the moving image and the book by appropriately synchronizing even when the display form of the book is changed during the synchronized reproduction of the moving image and the book. There is an effect. Hereinafter, this effect will be described with a specific example. As described above, moving image data is AV stream data including at least one of video data and audio data reproduced along a time axis, and book data includes a plurality of character string data constituting a sentence of a book. Contains. The character string data is assigned a character string index corresponding to the number of chapters, sections, and paragraphs that divide the book and the number of words in each paragraph, or a character assigned to each character string in a certain range. Assume that a column index is assigned.

  (A) of FIG. 8 is a figure which shows the example of a display before changing the display form of a book, (b) of the figure is a figure which shows the example of a display after changing the display form of a book. In FIG. 5A, a moving image scene with a time stamp “1:15:22” is displayed in the moving image area 202. On the other hand, in the book area 203, the text of the character string index associated with the time stamp, that is, the text that starts with “then two people” near Chapter 11 is displayed. At this time, the page number of the book is “110”.

  Next, as shown in FIG. 8B, when the size of the text of the book is enlarged, the page number of the book is recalculated to become “201”. On the other hand, in the moving image area 202, there is no display change, and accordingly, a moving image scene with a time stamp of “1:15:22” is displayed. Thus, in this embodiment, even if the page configuration of the book is changed, the scene of the moving image is not changed, and the text of the book associated with the scene can be displayed.

  By the way, some books may include information that is not particularly related to the content of the moving image (unrelated information), information such as an index that does not need to be displayed in synchronization with the moving image (information that does not require synchronization), and the like. On the other hand, in the present embodiment, like the moving image / book association table 304 shown in FIG. 6A, the moving image start position and end position are associated with the book starting position and end position. Thereby, since only the text of the book which needs to be displayed in synchronization with the moving image is associated, the above case can be dealt with.

  For example, in the video / book association table 304 shown in FIG. 6A, focusing on the 63rd and 64th time indexes, the time stamp of the video (movie01) is continuous. However, the character string index of the book (book01) is discontinuous, and the word after the 110th word in the first paragraph of Chapter 5, Section 3, is the 130th word. Therefore, the 111th to 129th words are not displayed.

  Similarly, some books may not be displayed in order from the top in synchronization with the reproduction of the moving image, but may be displayed in a different order. Such a case can be dealt with by using the moving image / book association table 304 shown in FIG.

  For example, in the video / book association table 304 shown in FIG. 6A, focusing on the 64th and 65th time indexes, the time stamp of the video (movie01) is continuous. However, the string index of the book (book01) is discontinuous, and after the 150th word in the first paragraph of Chapter 5, Section 3, becomes the 0th word in the first paragraph of Chapter 1, Section 2. ing. Accordingly, the text is returned to the text in front of the book.

  In addition, there may be a period when the book is not displayed in synchronization with the moving image. Even in such a case, it is possible to cope with this by not assigning any character string index of the book corresponding to the time stamp of the period.

  As described above, according to the moving image / book association table 304, each time the time index advances, the text of the book displayed on the display screen 201 is switched. When the text of the book corresponding to the character string index of each time index fits in the book area 203, it is switched in units of time index.

  However, when the user changes the setting to enlarge the text, the book text may not fit in the book area 203. In this case, the sentence of the book may be divided into a plurality of sentence parts, and the sentence part may be switched and displayed so that the sentence of the book is displayed within the time stamp of the time index. In addition, as for the said several sentence part, it is desirable that the number of characters or the number of words is equal.

  When the page break code is included in the book data, the page break code is basically set at the end of the book data corresponding to the character string index of each time index in the video / book association table 304. In addition, it is desirable to set the start position and end position of the time stamp of the moving image. However, there may be a case where it is not set as such. For this reason, the book reproducing unit 33 checks whether or not a page break code is included before transferring the book data to the display control unit 34. It is desirable to set the display time and transfer.

  That is, when a page break occurs during playback of a moving image segment assigned to a certain time index, page switching is performed so that the display time per page becomes equal. Alternatively, the display time may be set according to the number of words per page. If the display time of the moving picture of the corresponding time index i is T (i) seconds, the total number of words in the corresponding section is N (i), and the number of characters of the page number j is n (j), the display time of the page number j t (j) is calculated by the following equation.

t (j) = T (i) × n (j) / N (i) (1)
Moreover, it is good also as what sets display time so that the number of characters and the number of sentences may become equal instead of the number of words.

  As described above, it has been described that the moving image and the book can be synchronously displayed using the moving image / book association table 304 in FIG. 6A. However, the character string index of the book data may be other than a word unit. . For example, a character unit, a sentence unit, a paragraph unit, a chapter / section / section unit, or the like may be used.

  The page break position determination by the book reproduction unit 33 may be performed after the display control unit 34 is notified of the character string index of the book data to be displayed. It may be calculated and display designation by page number is possible. In addition, when an operation for enlarging or reducing the book area 203 is performed by a user operation or the like during video reproduction, it is desirable to recalculate the position of the page break according to the display area after the execution. .

  As described above, when the book data designated by the book reproduction unit 33 from the display control unit 34 is divided into a plurality of pages, the book reproduction unit 33 includes all the time stamps indicating the display timing of each page. The pages may be transmitted to the display control unit 34 at a time, or may be transmitted at a timing for displaying each page. In the case of transmission for each page, the processing of S602 to S604 and the processing of S605 to S606 shown in FIG. 7 are executed asynchronously, and the video frame transfer processing is blocked while waiting for data input of the book It is desirable to consider so that it is not.

  Further, the start position and end position of the moving image may be specified in units of chapters instead of being specified in the time stamp range. When specifying in units of chapters, correspondence information between chapters and time stamps is included in the AV stream data 308 of FIG. 4 or as metadata different from the AV stream data 308, and the AV playback unit 32 and the display control unit 34 may convert the chapter into a time stamp using the correspondence information and control the display time.

  Further, when an image such as a photograph or a figure is inserted in the book, it is desirable to display the image information together with the text information in the book area 203 of FIG. In this case, it is desirable to display the image information of the book in synchronization with the moving image as well as the text information.

  FIG. 6B shows an example of the moving image / book association table 304 in the form of a table when an image is included in the book. As shown in the figure, in the moving picture / book association table 304, identifiers (image001, image002, etc.) of image data to be displayed are designated for each time index. The book reproducing unit 33 notifies the display control unit 34 of the identifier of the image to be displayed on each page together with the text information.

  As for the display position of the image data, the coordinate information is included in the content in advance, and the image data may be displayed at the designated position for each image, or displayed at a fixed position by performing enlargement / reduction processing. It is good also as a thing. Further, when a plurality of images are designated and do not fit in the book area 203, the images may be divided into a plurality of pages for display. In that case, the display time for each page is adjusted as described above.

  Further, as in the moving picture / book association table 304 shown in FIG. 6B, information relating to the character string index of book data is prepared separately instead of specifying image data in association with the time index. The book reproduction unit 33 may indirectly control the display timing by referring to the image data based on the character string index of the book data.

  The terminal 11 may have a function of displaying only a moving image on a full screen and a function of displaying only a book on a full screen. In this case, the image in the book may be displayed when the book is displayed on the full screen, and may not be displayed when the book is displayed in synchronization with the moving image. In addition, flag information for designating whether or not to display synchronously with a moving image is included for each image in the content file 301, and the display control unit 34 controls display switching according to the information. Also good. For example, if the still image inserted in the book is a one-cut still image included in the movie and it is not meaningful to display it at the same time as the movie, settings are made so that the still image is not displayed synchronously. can do.

  In addition to still images, moving images are conceivable as images included in books. In this case, different moving images are reproduced and displayed in both the moving image area 202 and the book area 203.

  In addition, the table of contents page of the book is not displayed in synchronization with a specific scene of the video, but when selecting content, the table of contents page is displayed first, and the table of contents is selected to correspond to each table of contents. Playback may be started from a video chapter. In this case, the content file 301 may include a table associating positions of chapters and sections of a book corresponding to the table of contents with chapters of moving images. At this time, the position of the book corresponding to the table of contents is specified by the character string index of the book shown in FIG. 6, and the chapter of the moving image may be specified by the time stamp of the moving image shown in FIG. Synchronous reproduction of a moving image and a book can be started from the position of the book corresponding to the table of contents by a method similar to the method of starting reproduction of the moving image from the specified position of the book.

  Further, the table of contents layout information may be separately included in the content file 301. FIG. 9 is a diagram showing an example of a display layout based on the table of contents layout information. In the illustrated example, the table of contents and the representative image of the chapter of the moving image are arranged to be displayed side by side. Specifically, a book area 203 is arranged on the left side of the display screen 201, a moving picture area 202 is arranged on the right side, a table of contents is displayed in the book area 203, and a representative image of a chapter of a moving picture is displayed in the moving picture area 202. It is displayed. The display control unit 34 displays a display image as shown in FIG. 9 on the display screen 201 based on the table of contents layout information.

  Further, when the title text of the table of contents is selected, the corresponding page of the book is displayed on the full screen, and when the representative image of the chapter is selected, the moving image may be reproduced on the full screen from the top of the corresponding chapter. The chapter image may be included in the content file 301 in advance together with index information for associating with the chapter.

  In the above description, the content file 301 including AV stream data and book data is downloaded according to the format shown in FIG. However, the content file 301 may include a URL (Uniform Resource Locator) for acquiring the data body from the server via the communication network 13 instead of including the data body. In this case, all the information included in the content file 301 may be a URL, or some information may be a URL. Further, the AV stream data 308 and the book data 305 may be reproduced while being sequentially acquired from the server 12 in a streaming format, instead of downloading the entire data and temporarily storing it in the content storage unit 22.

  Further, the terminal 11 of the present embodiment can change the playback position of the moving image in synchronization with the displayed page by operating the book area 203 and the operation panel area 204 as described above. When a command such as display of the next page, display of the previous page, page display of a specific page number, or the like is input to the book reproducing unit 33 by a user operation, the character string index of the book data corresponding to the corresponding page It is calculated based on the number of display characters per page, the layout information 303 of the book area 203, and the like. The book reproducing unit 33 refers to the moving image / book association table 304 in FIG. 6, obtains the time stamp of the moving image corresponding to the calculated character string index of the book data, and displays the display target to the display control unit 34. A time stamp is notified together with the book data. The display control unit 34 issues a playback command with the notified type stamp as a playback start position to the AV playback unit 32.

  As the time stamp of the specified moving image, in the example of FIG. 6, a value specified as the starting position of the moving image whose media identifier is “movie01” is assumed. However, as described above, when a plurality of pages are included per time index, a value obtained by adding an offset value for each page to the start position may be designated. For example, if 3 pages are allocated to a 30-second movie, and the display time per page is set to be equal, for the second page, the start position of the table is 10 seconds. A value obtained by adding is designated as the start position.

  Next, the display in the asynchronous mode will be described in comparison with the display in the synchronous mode. FIG. 10 is a diagram illustrating transition of the display screen 201 in the synchronous mode and the asynchronous mode. At the time of synchronous reproduction, as shown in (a) of the figure, the time stamp in the video frame is associated with the page number of the book. On the other hand, in FIG. 5B, a manual page feed operation is started when the time stamp of the video frame is “1:15:19”. Then, after browsing page 101 and page 102 until the time stamp is “1:15:21”, the page is returned to the synchronized playback. Thus, in the case of (b) in the figure, the reproduction of the moving image proceeds in the same manner as in (a) in the figure, but the book can browse the previous page.

  In order not to perform the synchronous display of the moving image at the time of the page turning operation of the book, for example, the time stamp of the moving image corresponding to the character string index of the book data is sent from the book reproducing unit 33 to the display control unit 34. This can be achieved by not sending notifications. After a manual page turning operation is executed, when an instruction to return to synchronous playback is input to the book playback unit 33 or an operation instruction command for a book is not input for a predetermined time, an asynchronous display state It is also possible to return to the synchronous display state by canceling and restarting the notification of the time stamp.

  Further, (c) of FIG. 10 shows an example of screen transition when the synchronized playback is canceled and the trick play operation of the moving image is performed. In the example shown in the figure, when the triple-speed fast-forward operation is started when the time stamp of the video frame is “1:15:19”, the video of the time stamp “1:15:22” which is 3 seconds ahead after 1 second. A frame is displayed, and a video frame with a time stamp of “1:15:25” 6 seconds ahead is displayed 2 seconds later. Thereafter, when returning to the synchronized playback, a video frame with a time stamp of “1:15:22” when the playback is performed at the normal speed is displayed as in FIG. In this way, by temporarily canceling the synchronized playback, the previous scene can be viewed or the previous video can be viewed while maintaining the display page of the book.

  For example, the display control unit 34 does not notify the book reproduction unit 33 of the character string index of the book data so as not to perform synchronous display of the book during trick play reproduction of the moving image. .

  FIG. 11 shows an example of a correspondence relationship between a time index of a moving image and a page number of a book when book data is allocated to a time index of a moving image based on a moving image / book association table 304 in a certain content file 301. It is a figure shown by. As described above, the terminal 11 of this embodiment operates the page-turning operation by switching to the asynchronous mode by operating the synchronous / asynchronous switching button 208 in FIG. 3 during the synchronous reproduction of the moving image and the book. Alternatively, for a predetermined period after starting the page turning operation, it is possible to browse the previous page or browse back to the previous page without synchronizing with the video playback.

  Therefore, in the example of FIG. 11, when the time index is 2 during synchronous reproduction, the displayed page number is 4. At this time, if the mode is switched to the asynchronous mode and manually advanced two pages, the displayed page number is 5. During this time, it is assumed that the moving image continues to be played at the normal speed. When the playback of the video has advanced to the range of time index 3 and the sync / asynchronous switching button 208 is operated to return to the synchronous mode, the displayed book page is switched and the page number is 2. Will be displayed. In addition, when there is no operation for a predetermined time after the page turning operation, when returning to the synchronous mode automatically, the time length for determining that there is no operation is such that the page is not switched immediately after the operation. It is desirable to set the time length sufficient to view one page.

  Also, if you return to synchronous mode after performing a page feed operation in asynchronous mode, the video playback position will be skipped to the current book page instead of aligning the book page to the video playback position. So that playback can be resumed. In order for the user to instruct this function, a button to which the function is assigned may be prepared in advance, or the book may be assigned to an operation such as continuously tapping on the page after the book is manually paged. . In FIG. 11, when the time index is 2, the page is manually advanced by 2 pages, the page with the page number 5 is displayed, and the above operation is performed while the movie is playing the range where the time index is 3. Then, the moving image is skipped to the position where the time index is 4.

  In the above example, in the video / book association table 304, the pages are assigned in the order of the pages assigned according to the time index even when the page is manually paged. Good. In the case of the example in FIG. 11, when the time index is 2, when the next page is advanced, the displayed page number is 5 which is the next to 4. You may make it switch the mode displayed according to the original page order of a book, and the mode displayed in the order allocated in the synchronous display mode. For example, when the target switching button 207 of FIG. 3 is in the book operation mode, the book is displayed according to the original page order of the book, and in the movie operation mode, it is assigned in the synchronous display mode. It is good also as what is displayed in order.

  Further, book data outside the range permitted to be displayed in advance may not be displayed. For example, when a book contains information about a story of a moving image, it is possible to avoid a situation where the user inadvertently knows the ending. This is because, when the display control unit 34 receives a page feed request from the user, the video / book association table 304 in FIG. 6 stores the book data that exceeds the range of the book data assigned to each time index. This can be realized by not issuing an acquisition request to the book reproducing unit 33. Alternatively, it is possible to cope with this by setting in advance a displayable range, such as allowing the next time index range.

  Conversely, when a specific page of a book is displayed, the reproducible range of the moving image may be limited. This can be realized by performing control so that the range exceeding the time stamp of the moving image assigned to the character string index of the book data is not reproduced.

  Also, it may be desirable that the display of the book and the sound output always synchronize with each other, such as when the narration sound corresponding to the contents of the book is included in the moving image. In this case, when a fast-forward operation of a moving image is performed, the video may be played back at a fast speed, while the audio may be played back at a normal speed. In this case, the AV playback unit 32 may be able to independently control the output timing of the video frame data to the display control unit 34 and the output timing of the audio data to the audio output control unit 35.

  Further, during normal playback, only a moving image may be displayed on the display screen 201, and the book may be displayed at the same time by a user operation such as touching the display screen 201. In addition, when displaying a book, the movie may be slowly played in consideration of checking the movie while reading the book. Furthermore, considering the number of characters for each display page of the book, the slow playback speed may be adjusted according to the number of characters.

  For example, the number of characters that can be read per minute is x, the display time of the time index i in the video / book association table 304 in FIG. 6 is T (i) seconds, and the number of characters on the assigned page is y ( If i), the magnification of the reproduction speed of the time index i is determined by the following equation.

T (i) / ((y (i) / x) × 60) (2)
For example, when T (i) is 60 seconds, y (i) is 400 characters, and x is 200 characters, the magnification is 0.5. Control may be performed according to the number of words instead of the number of characters.

  As described above, the synchronization method between the moving image and the book in the present embodiment has been described. However, the terminal 11 that does not display the moving image and views only the book and the sound in synchronization can be realized with the same configuration. In this case, the AV playback unit 32 and the display control unit 34 perform control using the audio time stamp by the same mechanism as the control related to the video.

  FIG. 12 is a diagram illustrating an example of a selection screen for the user to select a content file 301 from a plurality of content files 301 stored in the content storage unit 22. In the illustrated example, a total of nine content files 301 are stored in the content storage unit 22, and a list of these content files 301 is displayed. When the user selects a desired content file 301, the selected content file 301 is highlighted (inverted display in the illustrated example) as shown in FIG.

  In the example of FIG. 12, nine content files 301 are created, but the moving image (movie) included in the content file 301 is “Hello Tora Theater Version”, “Hello Tora TV Version”, and “Hoshi Tora”. There are three types of “privileged video”. Further, there are three types of books (books) included in the content file 301: “original”, “scenario”, and “commentary”. Assuming that a common value is set for the same video and book media identifiers for the aforementioned video and book, the list display of FIG. Like (b) of the figure, it can display collectively for every moving image, and it can be displayed collectively for every book like (c) of the figure.

  In the case of (b) in FIG. 12, the type of moving image can be confirmed, and a book to be reproduced in synchronization can be selected from the book column. In the case of (c) in the figure, the type of the book can be confirmed, and a moving image to be synchronously reproduced can be selected from the movie column. Therefore, the user can optimally switch the display format depending on whether the focus is on the moving image or the book, and the selection of the content is facilitated.

  As a means for specifying the content, each content may be managed by a content ID that can uniquely specify the moving image and the book content on the distribution system, in addition to the above-described media identifier. Alternatively, the content ID of each of the moving image and the book is recorded in the content file 301, and the content reproduction unit 23 can determine whether or not the same content is included by referring to the content ID. Good.

  Moreover, it is good also as what can switch the moving image and book to display on the display screen 201. FIG. In this case, a tab for switching a book to be displayed and a tab for switching a moving image to be displayed may be provided on the display screen 201 in FIG. The terminal 11 refers to the media identifier or content ID corresponding to the moving image and the book corresponding to the selected tab, uniquely identifies the content file 301 containing both, and reproduces the content file 301 to be played back. Can be switched.

[Embodiment 2]
Next, another embodiment of the present invention will be described with reference to FIGS. The terminal 11 shown in FIGS. 1 to 12 displays a set of videos and books synchronously, but the terminal 11 of the present embodiment displays a plurality of books in association with one video. Yes. In addition, the same code | symbol is attached | subjected to the structure which has the function similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  FIG. 13 is a diagram illustrating a configuration example of a display screen of the display 25 in the terminal 11 of the present embodiment. The display screen 201 shown in the figure is different from the display screen 201 shown in FIG. 3 in that a tab 214 for switching the book to be displayed is added, and the other configurations are the same. In the example of FIG. 13, it is shown that one of the three books “original”, “scenario”, and “commentary” can be selected and displayed.

  FIG. 14 is a diagram showing a format of the content file 301 acquired by the terminal 11 of this embodiment. Compared to the format shown in FIG. 4, the format shown in FIG. 4 includes the addition of second book data 312 having a media identifier “book11” and third book data 313 having a media identifier “book21”. The other configurations are the same.

  FIG. 15 is a diagram showing an example of the moving image / book association table 304 included in the content file 301 in a table format. The moving image / book association table 304 in FIG. 6 has correspondence information added to the moving image “movie01”, the book “book11”, and the book “book21”, compared to the moving image / book association table 304 shown in FIG. However, other configurations are the same. As in the case of FIG. 6, the range of the display target book is set for each time index by the character string index of the book data.

  When the user uses the tab 214 to select a book to be displayed, the media identifier of the selected book is notified to the display control unit 34 in FIG. In the example of FIG. 14, any one of “book01”, “book11”, and “book21” is notified. Thereafter, using the media identifier “movie01” of the moving image and the media identifier of the selected book in the same manner as in the above embodiment, referring to the table in FIG. Control of synchronous display can be performed.

  In addition, it is good also as what automatically selects the book to display according to the time slot | zone of the reproduction time of a moving image, instead of switching the book displayed by user operation. In that case, it can be realized by using a moving image / book association table 304 as shown in FIG. In the illustrated example, focusing on the case where the time index is 0, the character string index corresponds to only “book02” among the three book data whose media identifiers are “book02”, “book12”, and “book22”. The attachment is done.

  In this way, it is only necessary to associate the book data to be displayed. If there is a possibility that book data may be displayed overlappingly, such as when authoring is performed, or when a plurality of book data are displayed in the same area, display priority may be set. In the moving image / book association table 304 shown in FIG. 15B, a priority order is set in addition to the character string index. Focusing on the case where the time index is 65, since the priority order of “book02” is set to 2 and the priority order of “book12” is set to 1, “book02” is displayed above. . Note that the priority is set to 0 when no display is performed and no character string index is set.

[Embodiment 3]
Next, still another embodiment of the present invention will be described with reference to FIGS. The terminal 11 shown in FIGS. 1 to 12 displays a set of videos and books synchronously, but the terminal 11 of the present embodiment displays a plurality of videos in association with one book. Yes. In addition, the same code | symbol is attached | subjected to the structure which has the function similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  FIG. 16 is a diagram illustrating a configuration example of a display screen of the display 25 in the terminal 11 of the present embodiment. The display screen 201 shown in the figure is different from the display screen 201 shown in FIG. 2 in that a tab 215 for switching a moving image to be displayed is added, and the other configurations are the same. In the example of FIG. 16, it is shown that one can be selected and displayed from three moving images of “theatrical version”, “TV version”, and “privileged video”.

  FIG. 17 is a diagram showing the format of the content file 301 acquired by the terminal 11 of this embodiment. The format shown in the figure is different from the format shown in FIG. 4 in that AV stream data and a moving picture / book association table are added, and the other configurations are the same.

  Specifically, the second AV stream data 316 having the media identifier “movie11” and the third AV stream data 317 having the media identifier “movie21” are added to the content file 301 of the present embodiment. Has been. In addition, a second moving image / book association table 314 related to the second AV stream data 316 and a third moving image / book association table 315 related to the third AV stream data 317 are added. Yes.

  18A and 18B are diagrams showing examples of the second and third moving image / book association tables 314 and 315 added to the content file 301 in a table format. In the second moving image / book association table 314 shown in FIG. 10A, correspondence information between a moving image having a media identifier “movie11” and a book having a media identifier “book01” is set. Also, in the third moving image / book association table 315 shown in (b) of the same figure, correspondence information between a moving image with a media identifier “movie21” and a book with a media identifier “book01” is set.

  In the present embodiment, the correspondence table 304, 314, and 315 with book data is prepared for each AV stream data. However, by adjusting the number of time indexes without depending on the time length of the moving image content. One correspondence table may include correspondence between a plurality of moving images and one book.

  When the user selects a moving image to be displayed using the tab 215, the media identifier of the selected moving image is notified to the display control unit 34 in FIG. In the example of FIG. 17, any one of “movie01”, “movie11”, and “movie21” is notified. Thereafter, using the media identifier of the selected moving image and the media identifier “book01” of the book in the same manner as in the above-described embodiment, FIG. 13 (a), FIG. 18 (a), and FIG. The synchronized display of the moving image and the book can be controlled with reference to any one of the moving image / book association tables in (b).

[Embodiment 4]
Next, still another embodiment of the present invention will be described with reference to FIG. The terminal 11 of the present embodiment has a configuration in which the terminal 11 shown in FIGS. 13 to 15 and the terminal 11 shown in FIGS. 16 to 18 are combined. In addition, the same code | symbol is attached | subjected to the structure which has the function similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  FIG. 19 is a diagram illustrating a configuration example of a display screen of the display 25 in the terminal 11 of the present embodiment. The display screen 201 shown in the figure is different from the display screen 201 shown in FIG. 3 in that a tab 214 for switching a book to be displayed and a tab 215 for switching a moving image to be displayed are added. The configuration is the same. In the example of FIG. 19, one of three books of “original”, “scenario”, and “commentary” can be selected and displayed, and 3 of “theatrical version”, “TV version”, and “privileged video” are displayed. It shows that one video can be selected and displayed.

  The format of the content file 301 in the present embodiment is a combination of the format shown in FIG. 14 and the format shown in FIG. That is, a moving image / book association table is included for each moving image, and a plurality of book data and a plurality of AV stream data are included. In the example of FIG. 19, it is assumed that any video content is associated with three types of books, “original”, “scenario”, and “commentary”, but each video content is associated with a different book. It is good also as what has been done. In that case, when the moving image content is switched, the title name of the book in the tab 214 is also switched in conjunction with it.

[Embodiment 5]
Next, still another embodiment of the present invention will be described with reference to FIGS. In the present embodiment, the display of books and the display of moving images are performed by separate devices, and both are operated in cooperation. In addition, the same code | symbol is attached | subjected to the structure which has the function similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  FIG. 20 is a diagram showing an outline of the terminal system 40 of the present embodiment. As illustrated, the terminal system 40 includes a mobile terminal 41 that displays books and a television receiver (hereinafter referred to as “TV”) 42 that displays moving images.

  The portable terminal 41 has a function similar to that of the terminal 11 shown in FIG. 1 and further has a function of transferring AV stream data 803 (moving image data including video frames and audio data) to the TV 42. Communication between the portable terminal 41 and the TV 42 may be wired communication using a composite cable connection, HDMI cable connection, or the like, or wireless communication such as WirelessHD.

  FIG. 21 is a block diagram illustrating a schematic configuration of the mobile terminal 41. As shown in the figure, the mobile terminal 41 of the present embodiment has a video output unit 43 that outputs a video frame from the display control unit 34 to the outside, and audio data from the display control unit 34, as compared with the terminal shown in FIG. Is added to the audio output unit 44, and the other configurations are the same. Note that when HDMI or the like is used, the video signal and the audio signal are combined and output to the outside.

  In the present embodiment, books are displayed on the mobile terminal 41 and moving images are displayed on the TV 42. However, moving images may be displayed on the mobile terminal 41 and books may be displayed on the TV 42. .

[Embodiment 6]
Next, another embodiment of the present invention will be described with reference to FIG. In the terminal system 40 shown in FIGS. 20 and 21, the mobile terminal 41 performs AV stream data decoding processing and playback control, and outputs the decoded video and audio signals to an external TV (external display device) 42. It is the structure to do. On the other hand, the terminal system 40 of the present embodiment has a configuration in which an external TV 42 performs AV stream data decoding processing and playback control. In addition, the same code | symbol is attached | subjected to the structure which has the function similar to the structure demonstrated in the said embodiment, and the description is abbreviate | omitted.

  In the terminal system 40 of FIG. 20, AV stream data 803 is compressed video data such as MP4 stored in the built-in memory of the mobile terminal 41. Video reproduction control is performed from the portable terminal 41 to the TV 42 by a control signal 804.

  FIG. 22 is a block diagram showing a schematic configuration of the terminal system 40 of the present embodiment. As illustrated, the terminal system 40 has a configuration in which a mobile terminal 41 and a TV 42 are communicably connected. The portable terminal 41 is different from the terminal 11 shown in FIG. 1 in that an external device control unit 45 that controls an external device is added, and the other configurations are the same.

  As shown in FIG. 22, the TV 42 includes a network I / F 51, a content playback unit 53, a display 55, a speaker 56, a content analysis unit 61, an AV playback unit 62, a display control unit 64, an audio output control unit 65, and an external device. It is a structure provided with the control part 66, respectively, The network I / F21 of the portable terminal 41, the content reproduction part 23, the display 25, the speaker 26, the content analysis part 31, AV reproduction part 32, the display control part 34, and an audio | voice output control part 35 and the same function as the external device control unit 45.

  Next, in the terminal system 40 having the above configuration, when a content playback start command is issued by a user operation, the display control unit 34 instructs the external device control unit 45 to start playback. The external device control unit 45 notifies the playback start command to the TV 42 via the network I / F 21. Note that as a device control protocol between the portable terminal 41 and the TV 42 such as a playback start command, a protocol for controlling an AV device via a network such as DLNA is used.

  In the TV 42, the reproduction start command is received by the network I / F 51 and transferred to the external device control unit 66. The external device control unit 66 determines that the received data is a reproduction start command and transfers it to the display control unit 64. It is assumed that the reproduction start command includes the URL of the content for acquiring the moving image content via the network. The display control unit 64 extracts the URL of the content and notifies the external device control unit 66 of it. The external device control unit 66 analyzes that the notified URL is a data acquisition protocol such as HTTP, and the acquisition destination server is the mobile terminal 41, and sends it to the mobile terminal 41 via the network I / F 51. Then, the acquisition of the AV stream data is requested using the protocol specified by the URL.

  The content data is stored in the content storage unit 22 as in the above embodiment, and the content analysis unit 31 performs analysis processing such as separation of AV stream data and book data. The AV stream data acquisition request is received by the external device control unit 45 via the network I / F 21. The external device control unit 45 sequentially reads the AV stream data specified by the URL of the acquisition request from the content storage unit 22 and transmits it to the TV 42. In addition, when transmitting by HTTP, the external device control part 45 shall have an HTTP server function.

  The TV 42 receives the AV stream data, analyzes the header information of the data received by the content analysis unit 61, and extracts information necessary for reproduction and display, such as the time length of the moving image content and the size of the image. . The AV stream data is sequentially transferred to the AV playback unit 62 and subjected to decoding processing. The decoded video frame is transferred to the display control unit 64 together with time stamp information indicating the display timing. The display control unit 64 arranges the video frame at a predetermined position or the like depending on the resolution or the like of the display 55 and transfers it to the display 55. On the other hand, the decoded audio data is transferred to the audio output control unit 65 together with time stamp information indicating the output timing. The audio output control unit 65 transfers the data to the speaker 56 after performing sound quality processing or the like on the audio data according to user settings.

  The display control unit 64 transfers display time stamp information to the portable terminal 41 via the network I / F 51 in synchronization with the display processing of the video frame. The external device control unit 45 of the portable terminal 41 notifies the display control unit 34 of the received time stamp. Thereafter, the display control unit 34 controls the display of the book data corresponding to the time stamp of the video frame in a manner similar to the method described in the above embodiment, thereby displaying the video displayed on the external display device. The associated book data is displayed on the display of the content reproduction apparatus.

  By performing display control of books on the user interface (touch panel 24) of the portable terminal 41, the display control unit 34 refers to the video / book association table 304 in FIG. Find the time stamp of the video corresponding to the book data. The time stamp of the moving image is notified to the TV 42 via the network I / F 21 together with the playback control command. As a control protocol, as described above, a control protocol for AV equipment such as DLNA is used. The display control unit 64 of the TV 42 controls the AV reproduction unit 62 according to the received control command, thereby reproducing the moving image corresponding to the book data displayed on the portable terminal 41.

  22 includes all the functions of the terminal 11 shown in FIG. 1, and at the same time as the playback of the video on the TV 42, the display 25 of the mobile terminal 41 performs the synchronous playback of the video and the book. It is also possible to do this. Further, when only displaying a book without displaying a video, the function of the AV playback unit 32 or the function of combining the video and the book by the display control unit 34 may be omitted. Moreover, it is good also as a structure which displays a moving image with the portable terminal 41, and displays a book with TV42.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  Finally, each block of the terminal 11, the portable terminal 41, and the TV 42, in particular, the content playback units 23 and 53 may be configured by hardware logic, or may be realized by software using a CPU as follows. Good.

  That is, the terminal 11, the portable terminal 41, and the TV 42 each have a central processing unit (CPU) that executes instructions of a control program that realizes each function, a read only memory (ROM) that stores the program, and a RAM that expands the program. (Random access memory), a storage device (recording medium) such as a memory for storing the program and various data. The object of the present invention is to enable the computer to read program codes (execution format program, intermediate code program, source program) of control programs for the terminal 11, the portable terminal 41, and the TV 42, which are software that realizes the functions described above. This can also be achieved by supplying the recorded recording medium to the terminal 11, the portable terminal 41, and the TV 42, and the computer (or CPU or MPU) reads and executes the program code recorded on the recording medium. .

  Examples of the recording medium include a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and an optical disk such as a CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  Further, the terminal 11, the portable terminal 41, and the TV 42 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used.

  As described above, the content reproduction apparatus according to the present invention reproduces a character string to which a character string index corresponding to an elapsed time of reproduction of stream data being reproduced is reproduced, or a character string reproduced from book data. Since stream data is played back from the elapsed time corresponding to the character string index of, video and / or audio playback and book playback can be output in an appropriate manner, so different types of content can be played back in synchronization. The present invention can be applied to any content playback apparatus that performs the above.

DESCRIPTION OF SYMBOLS 10 Communication system 11 Terminal 12 Server 13 Communication network 22 Content storage part 23/53 Content reproduction part (content reproduction apparatus, content display control apparatus)
24 Touch Panel 25/55 Display (Display Device)
26/56 Speaker 31/61 Content analysis unit (content acquisition means, correspondence information acquisition means)
32.62 AV playback section (playback means)
33 Book reproduction part (reproduction means)
34/64 Display Control Unit (Display Control Unit)
35/65 Audio output control unit 40 Terminal system 41 Portable terminal 42 Television receiver 43 Video output unit 44 Audio output unit 45/66 External device control unit

Claims (16)

A content playback device for playing back content including stream data that is at least one of video and audio data and book data,
Content acquisition means for acquiring the content;
Correspondence information acquisition means for acquiring correspondence information in which information on the elapsed time of reproduction of the stream data is associated with a character string index assigned to each character string in the book data;
Reproduction means for reproducing the stream data and the book data included in the content acquired by the content acquisition means based on the correspondence information, wherein the stream data is reproduced and the elapsed time of the reproduction is The character string to which the corresponding character string index is assigned is reproduced from the book data, or the book data is reproduced and the stream data is reproduced from the elapsed time corresponding to the character string index of the character string to be reproduced. A content reproducing apparatus.   The character string index specifies any unit or combination of units when a sentence is separated by at least one of character unit, word unit, sentence unit, paragraph unit, section unit, and chapter unit. The content reproduction device according to claim 1, wherein the content reproduction device is information. The book data includes media data of at least one of video and audio,
In the correspondence information, information on elapsed time of reproduction of the stream data, a character string index assigned to each character string in the book data, and information on the media data are associated with each other. The content reproduction apparatus according to claim 1 or 2.   4. The reproduction unit according to claim 1, wherein the reproduction unit has a synchronous mode for reproducing the stream data and the book data based on the correspondence information, and an asynchronous mode for reproducing regardless of the correspondence information. The content reproduction device according to any one of the above.   5. The content reproduction apparatus according to claim 4, wherein the reproduction means can limit a reproduction range of one of the stream data and the book data in the asynchronous mode.   In the asynchronous mode, the reproducing means reproduces the book data in the order of the character string index in the correspondence information, or in the order of the character string index given in the book data in order. 6. The content reproduction apparatus according to claim 4, wherein whether or not to reproduce the content can be selected. The content includes a plurality of at least one of the stream data and the book data,
The content reproduction apparatus according to any one of claims 1 to 6, wherein the reproduction unit reproduces the selected stream data and the book data. The content playback device according to any one of claims 1 to 7, wherein content including stream data that is at least one of video and audio data and book data is played back.
A content display control device comprising display control means for controlling an external display device so as to display information reproduced by the content reproduction device.   9. The display control unit according to claim 8, wherein when the character string reproduced by the content reproduction device is divided and displayed, the display control unit displays the number of characters or words of each divided character string in alignment. Content display control device.   10. The display control unit according to claim 8, wherein when the character string reproduced by the content reproduction device is divided and displayed, the display control unit displays the divided character strings with the same display interval. Content display control device.   The display control means adjusts a speed at which stream data reproduced by the content reproduction device is displayed based on the number of characters or the number of words of the character string reproduced by the content reproduction device. The content display control apparatus according to any one of the above. The correspondence information of the content reproduction device further includes a correspondence between a chapter image of the stream data and a table of contents in the book data,
The display control means displays the chapter image and the table of contents based on the correspondence information,
The playback unit of the content playback device specifies a selected elapsed time by selecting the chapter image or the table of contents, and plays back the stream data and the book data from the specified elapsed time. Item 12. The content display control device according to any one of Items 8 to 11.   The content display control device according to any one of claims 8 to 12, wherein the display control means transmits the stream data and book data reproduced by the content reproduction device to separate display devices.   The content display control apparatus according to claim 13, wherein the display control unit controls display timing on the separate display apparatuses.   A control program for operating the content reproduction apparatus according to any one of claims 1 to 7, wherein the control program causes a computer to function as each of the above means. A method for controlling a content playback apparatus for playing back content including stream data that is at least one of video and audio data and book data,
A content acquisition step of acquiring the content;
A correspondence information acquisition step of acquiring correspondence information in which information on the elapsed time of reproduction of the stream data is associated with a character string index assigned to each character string in the book data;
In the reproduction step of reproducing the stream data and the book data included in the content acquired in the content acquisition step based on the correspondence information, the stream data is reproduced and the progress of the reproduction A character string assigned a character string index corresponding to time is reproduced from the book data, or the book data is reproduced and stream data is calculated from an elapsed time corresponding to the character string index of the character string to be reproduced. And a playback step of playing back the content.

Images