JP7282118B2 - Program and information processing method - Google Patents

Description

The present invention relates to a program and an information processing method.

There are technologies related to audio text in video content. For example, Patent Literature 1 discloses an audio confirmation system that determines whether or not audio text obtained based on audio included in moving image content corresponds to caption text included in the moving image content.

JP 2020-166262 A

However, the invention according to Patent Document 1 cannot register voice text (spoken text) in video content.

One aspect is to provide a program or the like capable of registering spoken text in video content.

In one aspect, the program acquires video content and an utterance text describing the utterance content of a speaker in the video content, displays the utterance text step by step according to the reproduction of the acquired video content, and is displayed. a first tab that transmits the spoken text to an information processing device when an operation for registering the spoken text is received , and shows a list of the registered spoken texts by the registration operation for each video content viewed by the user; displaying a second tab showing a list of the registered utterance texts by the registration operation in the order of registration of the utterance texts, and displaying the registered utterance texts for each video content when the selection operation of the first tab is accepted. are displayed in a list, and when the selection operation of the second tab is accepted, the computer executes a process of displaying the utterance texts in a list in the order of registration.

In one aspect, it is possible to register spoken text in video content.

1 is an explanatory diagram showing an overview of a video distribution system; FIG. It is a block diagram which shows the structural example of a server. FIG. 4 is an explanatory diagram showing an example of record layouts of a moving image content DB and an utterance text DB; FIG. 4 is an explanatory diagram showing an example of record layouts of a user DB and a history DB; FIG. 4 is an explanatory diagram showing an example of record layouts of a note DB and a category DB; 2 is a block diagram showing a configuration example of a terminal; FIG. FIG. 10 is an explanatory diagram showing an example of a video content list screen; FIG. 10 is an explanatory diagram showing an example of a detailed screen of moving image content; FIG. 11 is an explanatory diagram showing an example of a category list screen; FIG. 10 is an explanatory diagram showing an example of a category detail screen; FIG. 10 is a flow chart showing a processing procedure when displaying spoken text in video content; FIG. FIG. 10 is a flow chart showing a processing procedure when registering or canceling the registration of an utterance text; FIG. FIG. 10 is an explanatory diagram showing an example of a list screen of utterance texts by moving image content; FIG. 4 is an explanatory diagram showing an example of a display screen of all spoken texts in video content; FIG. 10 is an explanatory diagram showing an example of a list screen of utterance texts in order of registration; FIG. 10 is a flow chart showing a processing procedure for displaying a list of spoken texts; FIG. FIG. 5 is an explanatory diagram showing an example of a switching screen;

Hereinafter, the present invention will be described in detail based on the drawings showing its embodiments.

(Embodiment 1)
Embodiment 1 relates to a form of displaying an utterance text describing the utterance content of a speaker in moving image content in accordance with the reproduction of the moving image content. The video content includes, for example, video content for classes, video content for seminars, video interview content, or video content related to cooking, makeup, culture, games, or the like.

FIG. 1 is an explanatory diagram showing an overview of a video distribution system. The system of this embodiment includes an information processing device 1 and an information processing terminal 2, and each device transmits and receives information via a network N such as the Internet.

The information processing device 1 is an information processing device that processes, stores, and transmits/receives various information. The information processing device 1 is, for example, a server device, a personal computer, a general-purpose tablet PC (personal computer), or the like. In the present embodiment, the information processing device 1 is assumed to be a server device, which is replaced with the server 1 for the sake of brevity.

The information processing terminal 2 is a terminal device that receives and reproduces moving image content, acquires and displays spoken text in the moving image content, and the like. The information processing terminal 2 is, for example, an information processing device such as a smart phone, a mobile phone, a tablet, or a personal computer terminal. For the sake of brevity, the information processing terminal 2 is replaced with the terminal 2 below.

The server 1 according to the present embodiment acquires video content and utterance text that describes the utterance content of the speaker in the video content. The server 1 displays the spoken text in the moving image content on the screen step by step in accordance with the reproduction of the acquired moving image content. The server 1 registers (stores) the utterance text when receiving a registration operation for the utterance text.

FIG. 2 is a block diagram showing a configuration example of the server 1. As shown in FIG. The server 1 includes a control section 11 , a storage section 12 , a communication section 13 , an input section 14 , a display section 15 , a reading section 16 and a mass storage section 17 . Each configuration is connected by a bus B.

The control unit 11 includes arithmetic processing units such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). , various information processing, control processing, etc. related to the server 1 are performed. It should be noted that the control program 1P is deployed to run on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network. can do. Note that although FIG. 2 illustrates the controller 11 as a single processor, it may be a multiprocessor.

The storage unit 12 includes memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory), and stores the control program 1P or data necessary for the control unit 11 to execute processing. The storage unit 12 also temporarily stores data and the like necessary for the control unit 11 to perform arithmetic processing. The communication unit 13 is a communication module for performing processing related to communication, and transmits and receives information to and from the terminal 2 via the network N.

The input unit 14 is an input device such as a mouse, keyboard, touch panel, buttons, etc., and outputs received operation information to the control unit 11 . The display unit 15 is a liquid crystal display, an organic EL (electroluminescence) display, or the like, and displays various information according to instructions from the control unit 11 .

The reader 16 reads a portable storage medium 1a including CD (Compact Disc)-ROM or DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the large-capacity storage unit 17 . Alternatively, the control unit 11 may download the control program 1P from another computer via the network N or the like and store it in the large-capacity storage unit 17 . Furthermore, the control unit 11 may read the control program 1P from the semiconductor memory 1b.

The large-capacity storage unit 17 includes a recording medium such as an HDD (Hard disk drive) or an SSD (Solid State Drive). The large-capacity storage unit 17 includes a video content DB (database) 171 , an utterance text DB 172 , a user DB 173 , a history DB 174 , a note DB 175 and a category DB 176 .

The video content DB 171 stores information on video content. The utterance text DB 172 stores utterance texts that describe the contents of the utterances of the speakers in the video content. The user DB 173 stores information about users. The history DB 174 stores a history of viewing video content by the user. The note DB 175 stores utterance texts registered by a user's registration operation. The category DB 176 stores categories of moving image content.

In addition, in this embodiment, the storage unit 12 and the large-capacity storage unit 17 may be configured as an integrated storage device. Also, the large-capacity storage unit 17 may be composed of a plurality of storage devices. Furthermore, the large-capacity storage unit 17 may be an external storage device connected to the server 1 .

The server 1 may be executed by a single computer, may be executed by a plurality of computers in a distributed manner, may be realized by a plurality of virtual machines provided in one server, or may be implemented by a cloud server. may be implemented using

FIG. 3 is an explanatory diagram showing an example of record layouts of the video content DB 171 and the spoken text DB 172. As shown in FIG.
The video content DB 171 includes a video content ID column, a category ID column, a video column, a title column, a thumbnail image column, and a release date column. The video content ID column stores IDs of video content that are uniquely identified in order to identify each video content.

The category ID column stores category IDs that specify categories of video content. The moving image column stores data of moving image content. The title column stores titles of video content. The thumbnail image row stores thumbnail images of moving image content. The release date column stores the dates when the video content was released.

The spoken text DB 172 includes a video content ID column, a spoken ID column, a spoken text column, a start time column, a display time column, and a spoken text type column. The video content ID column stores video content IDs for specifying video content corresponding to the spoken text. The utterance ID column stores IDs of uniquely specified utterance texts in order to identify each utterance text.

The utterance text string stores the utterance content of the speaker in the video content. The utterance text is text edited by an editor to express the intention (objective) or gist of the utterance based on the textual information of the utterance of the speaker. It should be noted that all the contents of the utterance may be converted into text and provided as the utterance text. The starting time column stores the timing of displaying the spoken text. The display time column stores the display time of the spoken text. The spoken text type column stores the type of spoken text. The types of speech texts include “representative”, which is representative speech texts identified from a plurality of speech texts, and “general”, which are speech texts other than the representative speech texts.

FIG. 4 is an explanatory diagram showing an example of the record layout of the user DB 173 and history DB 174. As shown in FIG.
The user DB 173 includes a user ID column, a name column, a membership type column, and a viewing start date column. The user ID column stores unique user IDs for identifying each user. The name column stores the names of users. The member type column stores the member type of the user. Membership types include, for example, "corporation" and "general". Note that membership types may be provided according to actual needs. The viewing start date column stores the date when the user started viewing the video content.

The history DB 174 includes a history ID column, a video content ID column, a user ID column, and a viewing date/time column. The history ID column stores IDs of history data that are uniquely identified in order to identify each history data. The video content ID column stores video content IDs that specify video content. The user ID column stores user IDs that identify users. The viewing date and time column stores the date and time when the user viewed the video content.

FIG. 5 is an explanatory diagram showing an example of the record layout of the note DB 175 and category DB 176. As shown in FIG.
The note DB 175 includes a user ID column, a video content ID column, an utterance ID column, and a registration date/time column. The user ID column stores user IDs that identify users. The video content ID column stores video content IDs that specify video content. The utterance ID column stores utterance IDs that specify utterance texts. The registration date and time column stores the date and time when the utterance text was registered.

The category DB 176 includes a category ID column, category name column, background color column and icon column. The category ID column stores a unique category ID to identify each category. The category name column stores the names of categories. The background color column stores the background color of the display area classified by category and displayed in different colors. The icon column stores icons indicating each category.

A category is an arbitrary type set according to video content. A category is a type that is set to deal with things around or related to a certain thing or topic, such as ``interesting life'', ``fascinating life'', or ``ask a doctor''. Also good.

FIG. 6 is a block diagram showing a configuration example of the terminal 2. As shown in FIG. The terminal 2 includes a control section 21 , a storage section 22 , a communication section 23 , an input section 24 , a display section 25 and a speaker 26 . Each configuration is connected by a bus B.

The control unit 21 includes an arithmetic processing device such as a CPU and an MPU, and reads and executes a control program 2P stored in the storage unit 22 to perform various information processing, control processing, and the like related to the terminal 2 . In addition, in FIG. 6, the controller 21 is described as being a single processor, but it may be a multiprocessor. The storage unit 22 includes memory elements such as RAM and ROM, and stores the control program 2P or data necessary for the control unit 21 to execute processing. The storage unit 22 also temporarily stores data and the like necessary for the control unit 21 to perform arithmetic processing.

The communication unit 23 is a communication module for performing processing related to communication, and transmits/receives information to/from the server 1 via the network N. The input unit 24 may be a keyboard, a mouse, or a touch panel integrated with the display unit 25 . The display unit 25 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 21 . The speaker 26 is a device that converts electrical signals into sound.

Next, the process of displaying the spoken text in accordance with the reproduction of the video content will be described. Note that, in the present embodiment, categories of moving image content are stored in the category DB 176 of the server 1 in advance. Also, the moving image content and the spoken text in the moving image content are stored in advance in the moving image content DB 171 and the spoken text DB 172 of the server 1 .

The spoken text may be obtained by manual input by an editor, or may be obtained by converting from audio data of video content. When the conversion processing of voice data is used, the server 1 extracts voice data from the moving image content and performs voice recognition processing for converting the extracted voice data into spoken text. Speech To Text (STT) technology, for example, may be used for speech recognition processing. Note that the speech data may be converted to spoken text by deep learning or other neural network implemented learning models. The server 1 stores the acquired spoken text in the spoken text DB 172 in association with the moving image content ID.

Then, the terminal 2 acquires the moving image content and the spoken text in the moving image content from the server 1 . Specifically, the terminal 2 transmits to the server 1 an acquisition request for moving image content and spoken text. The server 1 acquires information on a plurality of moving image contents from the moving image content DB 171 in response to the acquisition request transmitted from the terminal 2 . The information about the moving image content includes the moving image content ID, the moving image content category, the title, the thumbnail image, the release date, and the like.

The server 1 acquires information about the spoken text in each moving image content from the spoken text DB 172 based on each moving image content ID thus acquired. The information about the utterance text includes the utterance ID, the utterance text, the display start time and display time of the utterance text, and the like. The server 1 transmits to the terminal 2 information about the acquired video content and information about the spoken text. The terminal 2 receives the information about the video content and the information about the spoken text transmitted from the server 1 .

Terminal 2 identifies a representative spoken text from a plurality of spoken texts in each video content. A typical spoken text is set in advance by an editor, for example. For example, when the editor selects an utterance text that can express the intention or gist of the utterance from the contents of the utterance, the terminal 2 receives input of the utterance text selected by the editor. The terminal 2 stores the type of the "representative" utterance text in the utterance text DB 172 in association with the utterance ID of the received representative utterance text. In this way, when representative spoken texts are stored in advance in the spoken text DB 172, the terminal 2 selects, from the spoken text DB 172, an spoken text whose type of spoken text is "representative" from a plurality of spoken texts in the video content. Extract. Terminal 2 identifies the extracted speech text as a representative speech text.

It should be noted that the process is not limited to the above-described typical speech text specifying process. For example, the server 1 clusters the spoken texts based on the title of the video content included in the spoken texts or related words of the titles. The server 1 may randomly acquire a predetermined number (eg, 2) of spoken texts from clusters in which the number of spoken texts included in each cluster is equal to or greater than a predetermined number (eg, 10). This makes it possible to select utterance texts that have diversity and are likely to become topics of discussion.

The terminal 2 displays a representative spoken text on the screen in association with the title and thumbnail image of the moving image content for each category of moving image content. When terminal 2 receives an operation to reproduce video content, terminal 2 acquires data of the video content from video content DB 171 of server 1 based on the video content ID. The terminal 2 reproduces the acquired video content data. The terminal 2 displays the spoken text in the moving image content on the screen step by step in accordance with the reproduction of the moving image content. Specifically, the terminal 2 displays each utterance text on the screen based on the display start time of each utterance text during playback of the video content.

When the terminal 2 receives the registration operation for the utterance text, the terminal 2 transmits the utterance ID of the utterance text to the server 1 in association with the user ID and the video content ID. The server 1 receives the user ID, video content ID, and speech ID transmitted from the terminal 2 . The server 1 registers (stores) the utterance ID and the date and time of registration in the note DB 175 as one record in association with the received user ID and video content ID.

FIG. 7 is an explanatory diagram showing an example of a video content list screen. The screen includes a today's disclosure display field 11a, a category display field 11b, and an individual category display field 11c.

The today's release display column 11a is a display column for displaying the video content released today. In today's release display field 11a, a plurality of video contents released today, a plurality of recommended video contents, or the like are displayed side by side in the second direction (for example, horizontal direction). Note that the video content released today and the recommended video content may be displayed side by side at the same time. If the plurality of moving image contents arranged in the second direction cannot fit in the today's disclosure display field 11a, the plurality of moving image contents can be scrollably displayed in the second direction.

The today's release display field 11a includes a moving image display field 11a1, a play button 11a2, a title display field 11a3, a release date display field 11a4, and a details button 11a5. The moving image display column 11a1 is a display column for displaying thumbnail images of moving image content. The play button 11a2 is a button for playing video content. The title display field 11a3 is a display field for displaying the title of the video content. The release date display column 11a4 is a display column for displaying the release date of the video content. The detail button 11a5 is a button for transitioning to a detail screen (FIG. 8) of moving image content, which will be described later.

The category display column 11b is a display column for displaying categories (islands) of video content. If a plurality of categories arranged in a row adjacent to each other in the horizontal and vertical directions cannot fit in the category display field 11b, the plurality of categories can be horizontally and vertically scrollable. The category display field 11b includes a category list button 11b1 and a category name display field 11b2. The category list button 11b1 is a button for transitioning to a category list screen (FIG. 9), which will be described later. The category name display field 11b2 is a display field for displaying the name of the category.

The terminal 2 acquires category-related information including the category ID, category name, and background color from the category DB 176 of the server 1 . The terminal 2 displays the acquired information about the category in the category display field 11b. Specifically, the terminal 2 displays the acquired name of each category in the corresponding category name display field 11b2. The terminal 2 sets the acquired background color of each category as the background color of the corresponding category name display field 11b2. It should be noted that the category display is not limited to the color-coded display described above, and an icon indicating each category may be used. When receiving a touch operation on the category list button 11b1, the terminal 2 transfers the category ID and the name of the category to the category list screen (FIG. 9), and transitions to the category list screen.

The category individual display field 11c is a display field for displaying a plurality of moving image contents belonging to the category. The individual category display fields 11c are provided according to the number of categories, and displayed side by side in the first direction (for example, the vertical direction). If the plurality of individual category display fields 11c arranged in the first direction cannot be displayed on the screen, the plurality of individual category display fields 11c can be scrollably displayed in the first direction.

Also, in each category individual display field 11c, a plurality of moving image contents belonging to the category are displayed side by side in the second direction (for example, horizontal direction). If the plurality of moving image contents arranged in the second direction cannot fit in the corresponding category individual display field 11c, the plurality of moving image contents can be scrollably displayed in the second direction.

The individual category display field 11c includes a moving image display field 11a1, a title display field 11a3, a category name display field 11c1, a spoken text display field 11c2, and a category details button 11c3. The moving image display field 11a1 and the title display field 11a3 are the same as those described above, so description thereof will be omitted. The category name display field 11c1 is a display field for displaying the name of the category. The spoken text display field 11c2 is a display field for displaying the spoken text in the video content. The category details button 11c3 is a button for transitioning to a category details screen (FIG. 10), which will be described later.

The terminal 2 transmits information about video content (video content ID, video content category, title, thumbnail image, release date, etc.) and information about spoken text in the video content (speech ID, spoken text, etc.) to the video of the server 1. Acquired from the content DB 171 and the spoken text DB 172 .

The terminal 2 extracts the video content released today based on the release date of the acquired video content. The terminal 2 displays the extracted video content released today in the today's release display field 11a. Specifically, the terminal 2 displays the extracted thumbnail image of each moving image content in the corresponding moving image display column 11a1, displays the title of each moving image content in the corresponding title display column 11a3, and displays the release date of each moving image content. It is displayed in the corresponding publication date display field 11a4.

When the terminal 2 receives the touch (click) operation of the playback button 11a2, the terminal 2 acquires the data of the moving image content from the moving image content DB 171 of the server 1 based on the moving image content ID. The terminal 2 reproduces the acquired video content data. When the terminal 2 receives the touch operation of the detail button 11a5, the terminal 2 passes the moving image content ID to the detailed screen of the moving image content (FIG. 8), and transitions to the detailed screen of the moving image content.

The terminal 2 identifies representative spoken texts from the spoken texts in the video content acquired from the server 1 . For each category, the terminal 2 displays a representative spoken text in the individual category display field 11c in association with the video content. Specifically, the terminal 2 displays the thumbnail image of the moving image content in the moving image display field 11a1, displays the title of the moving image content in the title display field 11a3, and displays the typical spoken text in the moving image content in the spoken text display field 11c2. to display.

When the terminal 2 accepts the touch operation of the category details button 11c3, it passes the category ID and each video content ID belonging to the category to the category details screen (FIG. 10), and transitions to the category details screen.

FIG. 8 is an explanatory diagram showing an example of a detailed screen of moving image content. Note that FIGS. 8A, 8B, and 8C are explanatory diagrams showing an example of the detailed screen changed according to the user's operation. FIG. 8A is a detailed screen when expanding the spoken text. FIG. 8B is a detailed screen when collapsing the spoken text. FIG. 8C is a detailed screen when the moving image content is displayed on the full screen.

The screen includes a moving image display field 12a, a play button 12b, a fast rewind button 12c, a fast forward button 12d, a spoken text display field 12e, a folding button 12f, a full screen display button 12g, a speech play button 12h, a registration button 12i, and a share button 12j. and an utterance start point display field 12k.

The moving image display column 12a is a display column for displaying moving image content. The play button 12b is a button for playing/pausing moving image content. The rewind button 12c is a button for rewinding moving image content. The fast-forward button 12d is a button for fast-forwarding moving image content. The spoken text display field 12e is a display field for displaying the spoken text in the video content.

The folding button 12f is a button for expanding or folding the spoken text. The full screen display button 12g is a button for displaying video content in full screen. The speech playback button 12h is a button for playing back video content with the display start time (timing) of the speech text as the playback start position of the video content. The registration button 12i is a button for registering an utterance text. The share button 12j is a button for sharing the spoken text associated with the video content. The speech start time display field 12k is a display field for displaying the start time of display of the speech text.

The terminal 2 receives the video content ID passed from the video content list screen (FIG. 7). Based on the received video content ID, the terminal 2 acquires information about the video content (video content data, thumbnail images, etc.) and spoken text in the video content from the video content DB 171 and the spoken text DB 172 of the server 1. .

The terminal 2 displays thumbnail images of the plurality of acquired moving image contents in the moving image display field 12a. The terminal 2 identifies a representative speech text from the obtained speech texts, and displays the identified representative speech text in the speech text display field 12e. Note that the process of specifying a representative utterance text is the same as the process described above, so the description is omitted. For example, the terminal 2 may display the first utterance text in chronological order from the time when the display of the utterance text is started, without being limited to representative utterance texts.

The terminal 2 reproduces the data of the video content when receiving the touch operation of the reproduction button 12b. The terminal 2 switches and displays each utterance text to the utterance text display field 12e based on the start time of display of each utterance text in the moving image content during playback of the moving image content. It should be noted that, when the terminal 2 receives the touch operation of the play button 12b again during the reproduction of the video content, the reproduction of the data of the video content is temporarily stopped. When the terminal 2 receives the touch operation of the fast-rewind button 12c, the terminal 2 fast-reverses the moving image content in a predetermined number of seconds (for example, 10 seconds). When the terminal 2 receives the touch operation of the fast-forward button 12d, the terminal 2 fast-forwards the video content for a predetermined number of seconds (for example, 10 seconds).

When receiving the touch operation of the fold button 12f, the terminal 2 expands the speech texts and displays all the speech texts in the speech text display field 12e (FIG. 8B). If the plurality of utterance texts arranged in the vertical direction cannot fit in the utterance text display field 12e, the plurality of utterance texts can be displayed so as to be scrollable in the vertical direction. In addition, while the moving image content is being reproduced, the terminal 2 displays the displayed spoken text in a highlight (for example, bold line, diagonal line, or color) in accordance with the reproduction timing of the moving image content. When the terminal 2 accepts again the touch operation of the fold button 12f for the speech text that is "developing", the terminal 2 folds the speech text (FIG. 8A).

When the speech texts are displayed in the "folded" state, the terminal 2 generates a registration button 12i and a share button 12j for each speech text and displays them on the screen. When the utterance text is displayed in the "deploying" state, the terminal 2 generates an utterance reproduction button 12h, a registration button 12i, a share button 12j, and an utterance start time display field 12k for each utterance text, and displays them on the screen. do. The terminal 2 displays the start point of display of each utterance text in the corresponding utterance start point display field 12k.

When the terminal 2 receives the touch operation of the speech reproduction button 12h, the terminal 2 reproduces the moving image content at the reproduction timing corresponding to the display timing (start point) of the speech text. For example, if the speech text display start time is "02:34", the terminal 2 reproduces the video content at the reproduction timing of "02:34".

When the terminal 2 receives the touch operation of the registration button 12i during playback of the moving image content, the terminal 2 transmits the utterance ID of the utterance text to the server 1 in association with the user ID and the moving image content ID. The server 1 receives the user ID, video content ID, and speech ID transmitted from the terminal 2 . The server 1 registers the utterance ID and the registration date and time as one record in the note DB 175 in association with the received user ID and video content ID. The terminal 2 changes the registration button 12i corresponding to the registered utterance text to the registered state. For example, the terminal 2 changes the background color of the registration button 12i to red.

It should be noted that, when the terminal 2 receives again the touch operation of the registration button 12i for the registered utterance text, the terminal 2 performs registration cancellation processing of the utterance text. Specifically, the terminal 2 transmits the user ID, the video content ID, and the speech ID of the speech text to the server 1 . The server 1 deletes the corresponding utterance text record from the note DB 175 based on the user ID, the video content ID, and the utterance ID of the utterance text transmitted from the terminal 2 . The terminal 2 changes the corresponding registration button 12i to the unregistered state. For example, the terminal 2 changes the background color of the registration button 12i to gray.

When receiving the touch operation of the share button 12j, the terminal 2 associates the corresponding utterance text with the moving image content and shares it. The sharing destination may be, for example, an SNS (Social Networking Service) site, an electronic bulletin board, or the like. The SNS site is, for example, a site such as Twitter (registered trademark), Facebook (registered trademark), or LINE (registered trademark). An electronic bulletin board is a site where articles can be written, viewed, and comments can be added in an environment using a network.

For example, the thumbnail image of the video content to be shared and the spoken text itself may be shared with a specified sharing destination, or the URL (Uniform Resource Locator) describing the thumbnail image and spoken text of the video content may be sent to the sharing destination. may be shared to

When accepting the touch operation of the full screen display button 12g, the terminal 2 displays the corresponding video content in full screen (FIG. 8C), and switches the spoken text in the video content to non-display. When the terminal 2 receives again the touch operation of the full-screen display button 12g for the video content displayed in full screen, the terminal 2 cancels the full-screen display of the video content and displays the spoken text in the video content. switch to

FIG. 9 is an explanatory diagram showing an example of a category list screen. The screen includes a category list 13a. The category list 13a is a list displaying a list of categories. The category list 13a includes category icons 13a1, category name display fields 13a2, and category detail buttons 13a3. The category icon 13a1 is an icon indicating a category. The category name display field 13a2 is a display field for displaying the name of the category. The category details button 13a3 is a button for transitioning to a category details screen (FIG. 10), which will be described later.

The terminal 2 acquires information about categories from the category DB 176 of the server 1 . The information about the category includes category ID, category name, and icon indicating the category. The terminal 2 displays the acquired information about the category on the category list 13a. Specifically, the terminal 2 displays an icon indicating each category in the corresponding category icon 13a1, and displays the name of each category in the category name display field 13a2.

When the terminal 2 receives the touch operation of the category detail button 13a3, the terminal 2 passes the category ID of the corresponding category to the category detail screen (FIG. 10), and transitions to the category detail screen.

FIG. 10 is an explanatory diagram showing an example of a category detail screen. The screen includes a category name display field 14a and a moving image display field 14b. The category name display field 14a is a display field for displaying the name of the category. The moving image display column 14b is a display column for displaying information regarding moving image content. The moving image display field 14b includes a title display field 14b1, a thumbnail image display field 14b2, a spoken text display field 14b3, and a detail button 14b4.

The title display column 14b1 is a display column for displaying the title of moving image content. The thumbnail image display column 14b2 is a display column for displaying thumbnail images of moving image content. The spoken text display field 14b3 is a display field for displaying the spoken text in the video content. The detail button 14b4 is a button for transitioning to a detailed screen (FIG. 8) of moving image content.

The terminal 2 acquires the category name and background color from the category DB 176 of the server 1 based on the category ID passed from the transition source screen (for example, the category list screen). Based on the category ID, terminal 2 acquires information on all video content belonging to the category from video content DB 171 of server 1 . The information about moving image content includes a moving image content ID, moving image content data, a title, a thumbnail image, and the like. The terminal 2 acquires a plurality of spoken texts in the moving image content from the spoken text DB 172 of the server 1 based on the moving image content ID.

The terminal 2 displays the name of the acquired category in the category name display field 14a, and sets the background color of the category as the background color of the category name display field 14a. The terminal 2 arranges and displays the plurality of acquired moving image contents vertically in each of the moving image display fields 14b. The terminal 2 displays the title of the moving image content displayed in each moving image display column 14b in the title display column 14b1, and displays the thumbnail image of the moving image content in the thumbnail image display column 14b2.

The terminal 2 displays a plurality of utterance texts in the moving image content displayed in the utterance text display column 14b3 in association with the moving image content displayed in each moving image display column 14b, arranged horizontally in ascending order of the utterance IDs, for example. If the plurality of uttered texts arranged in the horizontal direction cannot fit in the moving image display field 14b, the plurality of uttered texts can be displayed in a horizontal scrollable manner.

When receiving a swipe operation on the spoken text display field 14b3, the terminal 2 switches and displays a plurality of spoken texts. As illustrated, a plurality of spoken texts including spoken text 1 and spoken text 2 are associated with the video content A, and spoken text 1 is displayed on the screen. When the terminal 2 accepts a leftward swipe operation on the spoken text display field 14b3, the terminal 2 switches the spoken text 1 to the next spoken text 2 and displays it. When the terminal 2 accepts again the swipe operation to the left of the spoken text display field 14b3 while displaying the spoken text 2, the terminal 2 switches and displays the spoken text 2 to the next spoken text 3 (not shown). Further, when the terminal 2 accepts a swipe operation to the right of the spoken text display field 14b3 while displaying the spoken text 2, the terminal 2 switches the spoken text 2 to the previous spoken text 1 and displays it. In this manner, the user's swipe operation can sequentially switch and display the spoken text. Note that the terminal 2 may automatically switch and display a plurality of spoken texts at predetermined time intervals (for example, 1 second). When the terminal 2 receives the touch operation of the detail button 14b4, the terminal 2 passes the moving image content ID to the detail screen of the moving image content (FIG. 8), and transitions to the detail screen of the moving image content.

FIG. 11 is a flow chart showing a processing procedure for displaying spoken text in video content. The control unit 11 of the server 1 acquires information on video content and information on spoken text (step S101). Specifically, the control unit 11 acquires information about the moving image content including the moving image content ID, moving image content category, title, thumbnail image, release date, and the like from the moving image content DB 171 of the large-capacity storage unit 17 . The control unit 11 acquires information about the spoken text in each moving image content from the spoken text DB 172 of the large-capacity storage unit 17 based on the acquired moving image content ID. The information about the utterance text includes the utterance ID, the utterance text, the display start time and display time of the utterance text, and the like.

The control unit 11 transmits the acquired information about the moving image content and information about the spoken text to the terminal 2 through the communication unit 13 (step S102). The control unit 21 of the terminal 2 receives, through the communication unit 23, the information about the moving image content and the information about the spoken text transmitted from the server 1 (step S201). The control unit 21 identifies a representative spoken text from a plurality of spoken texts in each video content (step S202).

The control unit 21 causes the display unit 25 to display a representative spoken text in association with the title and thumbnail image of the moving image content for each moving image content category (step S203). When the input unit 24 receives a video content reproduction operation (step S204), the control unit 21 transmits the video content ID to the server 1 through the communication unit 23 (step S205).

The control unit 11 of the server 1 receives the video content ID transmitted from the terminal 2 through the communication unit 13 (step S103). The control unit 11 acquires data of the moving image content from the moving image content DB 171 of the large-capacity storage unit 17 based on the received moving image content ID (step S104). The control unit 11 transmits the acquired video content data to the terminal 2 through the communication unit 13 (step S105).

The control unit 21 of the terminal 2 receives the moving image content data transmitted from the server 1 through the communication unit 23 (step S206). The control unit 21 reproduces the received video content data via the speaker 26 (step S207). The control unit 21 causes the display unit 25 to display the spoken text in the moving image content step by step in accordance with the reproduction of the data of the moving image content (step S208), and ends the process.

FIG. 12 is a flow chart showing a processing procedure for registering or canceling the registration of an utterance text. When the input unit 24 receives a registration operation for an unregistered utterance text (step S211), the control unit 21 of the terminal 2 sends the utterance ID of the utterance text to the server through the communication unit 23 in association with the user ID and the video content ID. 1 (step S212).

The control unit 11 of the server 1 receives the user ID, the video content ID, and the speech ID transmitted from the terminal 2 through the communication unit 13 (step S111). The control unit 11 registers the utterance ID and the registration date and time as one record in the note DB 175 of the large-capacity storage unit 17 in association with the received user ID and video content ID (step S112).

When the control unit 21 of the terminal 2 receives a registration cancellation operation for the registered utterance text by the input unit 24 (step S213), the utterance ID of the utterance text is associated with the user ID and the video content ID by the communication unit 23. It is transmitted to the server 1 (step S214).

The control unit 11 of the server 1 receives the user ID, the video content ID, and the utterance ID of the utterance text transmitted from the terminal 2 through the communication unit 13 (step S113). Based on the received user ID, video content ID, and speech ID, the control unit 11 deletes the corresponding speech text record from the note DB 175 (step S114).

According to this embodiment, it is possible to display the spoken text in the moving image content in synchronization with the reproduction of the moving image content.

According to this embodiment, by displaying the video content and the spoken text of the video content for each category set based on the video content, it becomes easier for the user to search for the video content that matches the interest or needs of the user. It is possible to arouse motivation to watch video content.

According to this embodiment, by displaying the spoken text in the video content, it is possible for the user to roughly understand the content of the video content without viewing the video content.

According to this embodiment, it is possible to register or cancel the registration of the spoken text in the video content.

(Embodiment 2)
Embodiment 2 relates to a form in which registered speech texts are displayed in a list. In addition, description is abbreviate|omitted about the content which overlaps with Embodiment 1. FIG.

When the spoken texts in the video content are registered, the registered spoken texts can be displayed in a list. Specifically, based on the user ID, the terminal 2 acquires from the notebook DB 175 of the server 1 the moving image content ID of the viewed moving image content, the utterance ID of the registered utterance text in the moving image content, and the registration date and time. The terminal 2 acquires the start point of display of the utterance text from the utterance text DB 172 of the server 1 based on the acquired utterance ID of the registered utterance text. The terminal 2 displays a list of the acquired registered speech texts and the display start time points of the speech texts by moving image content viewed by the user or in the order of registration of the speech texts.

When registered utterance texts are displayed for each moving image content, the terminal 2 transmits information (thumbnail image, title, etc.) about each moving image content to the moving image content of the server 1 based on the moving image content ID of each viewed moving image content. Acquired from the DB 171. The terminal 2 displays on the screen the registered utterance text in the moving image content acquired from the note DB 175 in association with the thumbnail image and title of the moving image content for each moving image content.

When the registered speech texts are displayed in the order of registration, the terminal 2 sorts the speech texts in ascending or descending order based on the registration dates and times of the speech texts acquired from the note DB 175 . Terminal 2 displays the sorted speech texts on the screen.

FIG. 13 is an explanatory diagram showing an example of a list screen of spoken texts by moving image content. The screen includes a moving image content tab 15a, a registration order tab 15b, and a moving image information display field 15c. The video content tab 15a is a tab (first tab) for showing a list of registered utterance texts for each video content. The registration order tab 15b is a tab (second tab) for showing a list of registered utterance texts in the order of registration of the utterance texts.

The moving image information display field 15c includes a thumbnail image display field 15c1, a title display field 15c2, a spoken text display field 15c3, a registration button 15c4, and an all button 15c5. The thumbnail image display field 15c1 is a display field for displaying thumbnail images of moving image content. The title display column 15c2 is a display column for displaying the title of the video content. The utterance text display field 15c3 is a display field for displaying the utterance text and the start point of display of the utterance text. The registration button 15c4 is a button for registering or canceling the registration of the spoken text. The all button 15c5 is a button for transitioning to a display screen (FIG. 14) of all spoken texts in the video content.

Based on the user ID, the terminal 2 acquires the moving image content ID of the viewed moving image content, the registered utterance text in the moving image content, and the date and time of registration from the notebook DB 175 of the server 1 . The terminal 2 acquires the start point of display of the utterance text from the utterance text DB 172 of the server 1 based on the acquired utterance ID of the registered utterance text. The terminal 2 acquires information (thumbnail image, title, etc.) about each moving image content from the moving image content DB 171 of the server 1 based on the moving image content ID of each viewed moving image content.

The terminal 2 associates the moving image content with the spoken text in the moving image content for each moving image content, and displays them side by side in the vertical direction in each moving image information display column 15c. If the plurality of moving image information display columns 15c arranged in the vertical direction cannot be displayed on the screen, the plurality of moving image information display columns 15c can be displayed so as to be scrollable in the vertical direction.

Specifically, the terminal 2 displays the thumbnail image of each moving image content in the corresponding thumbnail image display column 15c1, and displays the title of each moving image content in the corresponding title display column 15c2. The terminal 2 displays the speech text in each moving image content and the display start time of the speech text in the corresponding speech text display field 15c3.

As illustrated, three speech text display columns 15c3 are provided in one moving image information display column 15c. For example, the terminal 2 displays the first to third speech texts in the speech text display fields 15c3 in ascending order of the speech IDs of the speech texts. Note that the terminal 2 may randomly extract three utterance texts from a plurality of utterance texts and display them in each utterance text display field 15c3. The number of utterance text display fields 15c3 in the moving image information display field 15c may be provided according to the user's needs.

When the terminal 2 receives the touch operation of the registration button 15c4 for the registered utterance text, the terminal 2 performs registration cancellation processing of the utterance text. Specifically, the terminal 2 transmits the user ID, the video content ID, and the speech ID of the speech text to the server 1 . The server 1 deletes the corresponding record from the note DB 175 based on the user ID, the video content ID, and the speech ID of the speech text transmitted from the terminal 2 . The terminal 2 changes the corresponding registration button 15c4 to the unregistered state. For example, the terminal 2 changes the background color of the registration button 15c4 to gray.

Further, when the terminal 2 receives again the touch operation of the registration button 15c4 for the unregistered speech text, the terminal 2 performs registration processing of the speech text. Specifically, the terminal 2 transmits the user ID, the video content ID, and the speech ID of the speech text to the server 1 . The server 1 receives the user ID, the video content ID, and the speech ID of the speech text transmitted from the terminal 2 . The terminal 2 registers the utterance ID and the registration date and time of the utterance text in the note DB 175 as one record in association with the received user ID and video content ID. The terminal 2 changes the corresponding registration button 15c4 to the registered state. For example, the terminal 2 changes the background color of the registration button 15c4 to red.

When the terminal 2 accepts the touch operation of the all button 15c5, the terminal 2 transitions to the display screen (FIG. 14) of all the spoken texts in the video content described later.

FIG. 14 is an explanatory diagram showing an example of a display screen of all spoken texts in video content. Note that the same reference numerals are assigned to the contents that overlap with those in FIG. 13, and the description thereof is omitted. As illustrated, the terminal 2 displays the thumbnail image of the video content in the thumbnail image display field 15c1, and displays the title of the video content in the title display field 15c2.

The terminal 2 displays each utterance text in the moving image content and the start point of display of each utterance text in each utterance text display field 15c3. The display order of the speech texts may be, for example, in ascending order of the speech IDs of the speech texts, or may be in order of newest registration date and time of the speech texts. Note that the registration or deregistration processing of the registration button 15c4 is the same as the processing in FIG. 13, so description thereof will be omitted.

FIG. 15 is an explanatory diagram showing an example of a list screen of utterance texts in order of registration. The screen includes a speech text display field 16a and a registration button 16b. The utterance text display field 16a is a display field for displaying the utterance text and the start point of display of the utterance text. The registration button 16b is a button for registering or canceling the registration of the spoken text.

Based on the user ID, the terminal 2 acquires the registered utterance texts from the notebook DB 175 of the server 1 in order of registration date and time of the utterance texts (in order of registration). The terminal 2 acquires the start point of display of the utterance text from the utterance text DB 172 of the server 1 based on the utterance ID of the acquired utterance text. The terminal 2 displays each acquired utterance text and the display start point of the utterance text in the corresponding utterance text display field 16a in chronological order of registration date and time of the utterance text. The terminal 2 changes each registration button 16b to the registered state. For example, the terminal 2 changes the background color of the registration button 16b to red.

Also, the registration date and time of the spoken text may be displayed in the spoken text display field 16a. In addition, the terminal 2 may set the font color of the spoken text or the background color of the spoken text display area (the spoken text display field 16a) in accordance with the color set for each category. Furthermore, when the terminal 2 accepts the touch operation of the spoken text, for example, the terminal 2 transitions to the detailed screen of the video content (FIG. 8), and reproduces the video content at the playback timing corresponding to the display timing of the spoken text. can be

If the plurality of vertically arranged utterance text display fields 16a do not fit on the screen, the plurality of utterance text display fields 16a can be displayed in a vertically scrollable manner. Note that the registration or deregistration processing of the registration button 16b is the same as the processing in FIG. 13, so description thereof will be omitted.

FIG. 16 is a flow chart showing a processing procedure for displaying a list of spoken texts. The control unit 21 of the terminal 2 receives the display type of the list of spoken texts through the input unit 24 (step S221). The display types include "by video content" for displaying a list of spoken texts by video content, and "registration order" for displaying a list of spoken texts in the order of registration. The control unit 21 transmits the user ID and the accepted display type to the server 1 through the communication unit 23 (step S222).

The control unit 11 of the server 1 receives the user ID and display type transmitted from the terminal 2 through the communication unit 13 (step S121). The control unit 11 determines whether or not the received display type is "by video content" (step S122). When the control unit 11 determines that the display type is “by moving image content” (YES in step S122), based on the received user ID, information about each viewed moving image content (thumbnail image and title of the moving image content) is displayed. etc.), and registered utterance texts in each moving image content are obtained from the moving image content DB 171 and the note DB 175 of the large-capacity storage unit 17 (step S123).

The control unit 11 transmits the acquired information about each moving image and the registered utterance text in each moving image content to the terminal 2 through the communication unit 13 (step S124). The control unit 21 of the terminal 2 receives, through the communication unit 23, the information about the video content and the registered speech text transmitted from the server 1 (step S225). The control unit 21 displays the thumbnail image, the title, and the registered utterance text of the received moving image content for each moving image content (step S226), and ends the process.

When the control unit 11 of the server 1 determines that the display type is not “by video content” (NO in step S122), based on the user ID, acquires the registered speech text from the note DB 175 of the large-capacity storage unit 17. (step S125). The control unit 11 transmits the acquired registered speech text to the terminal 2 through the communication unit 13 (step S126). The control unit 21 of the terminal 2 receives the registered speech text transmitted from the server 1 through the communication unit 23 (step S223). The control unit 21 causes the display unit 25 to display the received utterance texts in order of the registration date and time of the utterance texts (in order of registration) (step S224), and ends the process.

In addition, it is possible to switch between the video content list screen and the spoken text list screen.
FIG. 17 is an explanatory diagram showing an example of the switching screen. The screen includes a video list button 17a and a spoken text list button 17b. The moving image list button 17a and the spoken text list button 17b are fixedly displayed in a partial area of the display screen (for example, the bottom of the screen). The video list button 17a is a button for transitioning to a video content list screen (first screen). The speech text list button 17b is a button for transitioning to a speech text list screen (second screen).

When the terminal 2 receives the touch operation of the moving image list button 17a, the terminal 2 transitions to the moving image content list screen and displays the moving image content list screen. When the terminal 2 receives the touch operation of the spoken text list button 17b, the terminal 2 transitions to the spoken text list screen and displays the spoken text list screen.

According to this embodiment, by displaying a list of registered speech texts by moving image content or in the order of registration, it is possible to view the speech texts at any time.

The embodiments disclosed this time are illustrative in all respects and should be considered not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the above-described meaning, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

1 Information processing device (server)
REFERENCE SIGNS LIST 11 control section 12 storage section 13 communication section 14 input section 15 display section 16 reading section 17 large capacity storage section 171 video content DB
172 Utterance Text DB
173 User database
174 History DB
175 Note DB
176 Category DB
1a portable storage medium 1b semiconductor memory 1P control program 2 information processing terminal (terminal)
21 control unit 22 storage unit 23 communication unit 24 input unit 25 display unit 26 speaker 2P control program

Claims (12)

Acquiring a video content and an utterance text describing the utterance content of a speaker in the video content,
displaying the spoken text step by step according to the playback of the acquired video content;
transmitting the spoken text to an information processing device when a registration operation for the displayed spoken text is received ;
A first tab showing a list of the registered speech texts by the registration operation for each video content viewed by the user, and a second tab showing a list of the registered speech texts by the registration operation in order of registration of the speech texts. to display
displaying a list of the registered utterance texts for each video content when an operation for selecting the first tab is received;
Displaying a list of the spoken texts in the order of registration when the selection operation of the second tab is accepted
A program that causes a computer to carry out a process. For each uttered text in the video content, a first object for reproducing at the reproduction timing of the video content corresponding to the display timing of the uttered text, a second object for registering the uttered text, and the uttered text. 2. The program according to claim 1, causing execution of a process of operably displaying a third object for sharing. 3. The program according to claim 1 or 2, which executes a process of switching between display and non-display of the spoken text according to a user's operation while the moving image content is being reproduced. obtaining a plurality of registered utterance texts in the video content;
4. The program according to any one of claims 1 to 3, which causes execution of a process of displaying a list of the acquired plurality of spoken texts. When the selection operation of the first tab is accepted, for each video content, a thumbnail image and title of each video content, registered utterance text in the video content, and a fourth object indicating the registered state of the utterance text to display
5. The method according to any one of claims 1 to 4 , wherein a process of canceling the registered state of the utterance text is executed when an operation input of the fourth object for canceling the registered state of the utterance text is received. program. fixedly displaying a fifth object for displaying the video content list and a sixth object for displaying the spoken text list in a partial area of the display screen;
When an operation input to the fifth object is received, transitioning to a first screen displaying a list of the video content,
6. The method according to any one of claims 1 to 5 , wherein when an operation input to the sixth object is accepted, a process of transitioning to a second screen displaying a list of the registered utterance texts including the first tab and the second tab is executed. The program according to any one . 7. The program according to any one of claims 1 to 6 , causing execution of a process of arranging and displaying the moving image content and the spoken text in the moving image content in association with each other for a plurality of the moving image contents. Multiple video content is registered for each category set based on video content,
Displaying the display areas classified by category and displayed in different colors in a first direction,
8. The display area according to any one of claims 1 to 7 , wherein the video content belonging to the category and the spoken text in the video content are associated with each other and displayed side by side in the second direction. program. Get multiple video contents belonging to the category,
Acquire multiple spoken texts in each acquired video content,
9. The program according to any one of claims 1 to 8 , which executes a process of switching and displaying a plurality of spoken texts in each video content by a swipe operation. Acquiring a video content and an utterance text describing the utterance content of a speaker in the video content,
displaying the spoken text step by step according to the playback of the acquired video content;
registering the spoken text based on a registration operation for the displayed spoken text;
A first tab showing a list of the registered speech texts by the registration operation for each video content viewed by the user, and a second tab showing a list of the registered speech texts by the registration operation in order of registration of the speech texts. to display
displaying a list of the registered utterance texts for each video content when an operation for selecting the first tab is received;
Displaying a list of the spoken texts in the order of registration when the selection operation of the second tab is accepted
Information processing methods. Acquiring a video content and an utterance text describing the utterance content of a speaker in the video content,
displaying the spoken text step by step according to the playback of the acquired video content;
transmitting the spoken text to an information processing device when a registration operation for the displayed spoken text is received ;
Multiple video content is registered for each category set based on video content,
Displaying the display areas classified by category and displayed in different colors in a first direction,
In each of the display areas, video content belonging to a category and registered utterance text in the video content by the registration operation are associated and displayed side by side in the second direction.
A program that causes a computer to carry out a process. Acquiring a video content and an utterance text describing the utterance content of a speaker in the video content,
displaying the spoken text step by step according to the playback of the acquired video content;
transmitting the spoken text to an information processing device when a registration operation for the displayed spoken text is received;
Multiple video content is registered for each category set based on video content,
Get multiple video contents belonging to the category,
Obtaining a plurality of registered utterance texts by the registration operation in each obtained video content,
Switch and display multiple spoken texts in each video content by swiping
A program that causes a computer to carry out a process.

Images