JP5627030B2 - Related information providing method, related information providing apparatus, and related information providing program - Google Patents

Description

  The present invention relates to a related information providing method, related information providing apparatus, and related information providing program for providing related information of video data.

  With the development of information devices and communication networks, users have increased opportunities to browse video data. In addition, when browsing video data, there is an increasing opportunity to acquire information related to video data, such as characters and introduced topics, who are interested in video content via a computer or a communication network.

  In order to meet such user needs, digital broadcasting provides related information of programs as data broadcasting during the broadcasting of the programs. Also, related information is provided in the same manner for users who wish to view related information even after the broadcast ends.

  Here, there is a method that enables acquisition of related information at a predetermined timing by using a data signal multiplexed in broadcasting (see, for example, Patent Document 1). In the method described in Patent Document 1, an image frame identifier or an audio frame identifier is superimposed on video data as a data signal. The user can acquire the related information using the image frame identifier or the audio frame identifier at a desired timing.

Japanese Patent No. 4494389

  However, in the method described in Patent Document 1, it is necessary to create in advance a data signal such as an image frame identifier or an audio frame identifier indicating that related information exists during digital broadcasting, and define it in the signal of the data broadcasting. is there. In addition, a broadcaster provides an image frame identifier or an audio frame identifier indicating that related information exists to the related information content providing server, and the related information content providing server needs to define related information at this timing in advance. .

  As described above, in the conventional method, it is necessary for the service provider to prepare in advance before broadcasting, which places a burden on the service provider.

  Accordingly, an object of the present invention is to provide a related information providing method, a related information providing apparatus, and a related information providing program for easily providing related information at a predetermined timing of video data.

In order to solve the above problem, a first feature of the present invention relates to a related information providing method for providing related information of video data. That is, in the related information providing method according to the first aspect of the present invention, the related information providing apparatus receives a request for related information of a video including the identifier of video data and a playback time from the user terminal, and the video information at the playback time. The information request detecting step and the related information providing device acquire text data corresponding to the identifier and playback time of the video data included in the request from the video data, and the proper noun included in the acquired text data is determined as the video data. The text information analysis step that is extracted as a keyword of the reproduction time of the video information , and the related information providing apparatus associates the identifier and the reproduction time of the video data with the keyword extracted in the text information analysis step, and identifies this correspondence. Add keyword identifiers and store keyword updates stored in keyword data. A step, related information providing apparatus, the keyword data, and a keyword selection step of obtaining a keyword corresponding identifier and the keyword corresponding to the identifier and the reproduction time of the video data received in the context information request detection step, related information providing apparatus And a related information output step of outputting related information related to the keyword selected in the keyword selecting step to the user terminal.

Here, the related information providing device acquires partial video data corresponding to the identifier and playback time of the video data included in the request, and the feature amount of the part where the person is imaged in the partial video data, the person name, A video information analysis step of comparing the feature quantity of the person feature data associated with the feature quantity of the person's image and extracting the person name included in the partial video data as a keyword of the playback time of the video data May be further provided. In this case, the keyword update step further associates the identifier corresponding to the keyword, the identifier of the video data, the reproduction time of the video data, and the keyword extracted in the video information analysis step, and stores them in the keyword data.

  In the keyword update step, the keyword-corresponding identifier may be further associated with the degree of matching of the keyword at the reproduction time of the video data and stored in the keyword data. In this case, the related information output step outputs the related information related to the keyword selected in the keyword selection step to the user terminal in association with the keyword corresponding identifier, and when the related information is selected in the user terminal, When the keyword data is updated to increase the fitness corresponding to the identifier corresponding to the keyword, and the keyword data has a plurality of keywords associated with the identifier of the video data and the playback time, the fitness of the keywords is Relevant information related to high keywords is output preferentially.

  In the keyword update step, the keyword correspondence identifier may be associated with the video relevance and stored in the keyword data. In this case, when the text information analysis step extracts the keywords corresponding to the video data and the playback time, if the extracted keywords are included in the synopsis information of the video data, the video relevance is increased so that the video relevance is increased. In the related information output step, when a plurality of keywords are associated with the identifier of the video data and the playback time in the keyword data, the related information related to the keyword having high video relevance among the plurality of keywords is prioritized. To output automatically.

  A second feature of the present invention relates to a related information providing apparatus that provides related information of video data. That is, the related information providing apparatus according to the second feature of the present invention includes a related information request detecting unit that includes a video data identifier and a playback time from a user terminal, and receives a request for video related information at the playback time; Text information analysis that acquires text data corresponding to the identifier and playback time of the video data included in the request from the video data, and extracts proper nouns included in the acquired text data as keywords of the playback time of the video data A keyword updating unit that associates the identifier, the playback time of the video data, and the keyword extracted by the text information analysis unit with a keyword correspondence identifier that identifies this association, and stores the keyword in the keyword data From the keyword data, the identifier of the video data received by the related information request detection unit and the Includes a keyword selection unit that acquires a keyword corresponding identifier and the keyword corresponding to the time, the relevant information related to the keyword selected by the keyword selection unit, the related information outputting section for outputting to the user terminal.

  Here, a storage device that stores person feature data in which a person name is associated with an image feature amount of the person, and a partial video data corresponding to an identifier and a playback time of video data included in the request are acquired, In the partial video data, the feature amount of the portion where the person is imaged is compared with the feature amount stored in the person feature data, and the person name included in the partial video data is determined as the reproduction time keyword of the video data. A video information analysis unit that extracts the information may be further provided. In this case, the keyword update unit further associates the identifier corresponding to the keyword, the identifier of the video data, the reproduction time of the video data, and the keyword extracted by the video information analysis unit, and stores them in the keyword data.

  In addition, the keyword update unit may store the keyword correspondence by further matching the keyword matching identifier with the identifier corresponding to the keyword in the reproduction time of the video data. In this case, the related information output unit outputs the related information related to the keyword selected by the keyword selection unit to the user terminal in association with the keyword corresponding identifier, and when the related information is selected at the user terminal, In addition to updating to increase the fitness level corresponding to the identifier corresponding to the keyword of the data, and in the keyword data, when a plurality of keywords are associated with the identifier and playback time of the video data, Preferentially output related information related to high keywords.

  A third feature of the present invention relates to a related information providing program for causing a computer to execute the steps described in any of the first features of the present invention.

  According to the present invention, it is possible to provide a related information providing method, a related information providing apparatus, and a related information providing program that easily provide related information at a predetermined timing of video data.

It is a figure explaining the system configuration | structure of the related information provision system which concerns on embodiment of this invention, and the functional block of a related information provision apparatus. It is a figure explaining the structure of the keyword data which concerns on embodiment of this invention, and an example of the data. It is a figure explaining an example of the structure of the user data which concerns on embodiment of this invention, and its data. It is a flowchart explaining the related information provision method by the related information provision apparatus which concerns on embodiment of this invention. It is a flowchart explaining the related information output process by the related information output part of the related information provision apparatus which concerns on embodiment of this invention. It is a figure explaining the structure of the image | video metadata which concerns on embodiment of this invention, and an example of the data. It is a figure explaining an example of the structure of the person characteristic data which concerns on embodiment of this invention, and its data. It is a flowchart explaining the video content analysis process by the video content analysis part of the related information provision apparatus which concerns on embodiment of this invention.

  Next, embodiments of the present invention will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

  The related information providing system 2 according to the embodiment of the present invention includes a related information providing device 1, video viewing devices 100 and 102, and user terminals 101 and 103, as shown in FIG. The related information providing apparatus 1, the video viewing apparatuses 100 and 102, and the user terminals 101 and 103 are connected via a communication network 110 so that they can communicate with each other. The communication network 110 is a network that can communicate with each other, such as the Internet.

  The video viewing device 100 is a device that allows the user A to view digital broadcast video data, video data provided by a video data providing server, and the like. Specifically, the video viewing apparatus 100 is a television, a personal computer, a smartphone, a mobile phone, or the like. The video viewing apparatus 100 may be a set-top box and a television that are used by receiving a pay broadcast and connected to a television or the like, or may be a television with a built-in function capable of receiving the pay broadcast.

  The video viewing apparatus 100 according to the embodiment of the present invention, in particular, when a user inputs a predetermined operation during playback of video data, the identifier of the video data being played back at the time of input, the playback time of the video data, Is transmitted to the related information providing apparatus 1 via the communication network. Here, the reproduction time of the video data is the reproduction time that has elapsed from the beginning of the video data in the case of storage-type video data such as recorded video data or video data downloaded from a video distribution server. In the case of video data such as digital broadcast broadcast in real time, the time may be used. The reproduction time may be in any format as long as the user's operation timing is known in the video data.

  The video viewing device 102 is a device that allows the user B to view video data and the like, and its function is the same as that of the video viewing device 100.

  The user terminal 101 is a terminal used by the user A, and is, for example, a personal computer, a smartphone, a mobile phone, or the like. The user terminal 101 is a terminal that receives a mail or the like and can browse a digest of related information input from the related information providing apparatus 1 using a browser or the like, or detailed related information.

  The video viewing device 102 is a device that allows the user B to view video data and the like, and its function is the same as that of the video viewing device 100. The user terminal 103 is a terminal used by the user B, and its function is the same as that of the user terminal 101.

  The related information providing apparatus 1 is an apparatus that provides related information of video data at a timing designated by a user.

  In the embodiment of the present invention, a user inputs a request for related information related to video content while browsing video data, for example, to the video viewing device 100 by a predetermined operation. At this time, the video viewing apparatus 100 transmits the identifier of the video data being reproduced at the time of operation and the reproduction time to the related information providing apparatus 1 together with the user identifier. This “predetermined operation” may be, for example, pressing of a button provided on the remote control of the video viewing apparatus 100 or a user operation on the user terminal 101 that implements the remote control function of the video viewing apparatus 100.

  When the user terminal 101 implements the remote control function, the user terminal 101 may transmit the identifier and the reproduction time of the video data being reproduced at the time of operation to the related information providing apparatus 1. Here, when the user terminal 101 implements the remote control function, the video data currently being reproduced may be specified based on the operation history of the user terminal 101 by the user. Further, when the user requests related information while viewing the video data recorded by the user himself / herself on the video viewing apparatus 100, the video viewing apparatus 100 associates the identifier of the video data from the metadata held at the time of recording. You may transmit to the information provision apparatus 1. FIG.

  When the related information providing apparatus 1 receives the identifier and the reproduction time of the video data, it analyzes the video data reproduced at that time and extracts keywords. The related information providing apparatus 1 accesses a search server (not shown) based on the extracted keyword, and provides the search result acquired by the search server to the user as related information on the keyword.

  There are various ways to provide the relevant results. For example, the related information providing apparatus 1 holds the user's mail address in advance, notifies the user that the related information has been acquired, and transmits a mail including a link such as a My Page to the user terminal 101. In the embodiment of the present invention, this link is a link corresponding to the user's My Page in the related information providing apparatus 1. When the user terminal 101 receives the mail, the user terminal 101 accesses this link, and the user can view the digest page of the related information requested by the user. Further, the user can select related information of interest from the digest page and know details of the related information.

(Related information provision device)
A related information providing apparatus 1 according to an embodiment of the present invention will be described with reference to FIG. The related information providing device 1 is a general computer including a storage device, a processing control device, a communication control device, and the like. The related information providing apparatus 1 is implemented with each unit shown in FIG. 1 by executing a related information providing program for executing a predetermined process on a general computer.

  The storage device of the related information providing device 1 includes a video metadata storage unit 11, a video data storage unit 12, a person feature data storage unit 13, a keyword data storage unit 31, and a user data storage unit 32. The processing control apparatus includes a text information analysis unit 21, a video information analysis unit 22, a fixed keyword input unit 23, a keyword update unit 24, a related information request detection unit 41, a keyword selection unit 42, a related information acquisition unit 43, and a related information output unit. 44 and a new related information notification unit 45.

  Here, the video metadata storage unit 11, the video data storage unit 12, the person feature data storage unit 13, the text information analysis unit 21, the video information analysis unit 22, the fixed keyword input unit 23, and the keyword update unit 24 are collectively referred to as Sometimes referred to as video content analysis unit 10. The video content analysis unit 10 registers a new keyword in the keyword data 31 a stored in the keyword data storage unit 31.

  The keyword data storage unit 31 is a storage area for storing the keyword data 31a. As shown in FIG. 2, the keyword data 31a is data in which a video data identifier, a reproduction time, a keyword, video relevance, and a fitness level are associated with each other and a keyword corresponding identifier for identifying this association is attached.

  The keyword correspondence identifier specifies a record in the keyword data 31a. The keyword correspondence identifier is an identifier that uniquely identifies a combination of a video data identifier, a reproduction time, and a keyword.

  The video data identifier and playback time are the identifier and playback time of video data that was played back when the user operated the request for related information. The identifier and playback time of the video data correspond to the identifier and playback time of the video data included in the related information request sequentially input from the user. In the example illustrated in FIG. 2, the playback time is the same time as the timing requested by the user, but is not limited thereto. For example, the time length of the video data may be expressed by a number when divided by 30 seconds and numbered, or 30 seconds before and after the time requested by the user may be set.

  The keyword is a keyword related to the video data of the reproduction time in the video data specified by the video data identifier. The keywords are extracted by the video content analysis unit 10. Here, the keyword may be extracted for the video data at the time indicated by the playback time. The keyword may be extracted based on video data near the playback time, such as 30 seconds before and after the playback time.

  The video relevance is an index indicating that the keyword is related to the video data. The video relevance is calculated by the text information analysis unit 21 or the video information analysis unit 22 of the video content analysis unit 10. In the embodiment of the present invention, the case where the video relevance is appropriately expressed by an arbitrary numerical value will be described. However, the video relevance may be expressed by “Yes (1)” or “No (0)”. .

  The fitness is an index indicating the appropriateness of the keyword in the playback time of the video data. When the user browses related information retrieved based on this keyword, the keyword in the reproduction time of the video data is considered appropriate. Therefore, the more relevant information is browsed by the user, the higher the fitness is. As described above, the fitness is appropriately changed by a user operation or the like.

  The user data storage unit 32 is a storage area for storing user data 32a. As shown in FIG. 3, the user data 32a is data in which a user identifier, a user's mail address, and a terminal type are associated with each other. The user's e-mail address is an address used when notifying the user that preparation for provision of related information is complete. The type of terminal is the type of terminal from which the user receives mail, and is retained when creating a mail in consideration of restrictions on the terminal and available characters.

  For users who do not like user notification, the user data storage unit 32 may be provided with a flag for setting whether to notify the user. In this case, a null value may be set in the mail address field.

  The related information request detection unit 41 receives, from the user terminal 101, a request for video related information including the identifier of the video data and the playback time. Here, the identifier of the video data may be in any form as long as the video data can be identified. For example, when the video data is video content data distributed by the video server, the video data identifier is a combination of the video server identifier and the identifier attached to the video data in the video server. In the case of video content data for data broadcasting, the video data identifier is a combination of a channel identifier and broadcast time. Further, the playback time may take any form as long as the timing when the user requests the related information is known.

  The request also includes a user identifier, and the related information request detection unit 41 also receives the user identifier. The user identifier is for recognizing which user is the source of the request and providing related information to the user.

  The keyword selection unit 42 acquires an identifier and a keyword corresponding to the keyword corresponding to the identifier and playback time of the video data received by the related information request detection unit 41 from the keyword data 31a. The keyword selection unit 42 searches the keyword data 31a for a keyword corresponding to the identifier and playback time of the video data received by the related information request detection unit 41. At this time, the keyword selection unit 42 may set a certain length of time, and search for a record in which the reproduction time associated with the received video data identifier is included in the set time length. For example, considering that it takes time for a user to request related information, the keyword selection unit 42 may search for a keyword in a range from 60 seconds before the received reproduction time to 10 seconds after the reproduction time. good.

  The keyword selection unit 42 associates the user identifier received from the related information request detection unit 41, the keyword searched from the keyword data 31a, and the keyword corresponding identifier, and inputs them to the related information acquisition unit 43. Further, the keyword selection unit 42 may input the video relevance and the fitness level related to the acquired keyword to the related information acquisition unit 43.

  If the keyword cannot be selected, specifically, if the keyword corresponding to the identifier and the playback time of the received video data is not registered in the keyword data 31a, the keyword selection unit 42 sends the video content analysis unit 10 Register keywords. At this time, the keyword selection unit 42 inputs the identifier and playback time of the video data input from the related information request detection unit 41 to the video content analysis unit 10. When a keyword is registered in the keyword data 31 a by the video content analysis unit 10, the keyword selection unit 42 selects a keyword and associates the selected keyword with a keyword correspondence identifier and inputs the keyword to the related information acquisition unit 43. This keyword correspondence identifier is used in the keyword data 31a to uniquely identify the identifier and playback time of the video data included in the request and the selected keyword.

  When the keyword selection unit 42 selects a large number of keywords from the keyword data 31 a, the keyword selection unit 42 may input information related to some keywords of the large number of keywords to the related information acquisition unit 43. For example, in the keyword data 31a, a plurality of keywords corresponding to the playback time included in the request may be registered. In that case, the number of keywords selected by the keyword selection unit 42 may be determined as m in advance, and m keywords having a high degree of fitness may be selected from a large number of keywords. In addition, when there are no more than m extracted keywords, information regarding all the keywords may be input to the related information acquisition unit 43.

  Here, since the fitness level is data changed by a user operation, the fitness level value may not be valid when the number of accesses from the user is small. Therefore, the keyword selection unit 42 selects m keywords using the video relevance when the number of accesses from the user is small, and selects m keywords using the fitness when the number of accesses exceeds a predetermined number. It is preferable to do.

  For example, when a request for a predetermined number or more of related information is received with respect to predetermined video data and playback time in digital broadcasting or the like, the keyword selection unit 42 has the highest degree of matching received from the keyword data 31a. Select information about keywords. Further, the keyword selection unit 42 may input information on the selected m keywords to the related information acquisition unit 43.

  Moreover, the keyword selection part 42 may determine the keyword input into the related information acquisition part 43 according to each user. For example, when the search history of the user's web can be used, if there is a keyword that matches the search history query, the keyword may be preferentially adopted and output to the related information acquisition unit 43.

  The related information acquisition unit 43 searches for information related to the keyword selected by the keyword selection unit 42, and holds the related information in association with the user identifier input from the keyword selection unit. The related information acquisition unit 43 has, for example, a web search function, and acquires related information using a keyword input from the keyword selection unit 42. The related information acquisition unit 43 outputs at least a part of the acquired related information to the related information output unit 44 as related information to be provided to the user. Further, the related information acquisition unit 43 notifies the new related information notification unit 45 that the related information has been acquired.

  The related information acquisition unit 43 may input a keyword using an API such as an existing search server and acquire the search result as related information of the keyword.

  As shown in FIG. 2, the related information acquisition unit 43 receives the scene of the video data identifier “crid: //aaa.co.jp/content/000001” and the playback time “04:05:06” from the keyword selection unit 42. Suppose that three keywords “Mariya Ryokan”, “Ashinoko”, and “Reiko Sonoda” are input. In this case, the related information acquisition unit 43 uses the top two search results for “Maruya Ryokan”, the top two search results for “Lake Ashinoko”, and the top two search results for “Reiko Sonoda” as related information. get. Here, when the related information acquired about each keyword overlaps, the related information acquisition unit 43 does not perform filtering of the related information, and the related information output unit 44 does not display the same related information multiple times. Display screen is generated.

  When the related information acquisition unit 43 acquires a large number of search results, the related information acquisition unit 43 may perform filtering by acquiring only the top n rankings of search results as related information. Here, n may be a variable number depending on the degree of matching of the keyword. For example, the related information acquisition unit 43 employs the top two search results for the top three keywords with the highest degree of matching as the related information, and sets the top one of the search results for the keywords with the lower degree of suitability. It may be adopted as related information.

  For example, the top three keywords with the highest degree of fitness are acquired as related information, of which the top two related information for the keyword with the highest related information, and the top one related information for the other keywords. Think about getting it. In the example shown in FIG. 2, “Mariya Ryokan” has a fitness of “0.9542”, “Ashinoko” has a fitness of “0.6464”, and “Sonoda Reiko” has a fitness of “0.3333”. is there. Therefore, the related information acquisition unit 43 sets the top two search results as related information for “Maruya Ryokan” having the highest fitness. The related information acquisition unit 43 sets the top one of the search results as related information for the other “Lake Ashinoko” and “Reiko Sonoda”.

  Similarly, even when the video relevance is used, the related information acquisition unit 43 employs the top two search results for the keyword having the video relevance “1” as the related information, and the video relevance is “ For the keyword “0”, the top one of the search results may be adopted as related information.

  For example, three keywords are acquired as related information, of which the top two related information is acquired for the keyword with the video relevance “1”, and the top one related information is acquired for the other keywords. Think about the case. In the example shown in FIG. 3, the video relevance of “Mariya Ryokan” is “1”, and the relevance of “Lake Ashinoko” and “Reiko Sonoda” is “0”. Accordingly, the related information acquisition unit 43 sets the top two search results as related information for “Maruya Ryokan” having a video relevance of “1”. The related information acquisition unit 43 sets the top one of the search results as related information for the other “Lake Ashinoko” and “Reiko Sonoda”.

  In this manner, which of the related information searched for the keyword is output by the related information acquisition unit 43 is determined based on the video relevance or the fitness. Here, the fitness is a valid value when many users browse and access. Therefore, when a certain number of accesses has not been obtained, the related information acquisition unit 43 filters the related information to be output based on the video relevance, and after obtaining the certain number of accesses, based on the fitness level. The related information to be output may be filtered.

  The related information acquisition unit 43 filters the related information of each keyword and inputs it to the related information output unit 44, thereby ensuring readability by the user. The related information acquisition unit 43 inputs information such as the title of the related information, the link of the related information, and the snippet together with the keyword, the user identifier, and the keyword correspondence identifier to the related information output unit 44 for the acquired related information. For example, the related information acquisition unit 43 may provide information such as {0000000001, AE2309841F, “Maruya Ryokan Website”, http: //www.maruyar.co.xx/, “Maruya Ryokan was founded in 1952…”} Output to the output unit 44. At this time, the related information output unit 44 inputs the user identifier to the new related information notification unit 45 in order to notify that the related information can be newly provided.

  When accessed from the user terminal 101, the related information output unit 44 authenticates the user terminal 101 and provides the user terminal 101 with an HTML web page. At this time, the related information output unit 44 generates a web page for displaying the related information acquired by the related information acquisition unit 43, and provides this web page when accessed from the user terminal 101. In the keyword data 31a, the related information output unit 44 does not need to output the related information for all the keywords when a plurality of keywords are associated with the identifier of the video data and the reproduction time. For example, the related information output unit 44 preferentially outputs related information related to a keyword having a high matching degree among a plurality of keywords. It should be noted that the related information output unit 44 or the related information acquisition unit 43 may determine which of the plurality of keywords is to be output related information.

  In the embodiment of the present invention, the related information acquisition unit 43 filters predetermined related information from a plurality of pieces of related information acquired based on one keyword, and the related information output unit 44 sets the related information after filtering. This is realized by acquiring information, but is not limited to this. For example, the related information output unit 44 may acquire all the related information acquired by the related information acquisition unit 43, and the related information output unit 44 may filter the related information.

  The related information output unit 44 outputs related information related to the keyword selected by the keyword selection unit 42 to the user terminal 101 in association with the keyword corresponding identifier of the keyword data 31a. Further, when the related information is selected on the user terminal 101, the related information output unit 44 updates the matching degree corresponding to the keyword corresponding identifier of the keyword data 31a.

For example, when the related information output unit 44 outputs the related information, a link for browsing the related information is provided, and the link includes a keyword correspondence identifier of the keyword data that matches the user's request. Accordingly, when the user selects a link, the related information output unit 44 can acquire the keyword correspondence identifier, and thus can grasp that the degree of matching of the keyword is high. Thus, every time the user selects, the keyword data 31a is updated so that the degree of matching corresponding to the keyword corresponding identifier of the keyword data 31a is increased.

As described above, when the link of the related information is clicked by the user, the related information output unit 44 increments the counter by one for the keyword corresponding identifier corresponding to the related information. Here, a plurality of keywords may be associated with one piece of related information. In this case, the related information output unit 44 links a plurality of keywords to one related information in advance so as to prevent the user from displaying the same related information multiple times. When the user clicks such a link and browses related information, it is preferable to update the keyword data 31a so that the degree of matching is high for each of the associated keywords. When a link in which three keywords are associated with one related information is clicked, the related information output unit 44 sets a counter to 1 for the corresponding identifier of the three keywords corresponding to the related information of this link. / Increment 3 times.

Here, the related information output unit 44 may disclose a keyword that is the basis of the related information clicked by the user to an external system. In an external system, this keyword can be used as a keyword indicating the user's preference for service or product recommendation.

There are several possible methods for the related information output unit 44 to change the degree of matching of the keyword data 31a. For example, a method of calculating the degree of fitness using the ratio of browsing related information can be considered. In the case of this method, for example, (the number of times the related information obtained by the relevant keyword is browsed) / (the total number of times of viewing of the relevant information obtained by the keyword of the relevant scene) is calculated. In the example shown in FIG. 2, it is assumed that the counter for “Mariya Ryokan” is “80”, the counter for the keyword “Lake Ashinoko” is “10”, and the counter for the keyword “Reiko Sonoda” is “5”. In this case, the keyword data 31a may be updated by calculating the fitness of “Maruya Ryokan”, “Ashinoko”, and “Reiko Sonoda” as {0.80, 0.10, 0.05}. .

Here, if the degree of matching of the keyword corresponding identifier corresponding to the related information browsed by the user is increased and the degree of matching of the keyword corresponding identifier corresponding to the related information that the user did not browse is updated, Such a method may be used.

The related information output unit 44 may generate a screen for providing related information in consideration of the type of the user terminal in the user data 32a. For example, as a type of terminal, there are three types of screens for browsing mail: a standard size, a size smaller than the standard size, and a size larger than the standard size. The related information output unit 44 can generate a screen that provides related information in consideration of the user's visibility.

  When notified that the related information is acquired from the related information acquisition unit 43, the new related information notification unit 45 notifies the user terminal 101 to that effect. Specifically, the new related information notifying unit 45 acquires the mail address and terminal type of the user from the user data 32a, and generates a mail text notifying that the related information has been acquired.

  Further, the new related information notifying unit 45 generates a mail in a format corresponding to the type of the terminal with the mail address acquired from the user data 32a as a destination based on the generated mail text. In general, as the user terminal 101, various information devices such as a computer, a tablet terminal, a smartphone, and a mobile phone can be considered, but the size of the display screen is different in each terminal. Therefore, as a type of terminal, for example, there are three types of screens for browsing mail: a standard size, a size smaller than the standard size, and a size larger than the standard size. The new related information notification unit 45 can generate an e-mail in consideration of the user's visibility.

  When the mail is generated, the new related information notification unit 45 transmits the generated mail to the user terminal 101.

  At this time, the mail may notify only that the related information has been acquired, or may include a digest of the related information and provide a link for browsing the related information. This link may include a keyword correspondence identifier of the keyword data 31a that matches the user's request. Also in this case, when the user selects a link, the related information output unit 44 can acquire the keyword corresponding identifier, so that it can be understood that the matching degree of the keyword is high and the matching degree of the keyword data 31a can be updated.

(Related information provision method)
With reference to FIG. 4, the related information provision method which concerns on embodiment of this invention is demonstrated.

  First, in step S <b> 1, the related information providing apparatus 1 receives a related information acquisition request from the user terminal 101. In step S2, the related information providing apparatus 1 acquires the user identifier, the video data identifier, and the reproduction time from the related information acquisition request.

  In step S3, the related information providing apparatus 1 reads the keyword data 31a, and determines whether or not there is a keyword corresponding to the video data identifier and the reproduction time acquired in step S2. If there is a keyword, the process proceeds to step S5. If there is no keyword, in step S4, the related information providing apparatus 1 analyzes the content based on the video data identifier and the playback time, and acquires a keyword corresponding to the playback time of the video data. In step S4, the related information providing apparatus 1 inserts the acquired keyword information into the keyword data 31a, and proceeds to step S5.

  In step S5, the related information providing apparatus 1 acquires the keyword corresponding to the video data identifier and the reproduction time acquired in step S2 and the keyword correspondence identifier from the keyword data 31a. In step S6, the related information providing apparatus 1 acquires related information based on the keyword acquired in step S5, and stores the related information in association with the user identifier and the keyword corresponding identifier.

  In step S <b> 7, the related information providing apparatus 1 acquires a mail address corresponding to the user identifier of the request source from the user data 32 a, and transmits a mail indicating that the related information can be provided to the user terminal 101. In step S8, the related information providing apparatus 1 generates a screen for providing related information. Thereafter, in step S9, the related information providing apparatus 1 provides related information and updates the fitness of the user data 32a based on the access from the user.

  The related information output processing by the related information output unit 44 according to the embodiment of the present invention will be described with reference to FIG. Here, the related information output unit 44 holds, in advance, the keyword correspondence identifier that is the source of outputting the related information together with the related information, in the user terminal 101. When accessed from the user terminal 101, the user identifier is authenticated and related information for the user is displayed. There are various methods for the user to display related information. For example, there is a method of clicking a link described in an email. In addition, there is a method in which the related information output unit 44 displays a link of related information for the user on the web page after the user is authenticated and the user clicks. Furthermore, there is a method in which the user selects the displayed link by accessing the related information output unit 44 via the remote control of the video viewing apparatus 100 or a remote control application. Here, the remote control application is an application having a function of a remote control that operates on a smartphone. In the example illustrated in FIG. 5, processing when the user browses related information will be described based on such a method.

  First, in step S <b> 101, the related information output unit 44 waits for access from the user terminal 101. When accessed from the user terminal 101, the related information output unit 44 displays a link of related information to be displayed to the user. This link is associated with the keyword correspondence identifier that is the source of the output of the related information. When the user presses this link and the related information is browsed on the user terminal 101, in step S103, the related information output unit 44 acquires the keyword correspondence identifier associated with the related information that is pressed and browsed. To do. In step S104, the related information output unit 44 updates the degree of matching corresponding to the keyword correspondence identifier acquired in step S102 so as to increase.

(Video Content Analysis Department)
The processing of the video content analysis unit 10 will be described in detail with reference to FIG. The video content analysis unit 10 includes a video metadata storage unit 11, a video data storage unit 12, a person feature data storage unit 13, a text information analysis unit 21, a video information analysis unit 22, a fixed keyword input unit 23, and a keyword update unit 24. Prepare.

  The video metadata storage unit 11 is a storage area for storing video metadata 11a, 11b, 11c,. The video metadata 11a has one video metadata for each video data, and is identified by a video data identifier. The video metadata 11b, 11c,... Are the same as the video metadata 11a.

  An example of the video metadata 11a will be described with reference to FIG. The video metadata 11a includes data of each item such as video data title, synopsis information, performer information, and the like. Here, the synopsis information is text data representing an outline of video data, and is used when a program guide, a content guide (ECG / EPG), or the like is created. The video metadata 11a shown in FIG. 6 has an XML structure, but is not limited to this structure. For example, the identifier of the video data and the data of each item shown in FIG. 6 may be expressed in a table structure.

  The video data storage unit 12 is a storage area for storing video data. This is a storage area for storing video data. This video data is identified by an identifier of the video data. In the example illustrated in FIG. 1, a case where the related information providing apparatus 1 holds the video data storage unit 12 will be described, but the present invention is not limited thereto. For example, other devices connected via a communication network or the like may be held, and the related information providing device 1 may acquire video data or data necessary for content analysis from this device as necessary.

  The person feature data storage unit 13 is a storage area for storing person feature data 13a. The person feature data 13a is data in which a person name is associated with a feature amount of the person's image. The feature amount of the person image is, for example, a feature amount analyzed from the face image of the person.

  The person feature data 13a is, for example, data as shown in FIG. The person feature data 13a shown in FIG. 7 associates the person identifier, the person name, and the feature amount of the person image. A person whose feature value is registered in the person feature data 13a is a person who is a character or subject of a topic in video data and can be a target for which the user wants related information.

  In FIG. 7, a case is described in which the person feature data 13a holds a person identifier and a person name as separate items, but the person name may be used as a person identifier.

  The text information analysis unit 21 acquires text data corresponding to the identifier and playback time of the video data included in the request from the video data. Here, the text data acquired by the text information analysis unit 21 is subtitle information that is broadcast while being superimposed as subtitle information, or text data of text information that is broadcast as character broadcast.

  The text information analysis unit 21 further extracts proper nouns included in the acquired text data as keywords of the reproduction time of the video data. The text information analysis unit 21 further extracts keywords corresponding to the video data and the playback time. When the extracted keywords are included in the synopsis information of the video data of the video data, the video information is highly relevant. Set relevance.

  The text information analysis unit 21 acquires text information from the video data storage unit 12 based on the video data identifier and playback time included in the request detected by the related information request detection unit 41. The text information analysis unit 21 displays subtitle information that is reproduced at the reproduction time of the video data specified by the request, subtitle information that is reproduced near the reproduction time, or subtitle information that is reproduced at a predetermined time including the reproduction time. , Get as text information.

  Here, the text information analysis unit 21 can refer to the data of the subtitle information before multiplexing as video data. If the subtitle information before multiplexing is associated with the reproduction time, the multiplexing is performed. Subtitle information before conversion is used. In this case, the video metadata 11a holds the URI of the caption information data before multiplexing. The text information analysis unit 21 acquires the URI of the caption information data from the video metadata 11a of the request target video data, and acquires the caption information data based on the URI. The text information analysis unit 21 is the closest to the reproduction time among the subtitle information of the reproduction time, the subtitle information in the vicinity of the reproduction time, or the subtitle information within the predetermined time including the reproduction time, from the acquired subtitle information data. The subtitle information of the playback time is acquired as text information.

  When the subtitle information data before being multiplexed as video data cannot be referred to, the text information analysis unit 21 acquires the requested video data based on the video data identifier included in the request. Further, the text information analysis unit 21 decodes the caption information from the acquired video data, identifies the caption information, and acquires it as text information. As a method for decoding the caption information from the video data, a technique in which the digital broadcast receiver decodes the caption information from a broadcast or distributed transport stream is conceivable.

  When the text data to be requested is acquired, the text information analysis unit 21 performs morphological analysis on the acquired text data, and extracts keywords that can be targets for which the user requests related information. The keyword extracted here is a proper noun, for example. When the video data is an information program or news, the user may be interested in commercial stores, product names, area names, event names, and the like. In the case of dramas and sports programs, the user's interest may be actors or sports players who appear in the dramas. As described above, the object of interest of the user can be a proper noun in many cases. Therefore, the text information analysis unit 21 extracts proper nouns from the acquired text data as objects of user interest. As a proper noun extraction method, for example, an existing method such as IREX (Information Retrieval and Extraction Exercise, http://nlp.cs.nyu.edu/irex/NE/) can be considered.

  Moreover, the text information analysis part 21 may extract nouns other than a proper noun as a user's interest object. For example, in the case of video data dealing with black tea, characteristic words such as nouns that characterize text data, such as the type of black tea such as “Darjeeling”, may be extracted as keywords. . The text information analysis unit 21 may use feature words extracted from the acquired text data as keywords. However, in this case, it is preferable to exclude date information, time expression, money amount expression, and the like from the keyword because they are considered to be less relevant to the user's interest.

  When the text information analysis unit 21 extracts a keyword for the reproduction time, the text information analysis unit 21 calculates video relevance for the extracted keyword. The video relevance is information serving as an index when the user selects related information. Since the video relevance is defined using synopsis information, it is difficult to be an index specialized in playback time, but it can be an index of relevance as a whole video data. In addition, video relevance can serve as an index even when there is little access history or information selection history from the user.

  When calculating the video relevance, the text information analysis unit 21 uses synopsis information. Synopsis information is thought to include terms that are closely related to the overview of the entire video data. Therefore, the text information analysis unit 21 acquires the synopsis information of the requested video data, and extracts proper nouns by performing morphological analysis in the same manner as the caption information of the video data. The text information analysis unit 21 compares the proper noun extracted from the caption information with the proper noun extracted from the synopsis information, and identifies the proper noun included in both the caption information and the synopsis information. The text information analysis unit 21 sets the video relevance of this keyword high for proper nouns included in both.

  For example, the proper nouns extracted from the caption information are “Yokohama”, “Motomachi”, “Okadaya”, “Keihin Tohoku Line”, “Ishikawacho”, and the proper nouns extracted from the synopsis information are “Ykari Katsuta”, Suppose that it was “Akemi Kurita” and “Yokohama”. When the video relevance is higher and the numerical value is larger, the text information analysis unit 21 sets the video relevance for the keyword “Yokohama” as “1” and the video relevance for other keywords as “0”. To do.

  When the related information request detection unit 41 detects a request for related information, the video information analysis unit 22 acquires, from the video data storage unit 12, partial video data corresponding to the video data identifier and playback time included in the request. The video information analysis unit 22 compares the feature quantity of the part in which the person is imaged in the partial video data with the feature quantity stored in the person feature data 13a, and determines the person name included in the partial video data as the video data. As a keyword for the playback time of.

  The video information analysis unit 22 acquires partial video data from the video data storage unit 12 based on the video data identifier and the playback time included in the request detected by the related information request detection unit 41. The video information analysis unit 22 is a video frame that is played back at the playback time of the video data specified by the request, video data that is played back near the playback time, video data that is played back at a predetermined time including the playback time, etc. Are acquired as partial video data.

  The video information analysis unit 22 analyzes the acquired partial video data and extracts a human face portion. The degree of similarity between the feature amount obtained from the extracted face portion image and the feature amount possessed by the person feature data 13a is calculated. The video information analysis unit 22 determines the feature quantity of the most similar person feature data 13a, and acquires the person name corresponding to the feature quantity. As described above, for the analysis of the face image, for example, a method described in JP 2009-157767 A can be employed.

  For example, when the video information analysis unit 22 extracts a large number of face parts from the partial video data, it is not necessary to estimate the person names for all the face parts. This is because in a scene where there are many performers, the user's interest is less likely to be a performer, but the process takes time. For example, when the video information analysis unit 22 extracts a face part, if a predetermined number or more of face parts are detected, it is not necessary to specify a person for all the face parts. For example, a method in which the video information analysis unit 22 omits the process of specifying the person name from the face portion and does not include it in the keyword of the person name can be considered. Further, the video information analysis unit 22 may analyze a face part having a large display area to identify a person, and may omit a process of identifying a person for a face part having a small display area.

When the video information analysis unit 22 specifies a keyword based on the feature amount of the face portion of the person, the video relevance may be calculated for this keyword. Specifically, the video information analysis unit 22 acquires synopsis information of video data to be requested, and extracts proper nouns by performing morphological analysis. The text information analysis unit 21 compares the person name extracted based on the feature amount of the face part with the proper noun extracted from the synopsis information, and specifies the person name included in both. The video information analysis unit 22 sets the video relevance of this keyword high for the person names included in both. Here, when the video relevance is already set by the text information analysis unit 22, the video information analysis unit 22 further updates the video relevance so as to increase the video relevance.

  The fixed keyword input unit 23 registers a predetermined keyword in the keyword data. Here, the keyword to be registered is input in advance by an operator or the like. The fixed keyword input unit 23 associates an arbitrary keyword with an arbitrary video data identifier and an arbitrary playback time regardless of the video data identifier and the playback time included in the request, and outputs them to the keyword update unit 24. You may do it.

  The keyword update unit 24 updates each keyword determined by the text information analysis unit 21, the video information analysis unit 22, and the fixed keyword input unit 23 to the keyword data 31a.

  At this time, the keyword updating unit 24 assigns a new identifier corresponding to the keyword. The keyword update unit 24 associates the identifier corresponding to the numbered keyword, the identifier of the video data related to the request, the reproduction time, and the keyword, and stores them in the keyword data. Here, the keyword update unit 24 may associate the video relevance and the fitness with the keyword and register them in the keyword data 31a. Here, since the fitness is data updated by a user operation, the keyword updating unit 24 may set an initial value as the fitness and register it in the keyword data 31a.

With reference to FIG. 8, the video content analysis processing by the video content analysis unit 10 according to the embodiment of the present invention will be described. In the example illustrated in FIG. 8, a case will be described in which keywords output by the text information analysis unit 21 and the video information analysis unit 22 are registered.

First, in step S <b> 201, the video content analysis unit 10 acquires the identifier and playback time of video data that is a keyword acquisition target from the keyword selection unit 42.

Next, in steps S202 to S204, keywords by the text information analysis unit 21 are extracted. Specifically, in step S202, the text information analysis unit 21 refers to the video metadata 11a corresponding to the identifier of the video data targeted for keyword processing, and refers to the URI from which the caption information is acquired. Furthermore, the text information analysis part 21 acquires the text data of the subtitle information in the reproduction time zone among the subtitle information.

In step S203, the text information analysis unit 21 extracts proper nouns from the text data of the caption information acquired in step S202. In step S204, the text information analysis unit 21 compares the proper noun acquired in step S203 with the proper noun of the synopsis information in the video metadata 11a to determine the video relevance.

Next, in steps S205 to S207, keywords are extracted by the video information analysis unit 22. Specifically, in step S205, the video information analysis unit 22 refers to the video data corresponding to the identifier of the video data targeted for keyword processing, and acquires partial video data in the playback time zone.

Further, in step S206, the video information analysis unit 22 analyzes the partial video data acquired in step S205, extracts the face portion of the person, and compares the feature amount of the person specifying data to estimate the person name. In step S207, the video information analysis unit 22 compares the person name acquired in step S206 with the person name of the synopsis information in the video metadata 11a to determine the video relevance.

As described above, when the keyword for the requested time zone is determined, in step S208, the keyword update unit 24 stores the keyword data 31a in association with the keyword and the video relevance specified in step S202 to step S207. To do. At this time, the keyword updating unit 24 assigns a keyword correspondence identifier for each keyword, generates a record in which the keyword correspondence identifier is associated with the video data identifier, the reproduction time, the keyword, and the video relevance, and the keyword data 31a. Insert into. Here, the video content analysis unit 10 is a keyword output in common by the text information analysis unit 21 and the video information analysis unit 22, and the keyword included in the synopsis information has the highest video relevance. Set the video relevance.

As described above, when the video data identifier and the playback time are input from the user terminal 101, the related information providing apparatus 1 according to the embodiment of the present invention receives the video data and video metadata corresponding to the playback time. Thus, a keyword that is considered to be effective for extracting the relevant information desired by the user is specified. As a result, it is possible to provide information related to a scene in which a user is interested without newly defining an identifier for each frame in broadcast data or video distribution data and setting related information in advance accordingly. it can. Moreover, by acquiring relevant information appropriately from the web, it is possible to dynamically generate relevant information and provide the latest relevant information.

In the embodiment of the present invention, the keyword is acquired by analyzing caption information and video data. In this way, related information is acquired based on the video data and the keyword estimated for each reproduction time, and provided to the user. Here, it is considered that keywords related to related information that many users are interested in are keywords indicating video data and reproduction time. Therefore, based on feedback from users, the degree of matching of keywords related to related information viewed by more users can be increased, so that related information based on appropriate keywords can be provided to users.

(Other embodiments)
As described above, the embodiments of the present invention have been described. However, it should not be understood that the descriptions and drawings constituting a part of this disclosure limit the present invention. From this disclosure, various alternative embodiments, examples, and operational techniques will be apparent to those skilled in the art.

  For example, the related information providing device described in the embodiment of the present invention may be mounted on one device as shown in FIG. 1 or may be mounted on a plurality of devices.

  Needless to say, the present invention includes various embodiments not described herein. Therefore, the technical scope of the present invention is defined only by the invention specifying matters according to the scope of claims reasonable from the above description.

DESCRIPTION OF SYMBOLS 1 Related information provision apparatus 2 Related information provision system 10 Video content analysis part 11 Video metadata storage part 12 Video data storage part 13 Person feature data storage part 21 Text information analysis part 22 Video information analysis part 23 Fixed keyword input part 24 Keyword update Unit 31 keyword data storage unit 32 user data storage unit 41 related information request detection unit 42 keyword selection unit 43 related information acquisition unit 44 related information output unit 45 new related information notification unit 100, 102 video viewing device 101, 103 user terminal 110 communication network

Claims (8)

A related information providing method for providing related information of video data,
A related information request detecting step including a video data identifier and a playback time from the user terminal, and receiving a request for video related information at the playback time;
The related information providing device acquires text data corresponding to an identifier and a playback time of the video data included in the request from the video data, and a proper noun included in the acquired text data is acquired from the playback of the video data. Text information analysis step to extract as a keyword of time,
The related information providing apparatus associates the identifier and playback time of the video data with the keyword extracted in the text information analysis step, and attaches an identifier corresponding to the keyword that identifies this association, to the keyword data. A keyword update step to memorize;
A keyword selection step in which the related information providing device acquires, from the keyword data, an identifier and a keyword corresponding to a keyword corresponding to the identifier and playback time of the video data received in the related information request detection step;
A related information output step in which the related information providing device outputs related information related to the keyword selected in the keyword selecting step to a user terminal;
The related information provision method characterized by comprising. The related information providing device acquires partial video data corresponding to an identifier and playback time of video data included in the request, a feature amount of a portion where a person is imaged in the partial video data, a person name, Video information analysis that compares the feature quantity of the person feature data with the feature quantity of the person image and extracts the person name of the person included in the partial video data as a keyword of the playback time of the video data Further comprising steps,
The keyword update step further includes associating an identifier corresponding to the keyword, an identifier of the video data, a reproduction time of the video data, and the keyword extracted in the video information analysis step, and storing them in the keyword data. The related information providing method according to claim 1, wherein the related information is provided. In the keyword update step, the identifier corresponding to the keyword is further associated with the matching degree of the keyword at the reproduction time of the video data, and stored in the keyword data.
The related information output step includes:
The related information related to the keyword selected in the keyword selection step is output to the user terminal in association with the keyword corresponding identifier.
When the relevant information is selected in the user terminal, the keyword data is updated to increase the matching degree corresponding to the keyword corresponding identifier,
In the keyword data, when a plurality of keywords are associated with the identifier of the video data and the playback time, related information related to a keyword having a high matching degree among the plurality of keywords is preferentially output. The related information providing method according to claim 1, wherein the related information is provided. In the keyword update step, the identifier corresponding to the keyword is associated with video relevance and stored in the keyword data,
In the text information analysis step, when the keywords corresponding to the video data and the playback time are extracted, the video relevance is set so that the video relevance is high when the extracted keywords are included in the synopsis information of the video data. And
In the keyword information, when a plurality of keywords are associated with the identifier of the video data and the playback time in the keyword data, the related information output step is related to the keyword having a high video relevance among the plurality of keywords. The related information providing method according to any one of claims 1 to 3, wherein the information is preferentially output. A related information providing device for providing related information of video data,
A related information request detecting unit that includes an identifier of video data and a playback time from a user terminal, and receives a request for video related information at the playback time;
Text that acquires text data corresponding to an identifier and playback time of the video data included in the request from the video data, and extracts a proper noun included in the acquired text data as a keyword of the playback time of the video data Information analysis department,
A keyword updating unit for associating the identifier and the reproduction time of the video data with the keyword extracted by the text information analysis unit, attaching a keyword correspondence identifier for identifying the correspondence, and storing the keyword in the keyword data;
A keyword selection unit that acquires an identifier and a keyword corresponding to the identifier of the video data and the playback time received by the related information request detection unit from the keyword data;
A related information output unit that outputs related information related to the keyword selected by the keyword selection unit to a user terminal;
A related information providing apparatus comprising: A storage device for storing person feature data in which a person name is associated with a feature amount of an image of the person;
The partial video data corresponding to the identifier and the playback time of the video data included in the request is acquired, and the feature amount of the portion where the person is imaged in the partial video data, and the feature amount stored in the person feature data In comparison, the video information analysis unit further extracts a person name included in the partial video data as a keyword of the playback time of the video data,
The keyword updating unit further associates the identifier corresponding to the keyword, the identifier of the video data, the reproduction time of the video data, and the keyword extracted by the video information analysis unit, and stores them in the keyword data. The related information providing apparatus according to claim 5, wherein The keyword update unit further associates the identifier corresponding to the keyword with the degree of matching of the keyword at the reproduction time of the video data, and stores it in the keyword data.
The related information output unit includes:
The related information related to the keyword selected by the keyword selection unit is output to the user terminal in association with the keyword corresponding identifier.
When the relevant information is selected in the user terminal, the keyword data is updated to increase the matching degree corresponding to the keyword corresponding identifier,
In the keyword data, when a plurality of keywords are associated with the identifier of the video data and the playback time, related information related to a keyword having a high matching degree among the plurality of keywords is preferentially output. 7. The related information providing apparatus according to claim 5 or 6, characterized in that:   The related information provision program for making a computer perform the step of any one of Claim 1 thru | or 4.

Images