JP6688368B1 - Video content structuring device, video content structuring method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video content structuring device, a video content structuring method, and a computer program capable of hierarchically structuring video contents in consideration of contents. A video content structuring apparatus 1 includes a block dividing unit 10 that divides a video content into blocks, a metadata adding unit 20 that adds metadata to each divided block, and a metadata that is added. And a structuring unit 30 for hierarchically structuring the video content. [Selection diagram] Figure 1

Description

  The present invention relates to a video content structuring apparatus, a video content structuring method, and a computer program for structuring video content.

  In recent years, in broadcasting stations and the like, the file-based program production environment has been advanced, and efficient management of video contents has become more and more important. It takes a great deal of work for a video editor to search for a content containing a specific keyword from a huge amount of video content, and to search for a video scene that can be used for program production from the content. Therefore, in order to facilitate the search of the video content, a technique of adding metadata to the video content has been proposed (Non-Patent Document 1). However, since it takes time to manually add metadata, a technique for automatically adding metadata has been studied (Non-Patent Document 2). In addition, in order to improve the efficiency of video editing work, a technique of dividing video content into scenes based on audio / video signals has been proposed (Non-Patent Document 3).

"Trends on support for metadata production", search October 17, 2018, Internet <URL: https://www.nhk.or.jp/strl/publica/rd/rd163/pdf/P04-21.pdf> "About adding metadata to content", search October 17, 2018, Internet <URL: http://www.soumu.go.jp/main_content/000225131.pdf> "ViaPlatz", search October 17, 2018, Internet <URL: http://www.viaplatz.com/spec/>

  However, according to the conventional technique, the scene is divided even if the contents are continuous. For example, even if the telop is the same and the content is continuous, it will be divided when the background image changes. Specifically, there is a case where a telop such as "Osaka" appears in a location program and the background image changes every few seconds. Even if the same person talks continuously, it will be divided if the camera cut changes, such as shooting from the front or shooting from the side.

  Further, according to the conventional technique, since the image data is uniquely divided, the granularity of the scene to be confirmed cannot be changed depending on the video editor. For example, as shown in FIG. 8 (a), the video content is divided into a plurality of blocks B1, B2, B3, B4, B5, ... And the metadata thereof is managed as shown in FIG. 8 (b). I shall. Here, when the “location scene related to food (blocks B1 to B3)” is composed of “scene outside the store (block B1)” and “scene inside the store (block B2 to B3)”, "People who want to work outside scenes and scenes inside the store together (people who want to combine blocks B1-B3 into one scene)", "People who want to check only scenes inside the store (blocks B2-B3 People who want to make one scene together) ”, the granularity of the scenes to check is different. Also, in the case of "Switching corners (block B4 and block B5)", when the studio caster talks "Conclusion of previous corner (block B4)" and "Connect to next corner (block B5)" in succession, "Person who wants to work as a caster scene collectively (person who wants to combine blocks B4-B5 into one scene)", "Person who wants to confirm separately the scene of the previous corner and the scene of the connection to the next corner (People who want to use blocks B4 and B5 as their respective scenes) "have different grain sizes of scenes to be confirmed.

  In view of the above-mentioned conventional technique, the present invention aims to provide a video content structuring apparatus, a video content structuring method, and a computer program capable of hierarchically structuring video content in consideration of the content. And

  In order to achieve the above object, the invention according to a first aspect is a video content structuring apparatus, which includes a block division unit for dividing video content into blocks, and a meta data for giving metadata to each divided block. The gist is to include a data adding unit and a structuring unit that hierarchically structures the video content based on the added metadata.

  The invention according to a second aspect is the invention according to the first aspect, wherein the metadata adding section derives metadata based on at least one of a voice recognition result, a character recognition result, and an image recognition result. , The derived metadata is weighted, and the weighted metadata is integrated for each block.

  The invention according to a third aspect is characterized in that, in the invention according to the second aspect, the metadata adding unit increases the weight of the keyword derived from both the voice recognition result and the character recognition result.

  The invention according to a fourth aspect is the invention according to the second aspect, wherein the metadata adding unit increases the weight of a keyword and an object that has been appearing for a long time, or has a large number of appearances. The main point is to increase the weight for keywords and objects.

  A fifth aspect of the invention is based on the invention of any one of the second to fourth aspects, and the gist is that the structuring unit reduces the weight of the word of the representative vector.

  According to a sixth aspect of the invention, in the invention according to any one of the second to fourth aspects, the structuring unit increases the weight for the object and decreases the weight for the keyword as the hierarchy becomes deeper. Is the gist.

  An invention according to a seventh aspect is a video content structuring method, comprising: a block dividing step in which a computer divides video content into blocks; and a metadata adding step of adding metadata to each of the divided blocks. , A structuring step of hierarchically structuring the video content based on the added metadata.

  The gist of the invention according to an eighth aspect is a computer program for causing a computer to function as the video content structuring device according to any one of the first to sixth aspects.

  According to the present invention, it is possible to provide a video content structuring apparatus, a video content structuring method, and a computer program capable of hierarchically structuring video content in consideration of contents.

It is a block diagram of the video content structuring device in the embodiment of the present invention. It is a block diagram of the metadata provision part in embodiment of this invention. It is a flow chart which shows operation of a keyword derivation part in an embodiment of the present invention. It is a flow chart which shows operation of an object derivation part in an embodiment of the present invention. It is a flow chart which shows operation of a structuring part in an embodiment of the present invention. 6 is a graph showing the relationship between the weight update function and the number of layers in the embodiment of the present invention. It is a figure which shows an example of the structuring result by the structuring part in embodiment of this invention. It is a figure for explaining an example of granularity of a scene.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description of the drawings, the same or similar reference numerals are given to the same or similar parts.

<< Overall structure >>
FIG. 1 is a configuration diagram of a video content structuring device 1 according to an embodiment of the present invention. The video content structuring device 1 is a computer for structuring video contents, and functionally includes a block dividing unit 10, a metadata adding unit 20, and a structuring unit 30.

  The block division unit 10 divides the video content into blocks. A conventional technique can be used as a technique for finely dividing the video content. For example, viaPlatz or open source can be used. The video content is input to the block division unit 10, and the block division result is output from the block division unit 10.

  The metadata adding section 20 adds metadata (keyword + object) to each block divided by the block dividing section 10. The metadata is information that describes the video content. Some metadata exists in the form of being embedded in the video content, but here, it is assumed that the metadata is managed separately from the video content. The block division result is input to the metadata addition unit 20, and the metadata for each block is output from the metadata addition unit 20.

  The structuring unit 30 hierarchically structures the video content based on the metadata (keyword + object) provided by the metadata providing unit 20. The structuring unit 30 receives the metadata and the number of layers for each block, and the structuring unit 30 outputs the structuring result of the video. The number of layers is the number to be hierarchized and is designated by a user such as a video editor.

  As described above, according to the video content structuring apparatus 1 in the embodiment of the present invention, it is possible to extract the metadata for each block, and thus the metadata (contents) for each block is taken into consideration to hierarchically video. It is possible to structure the content. As a result, it is possible to integrate sections in which the contents are continuous, and it is also possible for the video editor to change the granularity of the scene to be confirmed.

<< Metadata adder >>
FIG. 2 is a configuration diagram of the metadata adding unit 20. As shown in this figure, the metadata providing unit 20 includes various recognition units 21 and a metadata integration unit 22.

  The various recognition unit 21 is a functional unit that performs various recognition processes based on the block division result, and includes a voice recognition unit 21A, a character recognition unit 21B, and an image recognition unit 21C. The voice recognition unit 21A recognizes the voice included in the block division result. The character recognition unit 21B recognizes a character included in the block division result. The image recognition unit 21C recognizes an image included in the block division result. It is conceivable to use an external API such as NTT, Google, Azure, and Watson for the various recognition units 21.

  The metadata integration unit 22 derives metadata for each block based on the various recognition results by the various recognition units 21, weights the derived metadata, and integrates the weighted metadata for each block. And a keyword derivation unit 22A and an object derivation unit 22B. The keyword derivation unit 22A weights each keyword based on the voice recognition result (keyword) by the voice recognition unit 21A and the character recognition result (keyword) by the character recognition unit 21B. The object derivation unit 22B weights each object based on the image recognition result (object) of the image recognition unit 21C.

≪Keyword derivation part≫
FIG. 3 is a flowchart showing the operation of the keyword derivation unit 22A. Hereinafter, the function of the keyword derivation unit 22A will be described together with its operation with reference to FIG.

  First, the keyword derivation unit 22A derives a keyword based on the voice recognition result by the voice recognition unit 21A, and also derives a keyword based on the character recognition result by the character recognition unit 21B (steps S1 and S2). For example, it is possible to derive a keyword using an external API such as the NTT corevo keyword extraction API or the yahoo key phrase extraction API.

  Next, the keyword derivation unit 22A derives a weight (a_key) for each keyword derived in the keyword derivation steps S1 and S2, and weights each keyword (step S3). At this time, it is conceivable to increase the weight of the keywords derived in both the keyword derivation steps S1 and S2. Further, by integrating the results of the keyword derivation steps S1 and S2, the weight of the keyword having the longest appearance time is set to 1 and the weight of the keyword having the shortest appearance time is set to 0.1 based on the time of appearance for each keyword. , 0.1 to 1 can be standardized. Similarly, the results of the keyword derivation steps S1 and S2 are integrated, and for each keyword, based on the number of appearances, the weight of the keyword having the highest appearance frequency is 1, and the weight of the keyword having the lowest appearance frequency is 0.1. It is conceivable to standardize between 0.1 and 1.

  Finally, the keyword derivation unit 22A outputs each keyword weighted in the keyword weighting step S3 (step S4).

≪Object derivation part≫
FIG. 4 is a flowchart showing the operation of the object derivation unit 22B. Hereinafter, the function of the object derivation unit 22B will be described together with its operation with reference to FIG.

  First, the object derivation unit 22B derives an object based on the image recognition result by the image recognition unit 21C (step S11).

  Next, the object derivation unit 22B derives the weight (a_obj) for each object derived in the object derivation step S11, and weights each object (step S12). At this time, the weight of the object with the longest appearance time is set to 1 and the weight of the object with the shortest appearance time is set to 0.1 based on the time of appearance for each object, and normalized between 0.1 and 1. Can be considered. Similarly, for each object, based on the number of appearances, standardize between 0.1 and 1 with the weight of the object with the most appearances being 1 and the weight of the object with the least appearances being 0.1. Can be considered.

  Finally, the object derivation unit 22B outputs each object weighted in the object weighting step S12 (step S13).

<< Structured Department >>
FIG. 5 is a flowchart showing the operation of the structuring unit 30. Hereinafter, the function of the structured unit 30 will be described together with its operation with reference to FIG.

  First, when the number R of layers is input, the structuring unit 30 sets 1 to rank (steps S21 → S22). The number of layers R is designated by the user. rank is a variable indicating the number of layers.

  Next, the structuring unit 30 performs clustering based on the metadata (keyword + object) from the metadata adding unit 20 (step S23). Clustering is a technique or method for collecting similar items from a large amount of data and automatically classifying them.

  The clustering step S23 includes a metadata vectorization step for each block and a block clustering step. In the metadata vectorization step for each block, for each block, a keyword (key) and a weight (a_key) for each keyword, an object (obj) and a weight (a_obj) for each object are input, and a vectorization tool such as word2vec is input. A semantic vector (S (b)) for each block is derived by using (b is a block number). In the block clustering step, the semantic vector (S (b)) for each block is input, and clustering is performed using a clustering tool such as the k-means method.

  Next, the structuring unit 30 derives the representative metadata (step S24). In this representative metadata derivation step S24, the average value S (b, c) of the “semantic vector for each block (S (b))” of the block group forming each cluster is derived (c is the cluster number), and Let it be the representative vector of the block. Also, the representative vector S (b, c) of each block is converted into a word (W) using a vectorization tool such as word2vec.

  Next, the structuring unit 30 updates the weights so that the structuring is performed by dividing the hierarchy (step S25). In the weight updating step S25, the following processing can be considered for each cluster.

  First, it is possible to reduce the weight (a_W) of the representative vector with respect to the word W. For example, a_W = 0 is set in order to remove the influence of the metadata already extracted as the representative vector.

  Further, since the object is useful for finely dividing, it is possible to increase the value of the weight (a_obj) for the object and decrease the value of the weight (a_key) for the keyword as the number of layers increases. For example, update as follows.

a_obj (rank + 1) = α × a_obj (rank)
a_key (rank + 1) = (2-α) × a_key (rank)
α = β × exp (rank + γ)
As shown in FIG. 6, the relationship between the weight update function α and the rank rank is formulated such that the value of the weight (a_obj) for an object increases as the rank increases (the hierarchy becomes deeper). Here, the formula is formulated by an exponential function, but other formulas are also conceivable.

  Next, the structuring unit 30 adds 1 to the value of rank and repeats the same processing until the value of rank reaches the number R of layers (steps S26 → S27 → S23 → ...). When the value of rank reaches the number of layers R, the word (W) is output together with the structuring result (steps S27 → S28).

<< Example of structured result >>
FIG. 7 is a diagram showing an example of the structuring result by the structuring unit 30. Here, a UI image displayed to the user is illustrated. For example, when the number of layers is designated by the user, the representative metadata of each section in the layers 1, 2, and 3 is displayed.

  Specifically, it is assumed that a location program is composed of “scenes talking about Kyoto”, “scenes talking about Osaka”, and “scenes talking about Kobe”. Also, the "scene of talking in Kyoto" is assumed to be composed of "scene of talking in the studio", "scene of the temple", and "scene of the teahouse". In such a case, “Kyoto” may be displayed as the representative metadata of the section M11 in the layer 1. In addition, “studio”, “temple”, and “teahouse” may be displayed as representative metadata of the sections M21, M22, and M23 in the layer 2. Furthermore, "scenes inside the temple", "scenes outside the temple", etc. may be displayed as representative metadata of the sections M31, M32, M33, M34, M35 in the layer 3.

  As described above, according to the video content structuring apparatus 1 in the embodiment of the present invention, by structuring based on the division of the conventional technique, the portions that are finely divided in the conventional technique are integrated, It is possible to display as layers 1, 2, and 3. Since the number of layers can be specified by the user, the granularity of the scene to be confirmed can be changed by the video editor.

≪Modification≫
In the above embodiment, the number of layers R is specified by the user, but the number of layers R is not necessarily specified. For example, when the hierarchy can be structured up to 10 levels, it is possible to uniformly output all of the hierarchy 1 to hierarchy 10 as the structured result.

≪Other application examples≫
In the above embodiment, the description is made assuming that the video editor searches for the video scene, but the general user searches for only the scene in which the entertainer of his or her preference appears, so that the general user can use it. Can also be considered. For example, when a favorite entertainer appears in a singing program, in the conventional technology, metadata is randomly added to a place where the corresponding entertainer appears such as an opening, a singing place, or a closing, but the present invention is used. By structuring, you can make it easy to find only the part where the entertainer is singing.

≪Summary≫
As described above, the video content structuring apparatus 1 according to the embodiment of the present invention includes the block dividing unit 10 that divides the video content into blocks, and the metadata adding unit 20 that adds metadata to each divided block. And a structuring unit 30 that hierarchically structures the video content based on the added metadata. As a result, since the metadata for each block can be extracted, it becomes possible to hierarchically structure the video content in consideration of the metadata (contents) for each block.

  Further, the metadata adding section 20 derives metadata based on at least one of the voice recognition result, the character recognition result, and the image recognition result, weights the derived metadata, and weights the metadata. The metadata may be integrated for each block. As a result, since the features included in the video and audio can be captured, it is possible to extract the representative features of each block.

  Further, the metadata adding section 20 may increase the weight of the keyword derived from both the voice recognition result and the character recognition result. Thus, it can be considered that the keywords derived from both the audio and the video (telop) strongly represent the characteristics of the block.

  In addition, the metadata adding unit 20 may increase the weight for a keyword and an object that appear for a long time, or may increase the weight for a keyword and an object that appear a large number of times. As a result, it can be considered that the longer the appearance time of a keyword or object, the stronger the characteristic of the block is represented. Further, it can be considered that the keyword and the object that appear more frequently indicate the feature of the block more strongly.

  Further, the structuring unit 30 may reduce the weight of the word of the representative vector. This makes it possible to avoid the problem that a vector that has already become a representative vector is derived thereafter.

  The structuring unit 30 may increase the weight for the object and decrease the weight for the keyword as the hierarchy becomes deeper. As a result, the deeper the hierarchy, the easier it is for the object to be derived as representative metadata rather than the keyword.

  Note that the present invention can be realized not only as the video content structuring apparatus 1, but also as a video content structuring method that uses the characteristic functional units of the video content structuring apparatus 1 as steps, and It can also be realized as a computer program for causing a computer to function as the structured device 1. It goes without saying that such a computer program can be distributed via a recording medium such as a CD-ROM or a transmission medium such as the Internet.

<< Other Embodiments >>
While embodiments of the present invention have been described above, the discussion and drawings which form a part of the disclosure are to be understood as illustrative and not limiting. From this disclosure, various alternative embodiments, examples and operational techniques will be apparent to those skilled in the art. That is, the embodiments of the present invention include various embodiments not described here.

DESCRIPTION OF SYMBOLS 1 video content structuring device 10 block dividing unit 20 metadata assigning unit 21 various recognizing units 21A voice recognizing unit 21B character recognizing unit 21C image recognizing unit 22 metadata integrating unit 22A keyword deriving unit 22B object deriving unit 30 structuring unit

Claims (7)

A block division unit that divides the video content into blocks,
A metadata adding unit that adds metadata to each divided block,
And a structuring unit for hierarchically structuring the video content based on the added metadata ,
The metadata assigning unit derives metadata based on at least one of a voice recognition result, a character recognition result, and an image recognition result, weights the derived metadata, and weights the metadata. A video content structuring device characterized by integrating data for each block . The video content structuring apparatus according to claim 1 , wherein the metadata adding unit increases the weight of the keyword derived from both the voice recognition result and the character recognition result. The meta data providing section, the time have appeared to increase the weight longer keywords and objects, or, according to claim 1, characterized in that to increase the weight as keywords and objects more times that have emerged Video content structuring device. The video content structuring device according to claim 1 , wherein the structuring unit reduces a weight of a word of the representative vector. 4. The video content structuring apparatus according to claim 1 , wherein the structuring unit increases the weight for the object and decreases the weight for the keyword as the hierarchy becomes deeper. Computer
A block dividing step of dividing the video content into blocks,
A metadata adding step of adding metadata to each of the divided blocks,
And a structuring step for structuring the video content hierarchically based on the given metadata ,
The metadata applying step derives metadata based on at least one of a voice recognition result, a character recognition result, and an image recognition result, weights the derived metadata, and weights the weighted meta data. A video content structuring method characterized by integrating data for each block . A computer program for causing a computer to function as the video content structuring device according to any one of claims 1 to 5 .

Images