JP2009103945A - Video content processing device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately and efficiently classify video contents. <P>SOLUTION: A feature amount extracting section 130 in a server 110 extracts a feature amount of a speech data included in an uploaded video content. A classification processing section 140 provides service for classifying the uploaded video content and storing it in a content storage section 170, and search service of the stored video content. A classification processing section 140 utilizes the feature amount extracted by the feature amount extracting section 130, when the video content is stored and searched, as an index for determining a classification of the video content. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a video content processing technique, and more specifically to a technique for classifying video content.

  With the spread of the Internet and the speeding up of communication networks, even individual users can easily upload video content to a server on the Internet. Therefore, a large amount of video content is accumulated on the Internet, and these video content are actively used such as being viewed by a user.

  Normally, the user performs a search operation in order to reach the target video content. A search engine is required to be able to select target video content accurately and without omission according to a user's search operation.

  Conventionally, a keyword technique is used as a search technique. This method is originally for text content, and in order to apply it to the search of video content, for example, text information indicating the type of the video content is written in the thumbnail of the video content. Is compared with the text information written in the thumbnail, and the video content having the text information matching the keyword is extracted as the target video content.

In recent years, a technique has been developed that makes it possible to search for a type of image indicated by a keyword by improving the image analysis technique. According to this method, for example, for a keyword “flower”, an image determined to be a flower by the result of image analysis is obtained as a search result.
JP 2002-91481 A JP 2001-215882 A Special table 2006-501502 gazette JP 2005-250472 A

  There are two methods for writing text information indicating the type of the video content in the thumbnail of the video content. One is a method in which the administrator views the uploaded video content, determines the type of the video content, and writes the text information to the thumbnail according to the determination result. This method requires manual work by an administrator, and is unrealistic in the current situation where a large amount of video content is uploaded.

  The other is that the type of video content is specified in advance, the user who uploads the video content specifies the type, and the text information corresponding to the type specified by the user is written and stored in the thumbnail of the video content. It is a technique. This method requires designation by the user and is not efficient. In addition, since the user's designation is not always correct, there is a problem that inaccurate search results or search omissions are likely to occur during a later search.

  Further, since the image included in the video content is a normal moving image, it is difficult to specify the type of the video content by image analysis.

  One aspect of the present invention is a video content processing apparatus. This apparatus includes a feature amount acquisition unit that acquires a feature amount of audio data included in video content, and a classification processing unit that uses the feature amount acquired by the feature amount acquisition unit as an index for determining the type of video content. Prepare.

  It is also effective as an aspect of the present invention to express the above aspect by replacing a method, a system, or a program that causes a computer to operate as the above apparatus.

  According to the technology of the present invention, video content can be classified accurately and efficiently.

Before describing the embodiment of the present invention, first, the principle of the present invention will be described.
Video content is usually composed of moving image data and audio data. As a result of research exploration, the inventor of the present application has found that depending on the type of video content, the audio indicated by the audio data included therein has different characteristics. A description will be given with reference to FIGS.

  FIG. 1 shows an example of audio indicated by audio data included in video content. For such audio, the main frequency band, the ratio of the power spectrum of the main frequency band to the sum of the power spectrum of all frequency bands, the maximum volume, the average volume, the ratio of the average volume to the maximum volume, the low volume interval density Can be sought.

  The maximum volume is the maximum value among the level values of all samples of the audio data included in the video content.

  The low volume section is the total length of the section where the volume is low, for example, the section where the absolute value of the level is equal to or less than a predetermined threshold, and the low volume section density is the ratio of the low volume section to the entire section. It is.

  The average volume is an average value of the volume in the section excluding the low volume section over the entire section of the video content.

  In the following description, the main frequency band, the ratio of the power spectrum of the main frequency band to the sum of the power spectrum of all frequency bands, the maximum volume, the average volume, the ratio of the average volume to the maximum volume, the low volume interval density are characterized. It is called quantity.

  FIG. 2 shows the results of obtaining the six types of feature amounts for a plurality of types of video content. As types of video content, “News”, “Vocal”, “Piano”, “Classic”, and “CM” are “news broadcast on TV”, “vocal”, “solo performance of piano”, “ “Classical music performance by orchestra” and “Commercial broadcast on TV” are shown. Of “Vocal”, “F-Vocal” and “M-Vocal” are images of “female vocal” and “male vocal”. Each is shown. In addition, “Tone” and “Noise” for comparison with the above-mentioned types of video content indicate monotone sine waves and noise, respectively.

  From the results shown in FIG. 2, it is possible to find the sound tendency of each type of video content. As shown in FIG. 3, each type of video content has a tendency to take a value within a predetermined range for one or more of the above-described feature amounts.

  The inventor of the present application has found that the feature amount of the audio data of the video content is related to the type of the video content, and has come up with a technique of using the feature amount of the audio data of the video content as an index for determining the type of the video content. did. As the feature amount, the above-described main frequency band, the frequency feature amount represented by “power spectrum of main frequency band / power spectrum of all frequency bands”, the maximum volume, and the volume represented by “average volume / maximum volume”. Two or more of the three types of feature amounts, that is, the feature amount and the low volume interval density are used.

This technology can be applied in various forms.
For example, when recording or storing video content, the above-described feature amount of the audio data of the video content is extracted, and the type of the video content is determined based on the extracted feature amount. Then, text information indicating the determined type is stored in attached information such as a thumbnail of the video content. In this way, the type information of the video content is attached to the video content without any work such as specifying the type of the video content or inputting text information indicating the type of the video content. It can be used for searching by keywords.

  Further, the technique of the present invention can be applied not only to search by keyword but also to search for predetermined video content, for example, video content similar to content specified by the user. In this case, for example, when the video content is recorded or stored, the above-described feature amount of the audio data is extracted, and the extracted feature amount itself is stored in the attached information. When searching for video content similar to the predetermined video content, the target video content can be selected by extracting the feature amount of the predetermined video content and comparing it with the feature amount of the stored video content. Good.

  Of course, the technology according to the present invention is not limited to the two examples described above, and can be applied to any case where classification of video content is required. It is possible to achieve accurate and efficient content classification.

  Techniques for extracting and using feature values of audio data have been proposed from various viewpoints (Patent Documents 1 to 4). These techniques include the audio data including the audio data. There is no idea used as an index for determining the type of video content, and the characteristics of the extracted feature quantities are also different. For example, the technique disclosed in Patent Document 1 is a technique for extracting a feature amount from standard speech and input speech, calculating a similarity between the input speech and standard speech based on the extracted feature amount, and performing speech recognition. Disclosure. In this technique, for example, in order to detect a human utterance “ga” from input speech, a feature amount (reference feature amount) is previously extracted from the standard speech “ga”. Then, the feature amount of each local part is extracted from the input speech, and the presence or absence of a local part whose feature amount is similar to the reference feature amount is searched. That is, this technique specifies the specific contents of the local part of the voice, and the feature amount also represents the characteristic of the local part of the voice.

  On the other hand, the technology according to the present invention captures the characteristics of the entire audio data by extracting as features the features that can represent the characteristics of the entire audio data, regardless of the specific content of the local area. . This will be described with reference to FIG. In the figure, the reference sound is the sound of news broadcast on television, the comparison target sound 1 is the sound of another news broadcast on television, and the comparison target sound 2 is the sound of a music meeting. According to the technique of the present invention, the comparison target voice 1 and the reference voice are determined to have similar characteristics to the reference voice regardless of whether or not they are similar when viewed locally. It is determined that the video content including each of them is of a similar type. Further, regardless of whether the comparison target sound 2 and the reference sound are similar when viewed locally, according to the technique of the present invention, the comparison target sound 1 has characteristics that are different from the reference sound. Then, it is determined, and the video contents including them are also determined to be of different types.

Based on the above description, embodiments of the present invention will be described.
FIG. 5 shows a network system 100 according to the embodiment of the present invention. The network system 100 includes a server 110 and a plurality of terminals 190 that can be connected to the server 110 via a network such as the Internet 180.

  FIG. 6 shows the configuration of the server 110. In the present embodiment, the server 110 provides a storage service for video content uploaded from an arbitrary terminal 190 and a search service for stored video content. As illustrated in FIG. 6, the server 110 includes a user interface 120, a feature amount extraction unit 130, a classification processing unit 140, a reproduction unit 150, a reference data storage unit 160, and a content storage unit 170.

  The user interface 120 is an interface between the terminal 190 and the server 110 and mediates transmission / reception between the terminal 190 and the server 110. The user interface 120 also displays a service menu that can be used by the terminal 190. In the present embodiment, the user interface 120 displays the following three as the main menu.

1. Upload video content 2. Search for video content. Thumbnail display

  When “Upload video content” in the main menu is selected by the terminal 190, the user interface 120 receives the video content uploaded from the terminal 190 and outputs it to the feature amount extraction unit 130.

  In addition, when “search for video content” in the main menu is selected by the terminal 190, the user interface 120 displays a list of video content types as a submenu and allows the terminal 190 to select a type. Instruct. Then, the user interface 120 outputs information indicating the type selected by the user via the terminal 190 to the classification processing unit 140 to perform a search based on the type selected by the user.

  When “Thumbnail Display” in the main menu is selected by the terminal 190, the user interface 120 instructs the playback unit 150 to display thumbnails of all video content stored in the content storage unit 170.

  Further, when any of the thumbnails displayed by the playback unit 150 is selected in response to this instruction, the user interface 120 instructs the playback unit 150 to play the video content corresponding to the selected thumbnail.

  Further, when the playback unit 150 plays back the video content in response to this instruction, the user interface 120 displays “search for related content” as a submenu and when the submenu is selected by the terminal 190. In addition, the classification processing unit 140 is instructed to search related content of the reproduced video content.

  That is, in the present embodiment, the server 110 provides two types of search services: search for video content of the type pointed out by the user and search for related content of the video content specified by the user.

  When the video content is uploaded from the terminal 190 via the user interface 120, the feature amount extraction unit 130 extracts the feature amount from the audio data of the video content and outputs the feature amount to the classification processing unit 140. Here, the feature quantity extraction unit 130 extracts the main frequency band and “the sum of the power spectrum of the main frequency band / the power spectrum of all frequency bands” as the frequency feature quantity, and the maximum volume and the average volume as the volume feature quantity. And “average volume / maximum volume” are extracted, and the low volume interval density is also extracted. That is, in this embodiment, six types of feature quantities are used.

  Here, an example of a technique for extracting the main frequency band will be described with reference to the flowchart of FIG. The main frequency band can be a frequency band having the strongest power spectrum among the frequency bands of audio included in the video content. In order to acquire this frequency band, the feature amount extraction unit 130 first divides the audio data into a plurality of (divNum) frames. One frame is a discrete Fourier transform processing unit, and the frame length is the number of samples in one frame.

  The feature amount extraction unit 130 performs discrete Fourier transform on the first frame (S12, S14: No), and obtains a power spectrum result in each frequency band for the frame (S16). Thereafter, the feature quantity extraction unit 130 sequentially performs the Fourier transform in step S16 on each frame and accumulates the power spectrum for each frequency band (S14: No, S16, S18, S20). The feature amount extraction unit 130 performs the above-described processing until Fourier transform is performed on all frames to obtain a power spectrum cumulative value accm (divNum) for each frequency band (S14: Yes).

  Note that the discrete Fourier transform in step S16 is a fast Fourier transform FFT, and an example thereof can be represented by the following equations (1) to (3).

  The feature amount extraction unit 130 acquires the frequency band corresponding to the maximum value among the cumulative values accm (divNum) of the power spectrum for each frequency band obtained in this way as the main frequency band.

  As for “the power spectrum of the main frequency band / the sum of the power spectrum of all frequency bands”, the maximum value among the accumulated values accm (divNum) of the power spectrum and the accumulated value accm (divNum of the power spectrum of each frequency band). ) And the sum of the values.

  Next, an example of a low volume density extraction method will be described. As shown in the flowchart of FIG. 8, the feature amount extraction unit 130 sets a section in which samples having a volume, that is, a level value equal to or lower than the low volume determination threshold th, continue for the number of samples indicated by the low volume section detection threshold minTh. Then, the ratio of the total number of samples in the low volume section in the audio data and the total number of samples in the audio data is obtained as the low volume density.

  In this way, the feature amount extraction unit 130 extracts various feature amounts from the audio data of the video content and outputs them to the classification processing unit 140.

  When the video content is uploaded from the terminal 190, the classification processing unit 140 generates the auxiliary information of the video content and stores it in the content storage unit 170 together with the video content, and a search instruction is given from the user interface 120. At this time, based on the search instruction, the target content is searched from the content storage unit 170 and the playback unit 150 displays these thumbnails.

  First, the processing of the classification processing unit 140 when video content is uploaded will be described. At this time, the classification processing unit 140 uses the feature amount from the feature amount extraction unit 130 and the reference data stored in the reference data storage unit 160 to calculate the similarity as the attached information.

  The reference data storage unit 160 stores feature amounts of video content serving as a reference for a plurality of types of video content. For example, the feature values of “news” shown in FIG. 2 are stored for news, and the feature values of “F-vocals B1” and “F-vocals B2” shown in FIG. I remember it. As an example, the content shown in FIG. 2 is stored in the reference data storage unit 160 as reference data. In the following description, the feature amount of each type of video content stored in the reference data storage unit 160 is referred to as “reference feature amount” of that type of video content.

  For each type of video content, the classification processing unit 140 is characterized by the similarity between the feature amount from the feature amount extraction unit 130 and the reference feature amount stored in the reference data storage unit 160 according to the following equation (4). Calculate for each type of quantity.

  As a result, for each type of video content, the main frequency band, the ratio of the power spectrum of the main frequency band to the sum of the power spectrum of all frequency bands, the maximum volume, the average volume, the ratio of the average volume to the maximum volume, the main frequency band Six similarities for the low volume interval density are obtained.

  And the classification | category process part 140 calculates | requires a similarity score according to following formula (5) for every kind of video content.

Similarity score = k1 x main frequency band similarity
+ K2 x "Power spectrum of main frequency band /
"Sum of power spectra in all frequency bands" similarity
+ K3 x maximum volume similarity (5)
+ K4 x average volume similarity
+ K5 x “average volume / maximum volume” similarity
+ K6 × Low Volume Section Density k in the equation is a weighting coefficient, and by adjusting them, the degree of similarity indicated by the obtained similarity score can be expressed more accurately.

  FIG. 9 shows an example of the similarity score obtained by the classification processing unit 140 when the uploaded video content is vocal. As shown in the figure, the similarity score of the uploaded video content becomes lower as it is similar to the video content to be compared.

  FIG. 10 shows an example of a similarity score when the uploaded video content is a classical music concert. As can be seen from FIG. 10, the similarity score of the uploaded video content becomes lower as it is similar to the video content to be compared.

  That is, the lower the type of similarity score obtained by the classification processing unit 140, the more similar the uploaded video content to the type of video content.

  The classification processing unit 140 stores each feature quantity and similarity score of the uploaded video content as attached information in the content storage unit 170 together with the video content.

  FIG. 11 shows how video content is stored in the content storage unit 170. As shown in FIG. 11, the content storage unit 170 stores the video content main body and the attached information in association with each other, and the attached information includes the main frequency band, “power spectrum of the main frequency band / all frequency bands. ”Sum of power spectrum”, maximum volume, average volume, “average volume / maximum volume”, low volume interval density, 6 feature quantities, “News”, “F-vocal A”, “F-vocal B1”, etc. It is a similarity score for each type of video content.

  Next, processing performed by the classification processing unit 140 when a search instruction is issued from the user interface 120 will be described. As described above, in the present embodiment, the server 110 provides two types of search services: a search for video content of a type specified by the user and a search for related content of the video content specified by the user.

  The classification processing unit 140 sorts the similarity score for the type specified by the user for each video content stored in the content storage unit 170 when an instruction to search for the video content of the type specified by the user is given. Video content whose similarity score is below a predetermined threshold is selected. The video content selected in this way has a similarity score that is similar to the threshold value or less, that is, the user indicates the specified type of video content or more than the threshold value indicates.

  The classification processing unit 140 causes the playback unit 150 to play back the thumbnails of the selected video content in descending order of similarity score.

  On the other hand, when the search for the related content of the video content designated by the user is instructed, the classification processing unit 140 uses the feature amount of the instructed video content as a reference feature and stores the other content stored in the content storage unit 170. The similarity score between the video content and the video content is obtained. The method for obtaining the similarity score is the same as the method for obtaining the similarity score for the uploaded video content except that the reference feature amount is different.

  FIG. 12 shows an example of the similarity score obtained by the classification processing unit 140 when a search for related content of “content 2” among the video contents stored in the content storage unit 170 is instructed. As shown in the figure, a similarity score with content 2 is obtained for video content other than content 2, such as content 1, content 3, and content 4.

  The classification processing unit 140 sorts the similarity scores obtained in this way, and selects video contents whose similarity scores are equal to or less than a predetermined threshold. The video content selected in this way is similar in that the similarity score is equal to or less than the threshold, that is, the user indicates the designated video content and the threshold.

  The classification processing unit 140 causes the playback unit 150 to play back the thumbnails of the selected video content in descending order of similarity score.

  As described above, the server 110 according to the present embodiment can automatically classify video content and can search for content of a type similar to the type of video content arbitrarily specified.

In the above, the technique by this invention was demonstrated based on embodiment. The embodiment is an exemplification, and various changes and increases / decreases may be added without departing from the gist of the present invention. It will be understood by those skilled in the art that modifications to which these changes and increases / decreases are also within the scope of the present invention.
For example, in the above-described embodiment, five types of feature amounts are used. However, the types of feature amounts may be increased or decreased according to the types of video content to be handled.

  In the above-described embodiment, the technique according to the present invention is applied for searching. However, the technique of the present invention determines the type of video content or the type of video content. The present invention can also be applied when providing information for doing so. For example, when recording video content, the recording device may be configured to extract the above-described feature amount and record it along with the video content.

It is a figure which shows the example of the audio data contained in a video content. It is a figure which shows the example of the feature-value of various video content. It is a figure for demonstrating the relationship between the kind of video content, and its feature-value. It is a figure for demonstrating how to catch the characteristic of the audio | speech data by the technique concerning this invention. It is a figure which shows the network system concerning embodiment of this invention. It is a figure which shows the server in the network system shown in FIG. It is a flowchart which shows the example of the method in which the feature-value extraction part in the server shown in FIG. 6 calculates | requires a main frequency band. It is a flowchart which shows the example of the method in which the feature-value extraction part in the server shown in FIG. 6 calculates | requires a low sound volume area density. It is a figure which shows the example of the similarity score calculated | required for every kind of video content with respect to the uploaded video content by the classification | category process part in the server shown in FIG. It is a figure which shows the example of the similarity score calculated | required for every kind of video content with respect to the uploaded video content by the classification | category process part in the server shown in FIG. It is a figure which shows the storage form of the video content in the content memory | storage part in the server shown in FIG. It is a figure which shows the example of the similarity score of the video content designated by the user calculated | required by the classification | category process part in the server shown in FIG. 6, and the other video content stored in the content memory | storage part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Network system 110 Server 120 User interface 130 Feature-value extraction part 140 Classification processing part 150 Playback part 160 Reference data storage part 170 Content storage part 180 Internet 190 Terminal

Claims (8)

A feature amount acquisition unit for acquiring feature amounts of audio data included in the video content;
A video content processing apparatus comprising: a classification processing unit that uses the feature amount acquired by the feature amount acquisition unit as an index for determining a type of the video content.   The feature amount includes two or more of three types of features, that is, a frequency feature amount, a volume feature amount, and a proportion of a volume equal to or less than a predetermined volume of the sound represented by the sound data. The video content processing apparatus according to claim 1, wherein the video content processing apparatus is a video content processing apparatus.   The video content processing apparatus according to claim 2, wherein the frequency feature amount includes a main frequency band and a ratio of a power spectrum of the frequency band to a sum of power spectra of all frequency bands.   The video content processing apparatus according to claim 2, wherein the volume feature amount includes a maximum volume, and a ratio between an average volume and the maximum volume.   The classification processing unit uses the feature amounts acquired by the feature amount acquisition unit with respect to the audio data included in the first video content and the audio data included in the second video content. 5. The video content processing apparatus according to claim 1, wherein a degree of similarity between the type of one video content and the type of the second video content is obtained. 6.   The classification processing unit uses a plurality of types of feature amounts of the audio data included in the first video content and a plurality of types of feature amounts of the audio data included in the second video content for each type. And calculating the similarity between the type of the first video content and the type of the second video content by integrating the similarities of the various types of feature amounts. The video content processing apparatus according to claim 5, wherein the video content processing apparatus is a video content processing apparatus.   The video content processing apparatus according to claim 6, wherein the classification processing unit integrates the degree of similarity of the types of feature amounts by weighted addition. Acquire feature values of audio data included in video content,
A program for causing a computer to execute a process of using the acquired feature amount as an index for determining the type of the video content.

Images