JP2010198111A - Metadata extraction server, metadata extraction method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically generate and update a proper recognizer from information on a Web. <P>SOLUTION: A teacher information collection part 234 collects content from a Web server 240. A characteristic amount calculation part 204 calculates a characteristic amount matrix representing the content of the collected content. A characteristic amount DB 201 clusters the characteristic amount matrixes by group when storing them at predetermined time intervals, and determines a subspace for each group of the characteristic amount matrixes. A metadata arrangement part 203 morphologically analyzes the collected content, and generates a metadata matrix representing occurrence frequency of a morphologically-analyzed word. A correlation coefficient calculation part 214 calculates a correlation coefficient between the group of the characteristic amount matrixes and a group of the metadata matrixes in each subspace. A correlation coefficient DB 205 stores the correlation coefficient calculated for each subspace. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention mainly recognizes the contents of media content composed of video, music, etc., and extracts metadata for associating media data with each other from media data including various information, and metadata extraction It relates to a method and a program.

  As a technology for adding information to media data on the Web, it is possible to statistically collect pass / fail judgments from various users for various annotations given to videos from a plurality of users via a video browsing server on the Web. A technique for selecting and displaying an appropriate annotation by adding syntactic consistency and video feature amount to this is known (for example, see Non-Patent Document 1).

  In addition, as a technique for extracting information from media data on the network, a special bridge on the network is operated and desired processing is combined in a graph to bypass any stream to processing servers scattered on the network. A technique for processing a plurality of streams to which various processes can be applied via a route is known (for example, see Non-Patent Document 2).

Daisuke Yamamoto, Satoshi Nagao “Annotation and Online Video Content Annotation by Viewers”, Transactions of the Japanese Society for Artificial Intelligence 2005 Satoshi Kondoh, Takaaki Moriya, Hiroyuki Ohnishi, and Miki Hirano, “A method of bridging and processing media stream on network”, IEEE Globecom 2008 CS01M1-5.

  However, in the statistical annotation method described in Non-Patent Document 1, the main purpose is to select highly reliable annotations that are only targeted within the browsing system, and the video recognition portion has not been determined. As it is, the recognition function could not be provided to some other system.

  In the technique described in Non-Patent Document 2, the actual media recognition server is specialized for various uses. When it is assumed that only media streams whose use is unknown are transferred, the contents of the media data However, there is a problem that an appropriate recognizer cannot be selected because the contents (context) cannot be grasped in advance.

  The present invention has been made in view of such circumstances, and an object of the present invention is to automatically generate and update an appropriate recognizer from information existing on the Web. An object is to provide a metadata extraction server, a metadata extraction method, and a program capable of selecting an appropriate recognizer from itself.

  In order to solve the above problem, the present invention provides teacher information for collecting content from a site on a network in a metadata extraction server that extracts metadata to recognize the content (context) of media data based on media data. A collection unit; a feature amount calculation unit that calculates a feature amount matrix representing the content of the content collected by the teacher information collection unit; and a feature amount matrix calculated by the feature amount calculation unit is stored at regular time intervals. In this case, clustering is performed on a set of feature amount matrices calculated by the feature amount calculation unit, and a feature amount storage unit that determines a partial space for each set of feature amount matrices, and content collected by the teacher information collection unit A metadata organizing unit that performs a morphological analysis on the image and generates a metadata matrix that represents the frequency of occurrence of the morphologically analyzed word; A correlation coefficient calculating unit that calculates a correlation coefficient between the set of feature matrixes and the set of metadata matrices generated by the metadata organizing unit for each partial space determined by the collection amount storing unit; A metadata extraction server comprising a correlation coefficient storage unit that stores a correlation coefficient calculated by the correlation coefficient calculation unit for each partial space.

  The present invention further includes a media stream acquisition unit that acquires a media stream transmitted and received on a network, and the feature amount calculation unit represents a feature amount that represents the content of the media stream acquired by the media stream acquisition unit. The feature amount storage unit calculates a matrix, and in addition to the feature amount matrix representing the content content collected by the teacher information collection unit, the feature amount representing the content of the media stream acquired by the media stream acquisition unit When storing a matrix at regular time intervals, clustering is performed on a set of feature matrices representing the contents of the media stream calculated by the feature calculator, and a subspace is determined for each set of feature matrices. This is a metadata extraction server characterized by that.

  Further, the present invention specifies a subspace including a feature amount matrix representing the content of the media stream based on the subspace determined by the feature amount storage unit, and is calculated by the correlation coefficient calculation unit Applying a correlation coefficient corresponding to the specified subspace to calculate a metadata vector of the feature vector, and selecting a set determined as an appropriate word from the metadata vector as one metadata A correlation setting unit; a transmission setting unit that determines whether the metadata selected by the correlation applying unit is transmitted as the metadata itself or embedded in the media stream; and When it is determined that the metadata itself is to be transmitted, a metadata transmission unit that transmits the metadata to a metadata receiving terminal that is a specified destination; A metadata embedding unit that embeds the metadata in the media stream, and a medium in which the metadata is embedded by the metadata embedding unit when it is determined by the determination unit to embed the metadata in the media stream for transmission; A metadata extraction server comprising: a media stream sending unit that sends a stream to a media stream receiving terminal that is a designated destination.

  The transmission setting unit may selectively determine, as the designated destination, either a transmission source of the media stream or a transmission destination different from the transmission source. It is a data extraction server.

  Further, the present invention provides a metadata extraction method in a metadata extraction server that extracts metadata for recognizing the content (context) of media data based on the media data. A teacher information collection step for collecting the feature amount, a feature amount calculation unit for calculating a feature amount matrix representing the contents of the content collected in the teacher information collection step, and a feature amount storage unit for the feature amount When storing the feature quantity matrix calculated in the calculation step at regular time intervals, clustering is performed on the feature quantity matrix set calculated in the feature quantity calculation step, and a subspace is determined for each feature quantity matrix set. A feature amount storing step and a metadata organizing unit for the content collected in the teacher information collecting step. A metadata organizing step that performs morphological analysis and generates a metadata matrix that represents the frequency of occurrence of words subjected to morphological analysis, and a correlation coefficient calculation unit, for each subspace determined in the feature amount storing step, A correlation coefficient calculating step for calculating a correlation coefficient between the set of quantity matrices and the set of metadata matrices generated in the metadata organizing step; and a correlation coefficient storage unit for each of the subspaces, the correlation coefficient And a correlation coefficient storage step for storing the correlation coefficient calculated in the calculation step.

  Further, the present invention provides a teacher information collecting means for collecting content from a site on a network using a computer operating as a metadata extraction server for extracting metadata for recognizing the content (context) of the media data based on the media data. And a feature amount calculating means for calculating a feature amount matrix representing the contents collected by the teacher information collecting means, and a feature amount matrix calculated by the feature amount calculating means when storing the feature amount matrix at a constant time interval. Clustering with a set of feature quantity matrices calculated by the feature quantity calculation means, and a feature quantity storage means for determining a subspace for each set of feature quantity matrices; and for contents collected by the teacher information collection means To generate a metadata matrix that represents the frequency of occurrence of words that have undergone morphological analysis And a correlation coefficient for calculating a correlation coefficient between the feature matrix set and the metadata matrix set generated by the metadata organizing unit for each partial space determined by the processing unit and the feature storage unit A program for operating as a calculation means and a correlation coefficient storage means for storing the correlation coefficient calculated by the correlation coefficient calculation means for each partial space.

  According to the present invention, the metadata extraction server includes a teacher information collection unit that collects content from a site on the network, and a feature amount calculation that calculates a feature amount matrix representing the content of the content collected by the teacher information collection unit. And the feature amount matrix calculated by the feature amount calculation unit when the feature amount matrix calculated by the feature amount calculation unit is stored at a predetermined time interval, and the feature amount matrix set is clustered. A metadata storage unit that performs a morpheme analysis on the content collected by the feature information storage unit and the teacher information collection unit that determines a partial space for each time, and generates a metadata matrix that represents the frequency of occurrence of words subjected to the morpheme analysis And a set of feature matrixes and a metadata matrix generated by the metadata organizing unit for each partial space determined by the feature storage unit It has a correlation coefficient calculation unit for calculating a correlation coefficient between case and a correlation coefficient storage unit for storing a correlation coefficient calculated by the correlation coefficient calculation unit for each of the subspace. With this configuration, an appropriate recognizer can be automatically generated and updated from information existing on the Web.

  In addition, according to the present invention, the image processing apparatus further includes a media stream acquisition unit that acquires a media stream transmitted and received on the network, and the feature amount calculation unit represents the content of the media stream acquired by the media stream acquisition unit. A feature amount matrix is calculated, and the feature amount storage unit represents the content of the media stream acquired by the media stream acquisition unit in addition to the feature amount matrix that represents the content of the content collected by the teacher information collection unit. When the feature amount matrix is stored at regular time intervals, clustering is performed on a set of feature amount matrices representing the contents of the media stream calculated by the feature amount calculation unit, and a subspace is set for each set of feature amount matrices. To decide. With this configuration, an appropriate recognizer can be selected from the media stream itself.

It is a block diagram which shows the structure of the whole web cooperation metadata extraction system in one Embodiment of this invention. It is a block diagram which shows the structure of the metadata extraction server in this embodiment. It is the sequence diagram which showed the procedure which extracts metadata from a media stream by the metadata extraction server of this embodiment. It is the sequence diagram which showed the procedure which acquires a metadata converter and a correlation coefficient in this embodiment. It is the conceptual diagram which showed the structure and operation | movement at the time of applying the metadata extraction server of this invention to search service. It is the conceptual diagram which showed the structure and operation | movement at the time of comprising the metadata extraction server of this invention in cooperation with a bridge as a metadata extraction bridge. It is the conceptual diagram which showed the structure and operation | movement at the time of using the metadata extraction server of this invention for the stream backup service by a network.

(First embodiment)
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the entire Web collaboration metadata extraction system according to this embodiment. In the figure, a Web cooperation metadata extraction server 101 (hereinafter referred to as a metadata extraction server) is connected to user terminals 101-1 and 101-2 via a communicable network 100. The metadata extraction server 1-1 receives a media stream transmitted from the user terminal 102-1 used by the user. The metadata extraction server 101 extracts information describing the contents from the received media stream 103, generates another data stream 104, and sends it to the user terminal 102-2, which is a destination specified separately in advance. Send.

  Next, the configuration of the metadata extraction server 101 will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the metadata extraction server 101 in this embodiment. The metadata extraction server 101 includes a media stream acquisition unit 208, a feature amount calculation unit 204, a feature amount DB 201, a metadata DB 202, a metadata organization unit 203, a correlation coefficient DB 205, and a correlation coefficient calculation unit 214. A correlation coefficient application unit 207, a transmission setting unit 230, a metadata transmission unit 231, a metadata embedding unit 232, a media stream transmission unit 233, and a teacher information collection unit 234.

  The media stream acquisition unit 208 acquires a media stream transmitted from the media stream transmission terminal 210 on the network 100 and inputs the media stream to the feature amount calculation unit 204 and the metadata embedding unit 232. The feature amount calculation unit 204 calculates a feature amount vector from the input media stream, and inputs the calculated feature amount vector to the correlation coefficient application unit 207. The feature value DB 201 stores the calculated feature value vectors at regular time intervals.

  The teacher information collection unit 234 accesses the Web site 240 (the moving image posting site 240-1 and the blog site 240-2), acquires arbitrary content, and inputs it to the feature amount calculation unit 204 as media data. In addition, the teacher information collection unit 234 inputs media data to the metadata organization unit 203. The metadata organizing unit 203 performs morphological analysis on the input media data. The correlation coefficient calculation unit 214 creates a metadata vector representing the frequency of occurrence of words morphologically analyzed by the metadata organizing unit 203 with a time width Δt used for averaging feature quantities. The metadata DB 202 stores the metadata vector created by the correlation coefficient calculation unit 214.

  In addition, the correlation coefficient calculation unit 214 calculates the average feature vector by averaging the feature vectors stored in the feature DB 201 with the time width Δt, and also detects scene breaks based on changes in the feature. Then, the feature vector is calculated for the media data frame, and the obtained feature vector is input to the correlation coefficient DB 205. The correlation coefficient DB 205 stores the metadata converter R corresponding to the partial space ID and the correlation coefficient W.

  The correlation coefficient application unit 207 reads the metadata converter R corresponding to the partial space ID, the correlation coefficient W, and the correlation coefficient W from the correlation coefficient DB 205, and identifies which partial space the feature quantity vector belongs to. In addition, correlation coefficient application section 207 specifies which partial space the feature quantity vector enters, and then applies correlation coefficient W corresponding to the partial space to the feature quantity vector to calculate a metadata vector. To do. Further, the correlation coefficient application unit 207 selects a set determined as an appropriate word from the calculated metadata vector, and inputs it to the transmission setting unit 230 as one metadata.

  When the metadata is embedded in the media stream for output, the transmission setting unit 230 inputs the metadata to the media stream embedding unit 232. The media stream embedding unit 232 embeds the metadata input from the transmission setting unit 230 in the media stream input from the media stream acquisition unit 208 and inputs the metadata to the media stream transmission unit 233. The media stream sending unit 233 sends the media stream in which the metadata is embedded to the designated destination media stream receiving terminal 250.

  On the other hand, when outputting metadata alone, the transmission setting unit 230 inputs the metadata to the metadata sending unit 231. The metadata transmission unit 231 adds a packet header to the metadata input from the transmission setting unit 230 and transmits the metadata to the metadata reception terminal 251 that is the designated destination.

  As described above, the transmission setting unit 230 can set whether to embed metadata in a media stream for transmission or to output metadata alone. In addition, a transmission destination can be designated. For example, specifying whether to send a media stream or metadata with embedded metadata to the user terminal that sent the media stream, or to send a media stream or metadata with embedded metadata to another user terminal Can do.

  The metadata is, for example, data in XML (eXtensible Markup Language) format defined by MPEG-7 or the like.

  The media stream receiving terminal 250 is a terminal that can receive a media stream in which metadata is embedded. The metadata receiving terminal 251 is a terminal that can receive metadata.

  The web site 240 is, for example, a video posting site 240-1 or a blog site 240-2. The video posting site 240-1 is a site that provides video content. The blog site 240-2 is a site that provides text content.

  Next, the operation of the metadata extraction server of this embodiment will be described with reference to FIG. FIG. 3 is a sequence diagram showing a procedure for extracting metadata from a media stream by the metadata extraction server 101 of this embodiment.

  The media stream acquisition unit 208 first performs transmission setting in the transmission setting unit 230 (step S1), and then receives the media stream transmitted from the media stream transmission terminal 210 and inputs it to the feature amount calculation unit 204 (step S1). S2). The feature amount calculation unit 204 inputs the input media stream to the metadata embedding unit 232 (step S3). Also, the feature amount calculation unit 204 calculates a feature amount vector from the input media stream, and inputs the calculated feature amount vector to the correlation coefficient application unit 207 (step S4).

  Here, the media stream refers to data communicated by a media compatible protocol such as RTP (Real-time Transport Protocol), RTSP (Real Time Streaming Protocol), RTMP (Real Time Message Protocol). . In addition, it is assumed that a feature amount vector calculated is determined in advance in the metadata extraction server 101. At this time, the feature quantity DB 201 accumulates the calculated feature quantity vectors at regular time intervals (step S5).

  The correlation coefficient application unit 207 acquires the metadata converter R corresponding to the partial space ID and the correlation coefficient W from the correlation coefficient DB 205 (steps S6 and S7). In addition, the correlation coefficient application unit 207 specifies which partial space the feature amount vector calculated by the feature amount calculation unit 204 enters. The metadata converter R and the correlation coefficient W stored in the correlation coefficient DB 205 will be described later.

  Subsequently, after the correlation coefficient application unit 207 identifies which partial space the feature quantity vector calculated by the feature quantity calculation unit 204 enters, the correlation coefficient W corresponding to the identified partial space is determined as the feature quantity. Apply to the vector to calculate the metadata vector. Subsequently, the correlation coefficient application unit 207 selects a set determined as an appropriate word from the calculated metadata vector, generates one metadata, and inputs the generated metadata to the transmission setting unit 230. (Step S8). The metadata generated by the correlation coefficient application unit 207 is XML format data defined by MPEG-7 or the like as described above.

  The transmission setting unit 230 creates a packet header and determines whether to transmit the metadata itself or embed it in the original media stream. As described above, the transmission setting unit 230 selects the output result form and the transmission destination form from two forms. For example, as a form of output results, one form is selected from two forms: a form in which metadata is embedded in a media stream and a form in which metadata is output alone. Also, the form of the transmission destination is the form of transmitting the media stream or metadata in which the metadata is embedded to the user terminal that has transmitted the media stream, or the user terminal that is different from the user terminal that has transmitted the media stream. One form is selected from two forms of transmitting a media stream or metadata in which data is embedded.

  When the transmission setting unit 230 selects a form in which the metadata is embedded in the media stream for output, the transmission setting unit 230 inputs the metadata to the media stream embedding unit 232 (step S9). The media stream embedding unit 232 embeds the metadata input from the transmission selection unit 230 in step S9 into the media stream input from the media stream acquisition unit 208 in step S3, and inputs it to the media stream transmission unit 233 (step S10). ).

  Subsequently, the media stream transmission unit 233 transmits the transmission destination (media stream reception terminal 250) selected by the transmission setting unit 230 (step S11). Note that the media stream embedding unit 232 sets a number that is not used for the RTP dynamic payload type of the media stream, and inserts metadata into the corresponding payload.

  On the other hand, when the transmission setting unit 230 selects a mode for outputting metadata alone, the transmission setting unit 230 inputs the metadata to the metadata transmission unit 231 (step S12). The metadata transmission unit 231 attaches a packet header to the input metadata and transmits the packet header to the transmission destination (metadata reception terminal 251) set by the transmission setting unit 230 (step S13).

  Next, a method for obtaining the metadata converter R and the correlation coefficient W stored in the correlation coefficient DB 205 will be described with reference to FIG. FIG. 4 is a sequence diagram illustrating a procedure for obtaining the metadata converter R and the correlation coefficient W.

  The teacher information collection unit 234 accesses the Web site 240 and acquires arbitrary content (step S21). The teacher information acquisition unit 234 inputs the acquired content as media data to the feature amount calculation unit 204 (step S22). The feature amount calculation unit 204 calculates a feature amount vector from the input media data, and inputs the calculated feature amount vector to the feature amount DB 201 (step S23). The feature quantity vector is calculated using the formula (1) by averaging the feature quantities of the input media data over a certain time Δt. Here, k is a quantization number, and m is the number of feature dimensions.

  Note that the feature amount calculation unit 204 detects a scene break in the media data based on a change in the feature amount of the input media data. The feature amount calculation unit 204 calculates a feature amount vector for each frame of media data, and inputs the calculated feature amount vector to the feature amount DB 201.

  The feature DB 201 stores the input feature vector when the input feature vector has a feature that can be classified into a past cluster. Further, the feature amount DB 201 discards the input feature amount vector when the input feature amount vector does not have a feature amount that can be classified into a past cluster. Further, the feature amount DB 201 inputs information indicating whether the input feature amount vector is stored or discarded to the teacher information collecting unit 234 (step S24).

Further, the feature amount DB 201 performs clustering of feature amount vectors stored in the feature amount DB 201 at regular time intervals. The feature amount DB 201 performs clustering by a classification method based on unsupervised learning such as an X-means method. The feature DB 201 organizes the partial spaces obtained by clustering using a tree structure such as a kd tree, and assigns a partial space ID to each partial space. In addition, the feature value DB 201 sets a set of feature vectors in the subspace obtained by clustering as [v ₀ ,..., V _n−1 ], and performs principal component analysis on the set of feature vectors. The number of dimensions is reduced to create [v ′ ₀ ,..., V ′ _n−1 ].

  The teacher information collection unit 234 organizes the text data of the media data as metadata only when the feature value DB 201 stores the feature value vector based on information indicating whether the feature value DB 201 stores or discards the feature value vector. Input to the unit 203 (step S25). The metadata organizing unit 203 performs morphological analysis of text data, and generates a metadata vector yi representing the occurrence frequency of words within the time width Δt used for averaging feature quantity vectors from the words subjected to morphological analysis (2). Calculate according to the formula. Further, the metadata organizing unit 203 inputs the calculated metadata vector to the metadata DB 202 (step S26).

  Here, i is the number of types of words. The value of the word occurrence frequency is assumed to be a Gaussian distribution with the number of occurrence times being multiplied by 1. In addition, the distribution is assumed to be the above-mentioned scene break and no further diffusion.

Further, the metadata DB 202 sets a set of metadata vectors as [y ₀ ,..., Y _n-1 ], and performs a principal component analysis on the set of metadata vectors to reduce the number of dimensions. ₀ ,..., Y ′ _n−1 ] are created.

  The correlation coefficient calculation unit 214 reads the partial space ID, the set of feature vectors, and the set of metadata vectors from the feature DB 201 and the metadata DB 202 (Steps S27 to S30). Further, the correlation coefficient calculation unit 214 calculates a transformation matrix R (metadata transformer R) from the set of feature vectors and the set of metadata vectors. In addition, the correlation coefficient calculation unit 214 performs supervised learning such as logistic regression analysis using the equation (3) in a model in which the average feature vector v ′ is input and the metadata vector y ′ is output. However, in the case of logistic regression, f (x) = a / (1-exp (−x)) is set, and a is the maximum value of the elements of the metadata vector.

  The correlation coefficient calculation unit 214 calculates a regression coefficient matrix as the correlation coefficient W. Subsequently, the correlation coefficient calculation unit 214 inputs the metadata converter R, the correlation coefficient W, and the partial space ID into the correlation coefficient DB 205.

  The correlation coefficient DB 205 stores the input metadata converter R, the correlation coefficient W, and the partial space ID in association with each other.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. The second embodiment is an example in which the metadata extraction server 101 described in the first embodiment is applied to a search service. FIG. 5 is a conceptual diagram showing the configuration and operation when the metadata extraction server 101 is applied to a search service.

  The user transmits a video stream shot using the user terminal 102-3 in various places to the metadata extraction server 101 (step S41). The metadata extraction server 101 extracts metadata in real time from the received video stream, and transmits the data embedded in the received video stream as a data stream to the user terminal 102-3 (step S42). . Further, the metadata extraction server 101 transmits the extracted metadata to the search DB 260. The search DB searches for information based on the transmitted metadata, and transmits the search result to the metadata extraction server. The metadata extraction server 101 transmits the received search result to the user terminal 102-3 (step S43).

  Thus, the user can acquire and search keywords on the Web related to the captured video simply by capturing the surroundings with the user terminal 102-3 equipped with a camera such as a mobile phone. This not only reduces the number of search inputs, but also makes it possible to recommend search keywords to the user even when a clear search keyword cannot be conceived.

(Third embodiment)
Next, a third embodiment of the present invention will be described. FIG. 6 is a conceptual diagram showing the configuration and operation when the metadata extraction server of the present invention is configured as a metadata extraction bridge in cooperation with a bridge. Note that the metadata extraction server 101 in the third embodiment has the same configuration as the metadata extraction server 101 described in the first embodiment.

  Here, the stream reply destination of the metadata extraction server 101 is set to the stream transmission source address, and the reply stream is set to the metadata embedded media stream. In addition, a control bridge (refer to Non-Patent Document 1) is used as a bridge, and one or many control bridges 270 are provided on the network 100. In the illustrated example, only one control bridge is provided.

  Also, the video from the live camera 280 installed on the street or the video from the video phone function provided in the user terminal 102-4 is transmitted to the terminal user terminal 102-5 via the network 100. It shall be.

  The control bridge 270 captures a specific media stream by using a forwarding function, rewrites a packet header constituting the media stream, and transfers the media stream to the metadata extraction server 101 (step). S51). In this case, the specific media stream is assumed to be a video from a live camera 280 installed on the street or the video phone function provided in the user terminal 102-4. That is, the user terminal 102-5 at the end is browsing the video of the live camera 280 and communicating with the user terminal 102-4 (telephone call).

  Next, the metadata extraction server 101 that has received the media stream embeds metadata in the media stream (step S52), and returns the metadata-embedded media stream to the control bridge 270 (step S53). Finally, the control bridge 270 restores the packet header of the returned stream to the original one and sends it again on the network 100 (step S54). As a result, the user terminal 102-5 at the end has low processing performance because metadata is automatically added to the media stream during communication over the network 100 without performing complicated processing. Even for a stream transmitted from a mobile terminal or the like, a keyword suitable for the media stream can be provided.

  Also, by notifying the user terminal 102-6 of the supervisor in advance of the metadata payload type from the metadata extraction server 101 (step S55), the monitoring keyword is sent from the user terminal 102-6 to the proxy / PBX 290. By transmitting (step S56) and acquiring corresponding detection information from the metadata embedded media stream (step S57), the stream can be monitored on the network 100.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. FIG. 7 is a conceptual diagram showing the configuration and operation when the metadata extraction server 101 of the present invention is used for a stream backup service over a network. Note that the metadata extraction server 101 in the fourth embodiment has the same configuration as the metadata extraction server described in the first embodiment.

  First, the control bridge 270 described above is provided on the network 100. The control bridge 270 uses a mirroring function to capture a media stream from the live camera 280 or the videophone function provided in the user terminal 102-4, and duplicates the metadata stream to the metadata extraction server 101. It transmits to media DB300 (step S61, S62). Thereafter, the metadata extraction server 101 transmits the extracted metadata to the media DB 300 (step S63). The media DB 300 stores a pair with corresponding media data.

  It is assumed that the media DB 300 discards the media data and metadata in a certain time. Further, the media DB 300 can search for metadata by a keyword, and the time range of the corresponding media can be known. As a result, the user terminal 102-7 transmits a search keyword to the media DB 300 (step S64), thereby recording communication between terminals having a small recording capacity, and past media streams that have been forgotten to be recorded by the terminal. Search is possible (step S65), and convenience can be improved.

  According to the above-described embodiment, it is possible to improve accuracy by creating and recognizing a recognizer for each partial space. Moreover, by dividing into subspaces, not only the essential dimension of the feature quantity is reduced, but also the types of collected comments are naturally limited, so the types of output metadata are also limited, and the correlation coefficient The number of dimensions is also reduced, and the amount of calculation required when extracting metadata from the media stream can be reduced. In addition, since it is not necessary to attach any information to the media stream, it is possible even when other accompanying information that can be recognized by a device on the network side cannot be expected.

The above-described process of operation of the metadata extraction server 101 can be used as a program to be executed by a computer or a computer-readable recording medium as the program. Processing is performed. The “computer system” here includes a CPU, various memories, an OS, and hardware such as peripheral devices.
Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
The program may be for realizing a part of the functions described above.
Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

  DESCRIPTION OF SYMBOLS 100 ... Network, 101 ... Metadata extraction server, 102-1 to 102-7 ... User terminal, 201 ... Feature-value DB, 202 ... Metadata DB, 203 ... Metadata Arrangement unit, 204 ... feature amount calculation unit, 205 ... correlation coefficient DB, 207 ... correlation coefficient application unit, 208 ... media stream acquisition unit, 210 ... media stream transmission terminal, 214 ... correlation coefficient calculation unit, 230 ... transmission setting unit, 231 ... metadata sending unit, 232 ... metadata embedding unit, 233 ... media stream sending unit, 234 ... teacher information Collection unit, 240 ... Web server, 240-1 ... Video posting site, 240-2 ... Blog site, 250 ... Media stream receiving terminal, 251 ... Meta Over data receiving terminal, 260 ... search DB, 270 ··· control bridge, 280 ... live camera, 290 ... Puroshiki / PBX, 300 ··· media DB

Claims (6)

In a metadata extraction server that extracts metadata to recognize the content of media data based on media data,
A teacher information collection unit that collects content from sites on the network;
A feature amount calculation unit that calculates a feature amount matrix representing the contents of the content collected by the teacher information collection unit;
When storing the feature amount matrix calculated by the feature amount calculation unit at a constant time interval, clustering is performed on the set of feature amount matrices calculated by the feature amount calculation unit, and a part of each feature amount matrix set is stored. A feature storage unit for determining a space;
A metadata organizing unit that performs morphological analysis on the content collected by the teacher information collecting unit and generates a metadata matrix that represents the frequency of occurrence of words subjected to morphological analysis;
A correlation coefficient calculating unit that calculates a correlation coefficient between the set of feature matrixes and the set of metadata matrices generated by the metadata organizing unit for each partial space determined by the feature storage unit;
A metadata extraction server, comprising: a correlation coefficient storage unit that stores the correlation coefficient calculated by the correlation coefficient calculation unit for each partial space. A media stream acquisition unit that acquires media streams transmitted and received on the network;
The feature amount calculation unit includes:
Calculating a feature amount matrix representing the contents of the media stream acquired by the media stream acquisition unit;
The feature amount storage unit includes:
In addition to the feature amount matrix that represents the content of the content collected by the teacher information collection unit, when storing the feature amount matrix that represents the content of the media stream acquired by the media stream acquisition unit at a predetermined time interval, Clustering with a set of feature amount matrices representing the content of the media stream calculated by the feature amount calculation unit, and determining a subspace for each set of feature amount matrices;
The metadata extraction server according to claim 1, wherein: Based on the partial space determined by the feature quantity storage unit, a partial space including a feature quantity matrix representing the content of the media stream is specified, and the specified part calculated by the correlation coefficient calculation unit A correlation application unit that calculates a metadata vector of the feature vector by applying a correlation coefficient corresponding to a space, and selects a set determined as an appropriate word from the metadata vector as one metadata;
A transmission setting unit that determines whether the metadata selected by the correlation application unit is transmitted as the metadata itself or embedded in the media stream;
A metadata sending unit that sends the metadata to a specified destination metadata receiving terminal when the transmission judging unit decides to send the metadata itself;
A metadata embedding unit that embeds the metadata in the media stream when the transmission determination unit determines to embed and transmit the metadata in the media stream;
The metadata extraction server according to claim 2, further comprising: a media stream sending unit that sends the media stream in which the metadata is embedded by the metadata embedding unit to a media stream receiving terminal that is a designated destination. The transmission setting unit
As the designated destination, either the transmission source of the media stream or a transmission destination different from the transmission source is selectively determined.
The metadata extraction server according to claim 3. In a metadata extraction method in a metadata extraction server that extracts metadata to recognize the content of media data based on media data,
A teacher information collection unit for collecting content from a site on the network;
A feature amount calculating unit that calculates a feature amount matrix representing the content of the content collected in the teacher information collecting step;
When the feature amount storage unit stores the feature amount matrix calculated in the feature amount calculation step at a constant time interval, the feature amount storage unit performs clustering on the set of feature amount matrices calculated in the feature amount calculation step, and the feature amount A feature storage step for determining a subspace for each set of matrices;
A metadata organizing step for performing a morphological analysis on the content collected in the teacher information collecting step, and generating a metadata matrix representing a frequency of occurrence of words subjected to the morphological analysis;
A correlation coefficient calculation unit calculates a correlation coefficient between the set of feature amount matrices and the set of metadata matrices generated in the metadata organization step for each partial space determined in the feature amount storage step. A correlation coefficient calculating step;
A correlation coefficient storage unit that stores the correlation coefficient calculated in the correlation coefficient calculation step for each partial space;
A metadata extraction method comprising: A computer that operates as a metadata extraction server that extracts metadata for recognizing the contents of media data based on the media data.
Teacher information collection means to collect content from sites on the network;
A feature quantity calculating means for calculating a feature quantity matrix representing the contents of the content collected by the teacher information collecting means;
When storing the feature quantity matrix calculated by the feature quantity calculation means at a fixed time interval, clustering is performed on the set of feature quantity matrices calculated by the feature quantity calculation means, and a part is set for each set of feature quantity matrices. Feature quantity storage means for determining space;
Metadata organizing means for performing morphological analysis on the content collected by the teacher information collecting means, and generating a metadata matrix representing the frequency of occurrence of words subjected to morphological analysis;
Correlation coefficient calculating means for calculating a correlation coefficient between the set of feature quantity matrices and the set of metadata matrices generated by the metadata organizing means for each subspace determined by the feature quantity storage means;
The program for making it operate | move as a correlation coefficient memory | storage means to memorize | store the correlation coefficient calculated by the said correlation coefficient calculation means for every said partial space.

Images