KR101720685B1 - Apparatus and Method for Web Data based Identification System for Object Identification Performance Enhancement - Google Patents

Abstract

An apparatus and method for verifying a broadcast content object based on web data, the apparatus comprising: a web data processing unit for collecting and processing web data associated with broadcast content to generate content knowledge information in the form of tagging the broadcast content; And an object identification result verifying unit for verifying an identification result of the object included in the broadcast content using the content knowledge information.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and method for identifying a broadcast content object based on web data,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a broadcast communication technology, and more particularly, to a technique for identifying an object existing in broadcast content.

In general, a technique applied to object identification in broadcast content is to extract a feature from a specific scene (frame) of broadcast content using an image processing technique, and to select an object that best represents the feature from the identified object group. A technique applied for object identification in broadcast content is described in the patent "Object Identification in Images Sequences " (U.S. Patent Application No. 2013-792483).

However, the current level of image processing technology is AP (Average Precision) of about 0.45, and it drops exponentially as the size of object group increases. In a situation where various kinds of objects such as characters, vehicles, places, and carry items appear in broadcast contents, it is not enough to apply only image processing technology to object identification considering commercialization.

The present invention provides an apparatus and method for verifying and verifying an object that improves the performance of an object identification result obtained from an image processing technique by performing verification using information obtained from outside the broadcast content.

An apparatus and method for verifying a broadcast content object based on web data, the apparatus comprising: a web data processing unit for collecting and processing web data associated with broadcast content to generate content knowledge information in the form of tagging the broadcast content; And an object identification result verifying unit for verifying an identification result of the object included in the broadcast content using the content knowledge information.

The present invention relates to a Web data-based broadcast content object identification and validation method, comprising the steps of: collecting and processing web data associated with broadcast content to generate content knowledge information in the form of tagging the broadcast content; And verifying the identification result using the content knowledge information.

By improving object identification performance for broadcast contents, it can be used for object unit search, recommendation, mashup, and the like.

FIG. 1 is a block diagram of a web data based broadcast content object identification and verification apparatus according to an embodiment of the present invention. Referring to FIG.
2 is a block diagram of a web data processing unit according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating an example in which a knowledge network according to the present invention is tagged to broadcast contents.
4 is a block diagram of an object identification result verifying unit according to an embodiment of the present invention.
FIG. 5 is a flow chart for explaining a web data based broadcast content object identification verification method according to an embodiment of the present invention.
6 is a flowchart illustrating a process of generating content knowledge information using web data according to an embodiment of the present invention.
7 is a flowchart illustrating an object identification result verification step according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

Before describing the present invention, it is to be understood that the terms used throughout the specification are subject to change in accordance with the intentions and customs of the user or operator, and therefore the definitions of these terms should be based on the contents of the specification of the present invention will be.

The present invention proposes a technique for performing verification using information obtained from outside of broadcast contents on object identification results obtained from an image processing technique. Here, the broadcast content is not only rapidly transmitted to a large number of users as compared with a general image, but also reproduces contents frequently by a user. For example, in the case of the drama content "You are from the Star", the number of blog postings related to the broadcast period of about 3 months has exceeded 40,000.

Accordingly, noting that the web data associated with the broadcast content includes not only the information over the broadcast content but also the data thereof, the present invention utilizes the information extracted from the web data for object identification verification.

1 is a block diagram illustrating a web data based broadcast content object identification and verification apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the web data-based object identification and verification apparatus 100 includes a web data processing unit 110 for collecting and processing web data associated with broadcast contents to generate content knowledge information in the form of tagging broadcast contents, A content knowledge information storing unit 120 for storing content knowledge information, and an object identification result verifying unit 130 for verifying an identification result of an object included in the broadcast content using content knowledge information.

The broadcast content DB 10 includes a broadcast content request book 11 and a broadcast content image 12 in which terms included in broadcast content are stored. The broadcast content DB 10 may be stored in a broadcaster server or may be provided in various forms. The broadcast content DB 10 provides information related to the broadcast content to the web data processing unit 110 so that the web data processing unit 110 generates data that can verify the object identification result by the object identifier 20 .

The object identifier 20 identifies an object in the broadcast content image by using an image recognition technology, and the object identification result is verified by the object identification result verification unit 130.

2 is a block diagram of a web data processing unit according to an embodiment of the present invention.

Referring to FIG. 2, the web data processing unit 110 includes a web data collection unit 111, a knowledge network construction unit 112, and a knowledge network tagging unit 113.

The web data collection unit 111 collects web data using object-related keywords for broadcast contents. To this end, the web data collection unit 111 searches for a broadcast content-related glossary 11 existing in an external broadcast content DB, acquires keywords necessary for web data collection, combines the acquired keywords, And accesses the web portal 30 to collect Web data to be searched using the generated query. Here, the object related keywords may include words such as a content name, a main character name, a car, a place appearing in a product placement (PPL), and the like.

The knowledge network construction unit 112 constructs a knowledge network composed of at least one of terms and images extracted from the collected web data. Specifically, the knowledge network construction unit 112 includes an information / image extraction unit 112a and a knowledge network generation unit 112b. The information / image extracting unit 112a extracts web data, that is, terms and images included in a web page. The knowledge network generating unit 112b constitutes a knowledge network having nodes and images extracted by the information / image extracting unit 112a as nodes. The knowledge network will be described in more detail below with reference to FIG.

The knowledge network tagging unit 113 searches a frame including an image matching an image included in the knowledge network with the broadcast content image and tags the knowledge network on the retrieved frame. Specifically, the knowledge network tagging unit 113 includes a matching model learning unit 113a, A matching model generation unit 113b, an image matching unit 113c, and a tagging unit 113d.

The matching model learning unit 113a learns a matching model based on the frames constituting the broadcast content image 12 included in the broadcast content DB, and the matching model generation unit 113b generates the learned matching model.

In the matching model, the image matching unit 113c converts the feature vector X extracted from the broadcast content frame or the web images into X ' , and performs similarity measurement between the converted vectors. Here, the feature vector conversion is performed such that the degree of similarity between broadcast content frames satisfies Equation (1) as below.

&Quot; (1) "

In Equation (1), sim () denotes a similarity calculation function, and #x denotes a frame number of x. That is, the matching model satisfying Equation (1) is learned so that the closer the frame position is, the higher the degree of similarity.

According to one embodiment, the image matching unit 113c calculates the matching ranking once more by using a feature extraction technique used to construct the feature vector X and a feature extraction technique having different characteristics. That is, if color information is used to extract X , post-processing uses the information of the texture and boundary of the image to extract the feature vector Y By constructing and calculating the similarity, one more rank is adjusted for the n candidates obtained through the matching model. Various linear and nonlinear model learning techniques such as SVMRank and manifold learning can be applied to construct the matching model, and various feature extraction techniques can be used for feature extraction for post-processing.

The image matching unit 113c matches n frames with respect to one web image.

The tagging unit 113d selects the last frame of the highest score among the n frame numbers and tags the knowledge network for m frames before and after the last frame, Can be weighted based on the distance from the last determined frame.

The matching result and the knowledge network constructed through the knowledge network construction unit 120 are combined and stored in the content knowledge information storage unit 140. That is, the content information is set knowledge of the knowledge network _{G = g 1, g 2,} g 3, ..., a set of distribution for the g _m and tagging results obtained frame number _{T = t 1, t 2,} t 3,. .., it can be _{C = (g 1, t 1} ) combining the _{t m, (g 2, t} 2), (g 3, t 3), ..., (g m, t m).

FIG. 3 is a diagram illustrating an example in which a knowledge network according to the present invention is tagged to broadcast contents.

3, the knowledge network G 310 is composed of a plurality of nodes N 311 a and 311 b and edges E 312 connecting the nodes 311 a and 311 b. Here, the node N is a set of words 311a and 311b extracted from the collected web data. The edge (E) is constructed based on the importance information of the node (N) in the web page and the association information between the information. The image 311b included in the knowledge network 310 is matched with the frames 321 constituting the broadcast content 320 image.

4 is a block diagram of an object identification result verifying unit according to an embodiment of the present invention.

4, the object identification result verification unit 130 includes a knowledge network extraction unit 131, a graph search based probability calculation unit 132, a probability integration unit 133, an identification score reconstruction unit 134, And a final identification result output section 135.

C를 전달받는다. When receiving the object identification result, the knowledge network extracting unit 131 requests the content knowledge information storage unit 120 to store the knowledge network set C '

C is received.

O에 대해 다음의 <수학식 2>를 이용하여 출연 확률을 계산한다.The graph search based probability calculation unit 132 calculates probabilities of appearance of objects based on the object identification candidates O and C '. That is, the identification object o

O is calculated using Equation (2) below.

&Quot; (2) &quot;

C'에 대해 계산한다. In Equation (2), # f is the corresponding frame number at which the identification occurred. That is, the probability that the object o appears in the knowledge network g _c belonging to C 'is multiplied by the weight for the frame # f of each knowledge network,

C '.

The probability integrating unit 133 obtains the probability of the object o by summing the appearance probabilities calculated by the graph search based probability calculating unit 132. [

The identification-point re-configuration unit 134 performs verification of the identification result by readjusting the identification result of the object identifier 20 using the obtained probability value.

FIG. 5 is a flow chart for explaining a web data based broadcast content object identification verification method according to an embodiment of the present invention.

Referring to FIG. 5, the web data-based object identification and verification apparatus 100 collects and processes web data related to broadcast contents to generate content knowledge information in the form of tagging the broadcast content (S510) (S520) of verifying the identification result of the object included in the content knowledge information using the content knowledge information.

6 is a flowchart illustrating a process of generating content knowledge information using web data according to an embodiment of the present invention.

Referring to FIG. 6, the web data processing unit 110 includes a step S610 of collecting web data, a step S620 of building a knowledge network, and a step S630 of tagging a knowledge network.

In the step of collecting web data (S610), web data is collected using object-related keywords for broadcast contents. To this end, the web data collection unit 111 searches for a broadcast content-related glossary 11 existing in an external broadcast content DB, acquires keywords necessary for web data collection, combines the acquired keywords, And accesses the web portal 30 to collect Web data to be searched using the generated query. Here, the object related keywords may include words such as a content name, a main character name, a car, a place appearing in a product placement (PPL), and the like.

In step S620, the knowledge network is constructed by constructing a knowledge network composed of at least one of terms and images extracted from the collected web data. More specifically, step S621 of extracting information / image, (S622). &Lt; / RTI &gt; In step S621 of extracting information / image, terms and images included in web data, that is, a web page are extracted. In the step S622 of generating the knowledge network, a knowledge network is formed by using extracted terms and images as nodes.

In the knowledge network tagging step S630, a frame including an image matching an image included in the knowledge network is searched in the broadcast content image, and the knowledge network is tagged in the searched frame. More specifically, in the matching model learning step S631, A matching model generation step S632, an image matching step S633, and a step S633 of tagging the knowledge network in the frame.

The matching model learning unit 113a learns a matching model based on the frames constituting the broadcast content image 12 included in the broadcast content DB in step S631 and the matching model generation unit 113b generates the learned matching model (S632).

In the matching model, the image matching unit 113c converts the feature vector X extracted from the broadcast content frame or the web images into X ' , and performs similarity measurement between the converted vectors. Here, the conversion of the feature vector is performed such that the degree of similarity between broadcast content frames satisfies Equation (3) as below.

&Quot; (3) &quot;

sim(x', z'),if |(#x -#y)|

|(#x -#z)|for x, y, z in X ' , sim (x', y ')

sim (x ', z'), if | (#x - #y) |

| (#x - #z) |

In Equation (3), sim () denotes a similarity calculation function, and #x denotes a frame number of x. That is, the matching model satisfying Equation (3) is learned so that the similarity increases as the frame position becomes closer.

According to one embodiment, the image matching unit 113c calculates the matching ranking once more by using a feature extraction technique used to construct the feature vector X and a feature extraction technique having different characteristics. That is, if color information is used to extract X , post-processing uses the information of the texture and boundary of the image to extract the feature vector Y By constructing and calculating the similarity, one more rank is adjusted for the n candidates obtained through the matching model. Various linear and nonlinear model learning techniques such as SVMRank and manifold learning can be applied to construct the matching model, and various feature extraction techniques can be used for feature extraction for post-processing. The image matching unit 113c matches n frames with respect to one web image (S633)

The tagging unit 113d selects the last frame of the highest score among the n frame numbers and tags the knowledge network for m frames before and after the last frame, (S634) based on the distance from the finally determined frame.

7 is a flowchart illustrating an object identification result verification step according to an embodiment of the present invention.

Referring to FIG. 7, the step of verifying the object identification result includes steps S710 to S710, S720 to S720, S730 to S730, S740 to S740, And a final identification result output step S750.

C를 전달받는다(S710). When receiving the object identification result, the knowledge network extracting unit 131 requests the content knowledge information storage unit 120 to store the knowledge network set C '

C (S710).

O에 대해 다음의 <수학식 4>를 이용하여 출연 확률을 계산한다(S720).The graph search based probability calculation unit 132 calculates probabilities of appearance of objects based on the object identification candidates O and C '. That is, the identification object o

O is calculated using the following Equation (4) (S720).

&Quot; (4) &quot;

C'에 대해 계산한다. In Equation (4), # f is the corresponding frame number at which the identification occurred. That is, the probability that the object o appears in the knowledge network g _c belonging to C 'is multiplied by the weight for the frame # f of each knowledge network,

C '.

The probability integration unit 133 obtains the probability of the object o by summing the appearance probabilities calculated by the graph search based probability calculation unit 132 (S730).

The identification point re-configuration unit 134 performs verification of the identification result by rearranging the identification result of the object identifier 20 using the obtained probability value (S740), and outputs the rearranged final identification result (S750).

Claims (16)

A web data processing unit for collecting and processing web data associated with broadcast content to generate content knowledge information in the form of tagging the broadcast content;
A content knowledge information storage unit for storing the content knowledge information;
And an object identification result verifying unit for verifying an identification result of an object included in the broadcast content using the content knowledge information,
The object identification verifying unit
Wherein the identification result is readjusted by calculating an appearance probability for the object based on the object identification candidate group and the knowledge network set associated with the frame including the identified object.
The apparatus of claim 1, wherein the web data processing unit
A web data collection unit for generating a query using object-related keywords for broadcast contents, requesting a search for a web portal based on the generated query, and collecting web data;
A knowledge network construction unit for building a knowledge network composed of at least one of words and images extracted from the collected web data;
And a knowledge network tagging unit for tagging the knowledge network in a predetermined frame period of the associated broadcast content.
3. The system of claim 2,
Wherein each of the nodes comprises at least one of images and words extracted from the web data, and a connection edge between the nodes based on the importance and association of the information corresponding to the nodes. A broadcast content object identification verification device.
3. The system according to claim 2, wherein the knowledge network tagging unit
And tagging the image extracted from the web data to a frame of the object identification verification target broadcast content.
5. The system according to claim 4, wherein the knowledge network tagging unit
Wherein a matching model is learned based on broadcast content frames and an image matching ranking is calculated by measuring similarity between feature vectors extracted from broadcast content frames or web images in the learned matching model. Identification verification device.
6. The method of claim 5,
Wherein the matching ranking is calculated once more by using a feature extraction technique using features different from the features used for the similarity measurement.
6. The system according to claim 5, wherein the knowledge network tagging unit
Selecting a last frame having the highest score and tagging the knowledge network with respect to a predetermined number of frames before and after the selected last frame, and the predetermined number of frames have a weight based on the distance from the last frame A Web data based broadcast content object identification and verification device.
2. The apparatus of claim 1, wherein the content knowledge information storage unit
Wherein the set of content knowledge information is stored as a set of content elements having the frame number of the broadcast content tagged in the knowledge network and the knowledge network as elements.
delete A web data based broadcast content object identification verification method performed by a web data based broadcast content object identification verification device,
Collecting and processing web data associated with broadcast content to generate content knowledge information in the form of tagging the broadcast content;
And verifying an identification result of an object included in the broadcast content using the content knowledge information,
The verifying step
Wherein the identification result is readjusted by calculating an appearance probability for the object based on the object identification candidate group and the knowledge network set associated with the frame including the identified object.
11. The method of claim 10, wherein the generating comprises:
Generating a query using object-related keywords for the broadcast content, requesting the web portal for search based on the generated query, and collecting web data;
Constructing a knowledge network comprising at least one of words and images extracted from the collected web data;
And tagging the knowledge network on a predetermined frame interval of the associated broadcast content. 12. The system of claim 11, wherein the knowledge network
Wherein each of the nodes comprises at least one of images and words extracted from the web data, and a connection edge between the nodes based on the importance and association of the information corresponding to the nodes. A method of verifying broadcast content objects.
12. The method of claim 11, wherein the step of tagging
Learning a matching model based on broadcast content frames;
Generating a learned matching model;
Calculating an image matching ranking by measuring similarity between feature vectors extracted from broadcast content frames or images;
And tagging the knowledge network including the image to the corresponding frame according to the image matching order.
14. The method of claim 13, wherein calculating the image matching ranking comprises:
Wherein the matching ranking is calculated once more by using a feature extraction technique using features different from the features used for the similarity measurement.
14. The method of claim 13, wherein the step of tagging
Selecting a last frame having the highest score and tagging the knowledge network with respect to a predetermined number of frames before and after the selected last frame, and the predetermined number of frames have a weight based on the distance from the last frame A method for verifying identification of broadcast content objects based on Web data.
delete

Images