KR102534270B1 - Apparatus and method for providing meta-data - Google Patents

Abstract

An apparatus for providing metadata according to an embodiment includes a frame extractor extracting a plurality of frames spaced apart in time from a video, a feature vector extractor extracting a feature vector from each of the plurality of frames, and object recognition. At least one reference vector is selected from among a plurality of reference vectors that are the basis for, and each of the selected at least one reference vector is compared with a feature vector extracted from any one of the plurality of frames to perform an object recognition process. and an object recognizing unit that obtains metadata about an object within any one frame and provides the metadata to the video.

Description

Apparatus and method for providing meta data {APPARATUS AND METHOD FOR PROVIDING META-DATA}

The present invention relates to an apparatus and method for providing meta data.

Recently, various business models (BMs) for media contents are being formed, and media services reflecting them are being introduced. These media services can be largely divided into services that introduce products to users before or after consuming media content, and services that introduce products while consuming media content. A representative example of the former is the recommendation service widely used by Netflix or Amazon, and the latter is e-commerce, which has recently attracted attention.

Let's take a look at the service scenario of the latter, e-commerce. While a viewer is watching a movie or a broadcast through a content player, a specific product may appear in a certain screen section. In this case, the viewer may receive a description of the corresponding product from the corresponding content player. In addition, a screen may be connected to a site where a corresponding product can be purchased according to a viewer's manipulation.

Looking at this process technically, the content player reproduces video/audio/meta data according to time information. At this time, if meta data about an object appears at a specific time, the content player can inform the viewer of this and provide an interface necessary for subsequent purchase.

Korean Patent Publication, No. 10-2011-0014403 (published on February 11, 2011)

For the e-commerce service described above, it is necessary to recognize an object in a video, obtain meta-data for the recognized object, and then insert or provide such meta-data to the video. In the conventional case, the above-described object recognition process, meta data acquisition process, and meta data insertion process are performed by humans. However, in the reality where countless videos are produced, there is a limit to performing the above-described process individually by a person.

Accordingly, an object to be solved by the present invention is to provide a technique for recognizing an object in a video, obtaining meta data for the recognized object, and inserting or providing such meta data into the video.

In addition, in performing the above-described technology, it is to provide a technology capable of reducing resources required for this, for example, time or cost.

However, the problem to be solved by the present invention is not limited to those mentioned above, and another problem to be solved that is not mentioned can be clearly understood by those skilled in the art from the description below. will be.

An apparatus for providing metadata according to an embodiment includes a frame extractor extracting a plurality of frames spaced apart in time from a video, a feature vector extractor extracting a feature vector from each of the plurality of frames, and object recognition. At least one reference vector is selected from among a plurality of reference vectors that are the basis for, and each of the selected at least one reference vector is compared with a feature vector extracted from any one of the plurality of frames to perform an object recognition process. and an object recognizing unit that obtains metadata about an object within any one frame and provides the metadata to the video. In addition, when the object recognizing unit recognizes that an object exists in a frame temporally preceding any one frame, the remaining reference vectors excluding the reference vector corresponding to the object recognized to exist among the plurality of reference vectors are converted into the at least one reference vector. It is selected as one reference vector.

A meta data providing method according to an embodiment is performed by a meta data providing apparatus, and includes extracting a first frame from a video, extracting a first feature vector from the first frame, and extracting a first feature vector from the first frame. Performing an object recognition process by comparing the extracted first feature vector with each of a plurality of reference vectors serving as object recognition standards; The step of extracting a second frame temporally later than the first frame from the video, the step of extracting a second feature vector from the second frame, and the step of extracting a second feature vector from the plurality of reference vectors. selecting at least one reference vector other than a reference vector corresponding to one object; performing an object recognition process by comparing the selected at least one reference vector with the second feature vector; and acquiring meta data for a second object recognized in the video and providing the meta data to the video.

According to an embodiment, if it is analyzed that all predefined objects exist in a frame, a process of recognizing objects in the frame (a process of extracting a feature vector and a process of comparing a reference vector and a feature vector) is not performed for a predetermined time. . In addition, if it is analyzed that some of the predefined objects exist in the frame, the object recognition process for the above-mentioned some objects is not performed for a predetermined time. Accordingly, time and resources consumed by the meta data providing device to provide meta data to a video can be reduced.

1 illustrates a screen on which a video is being played in the video reproducing apparatus.
2 illustrates a system to which an apparatus for providing meta data according to an embodiment is applied.
FIG. 3 shows the configuration of the meta data providing apparatus shown in FIG. 2 .
4 illustrates a process of analyzing or processing a video by a metadata providing apparatus according to an embodiment on a time axis.
5 illustrates a procedure of a meta data providing method according to an embodiment.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs. It is provided to completely inform the person who has the scope of the invention, and the present invention is only defined by the scope of the claims.

In describing the embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the embodiment of the present invention, which may vary according to the intention or custom of a user or operator. Therefore, the definition should be made based on the contents throughout this specification.

1 shows a screen on which a video is being played in the video reproducing apparatus 10 . Referring to FIG. 1 , various objects 20 and 30 are displayed in a video reproduced by the video reproducing apparatus 10 . Among them, meta-data is assigned to the object 30 in advance. Looking at the video, a circle-shaped identifier may be displayed at the top right for a predetermined period of time. This identifier indicates that meta data can be provided for any one of the objects 20 and 30 displayed in the current video. Examples of meta data may include the name, price, or sales location of the object 30 .

2 illustrates a system to which the apparatus 100 for providing meta data according to an embodiment is applied. However, since FIG. 2 is merely illustrative, it should not be construed that the meta data providing device 100 is limitedly applied only to the system shown in FIG. 2 .

Referring to FIG. 2 , the video providing server 50 stores a video. An example of such a video may be a drama, entertainment, show program or movie. In addition, the video providing server 50 provides a video to which meta data is not attached to the meta data providing device 100, and in response, after receiving a video to which meta data is attached from the meta data providing device 100, , which is provided to the video reproducing apparatus 10. Such an image providing server 50 may be a broadcasting equipment installed in a broadcasting station or the like.

The video reproducing device 10 is a device that reproduces a video, and at this time, metadata may also be displayed on the video. The video reproducing device 10 may be, for example, a TV, computer, or smart device, but is not limited thereto.

The meta data providing device 100 is a device that receives a video to which meta data is not attached from the video providing server 50, adds meta data to the video, and returns it to the video providing server 50. 2, the meta data providing device 100 is shown as being provided separately from the video providing server 50 and the video reproducing device 10, but unlike the shown, the meta data reproducing device 100 is a video providing server ( 50) or may be implemented within the video reproducing apparatus 10. Hereinafter, the meta data providing device 100 will be described in more detail.

FIG. 3 illustrates the configuration of the meta data providing apparatus 100 according to an embodiment, but what is shown in FIG. 3 is merely exemplary. This meta data providing device 100 may be implemented in a PC or server.

Referring to FIG. 3 , the metadata providing device 100 includes a frame extracting unit 130, a feature vector extracting unit 140, an object recognition unit 150, and a metadata providing unit 160. Accordingly, a communication unit 110 and a storage unit 120 may be further included.

Here, the communication unit 110 can be implemented as a port for receiving video or a wired/wireless communication module. In addition, the storage unit 120 can be implemented as a memory for storing data. In addition, the frame extraction unit 130, the feature vector extraction unit 140, the object recognition unit 150, and the meta data providing unit 160 include a memory for storing instructions programmed to perform functions to be described below, and such instructions. It can be implemented by a microprocessor running

First, the communication unit 110 will be reviewed. The meta data providing device 100 receives a video input from the video providing server 50 shown in FIG. 2 through the communication unit 110, and transmits the video to which meta data is given through the communication unit 110 to the video providing server 50. ) is provided to

The storage unit 120 stores data. The stored data may include, but is not limited to, a video to which meta data is not attached, a video to which meta data is attached, or a plurality of reference vectors (reference vectors will be described later).

The frame extractor 130 extracts a frame from a video. A frame means each screen (still screen) constituting a video. At this time, the frame extracted by the frame extractor 130 may be an I frame, but the fact that the extracted frame is an I frame is only exemplary and is not limited thereto.

The frame extractor 130 may extract frames at predetermined time intervals, that is, periodically. At this time, this time interval may be changed according to circumstances or conditions. Alternatively, the frame extractor 130 may analyze the video and extract a frame whenever a screen transition occurs beyond a predetermined criterion.

The feature vector extractor 140 and the object recognizer 150 analyze and recognize which object exists in the frame extracted by the frame extractor 130 . The feature vector extractor 140 and the object recognizer 150 may be learned by employing a convolutional neural network (CNN) previously learned by deep learning as a learning model. However, it is not limited thereto. Here, since the convolutional neural network itself is a known technology, a detailed description thereof will be omitted, but a detailed process of driving the feature vector extraction unit 140 and the object recognition unit 150 in one embodiment will be looked at in detail. do.

The feature vector extractor 140 extracts a feature vector from a frame. A feature vector means that an object existing in a frame is represented in the form of a vector. If a plurality of objects exist in a frame, a plurality of feature vectors may also be extracted from the corresponding frame.

The object recognition unit 150 performs an object recognition process by comparing the feature vector with each of a plurality of previously defined reference vectors. A reference vector means a vector that is a reference for object recognition. For example, a reference vector is defined when an object is a 'cat', a reference vector is defined when an object is a 'smartphone', and a reference vector is defined when an object is a 'smartphone'. A reference vector for the case of 'computer' may be defined. That is, the reference vector means a set of numbers by generalizing or abstracting features of an object in an image. A plurality of these reference vectors may be previously stored in the storage unit 120 in advance, and at this time, the plurality of reference vectors are vectors used for recognizing objects that are different from each other.

The object recognition unit 150 calls the plurality of reference vectors stored in the storage unit 120 in the object recognition process and uses them for object recognition. Specifically, the object recognizer 150 compares the feature vectors extracted by the feature vector extractor 140 with a plurality of reference vectors. If N reference vectors are provided (N is a natural number), the comparison process performed by the object recognizing unit 150 is repeated N times for one feature vector.

As a result of the comparison, if there is a reference vector that matches the corresponding feature vector among the N reference vectors, the corresponding feature vector is recognized as an object represented by the corresponding reference vector. However, if there is no matching reference vector, the object represented by the corresponding feature vector is treated as unrecognizable.

Here, the process of extracting the feature vector from the frame by the feature vector extractor 140 and the process of recognizing the object from the feature vector by the object recognition unit 150 may be performed in the following order, Hereinafter, description will be made on the premise that N reference vectors are provided (N is a natural number of 2 or more).

First, the feature vector extractor 140 extracts a first feature vector from the first frame extracted by the frame extractor 130 (hereinafter referred to as a first frame). At this time, according to an embodiment, a plurality of first feature vectors may be extracted.

Next, the object recognizing unit 150 compares the first feature vector with N reference vectors, respectively. If there are a plurality of first feature vectors as described above, each of the plurality of feature vectors is compared with each of the N reference vectors.

The comparison result may be as follows.

(First case) When all N objects exist in the first frame:

When analyzed in this way, the feature vector extractor 140 does not extract feature vectors from the frame for a predetermined time from the time of analysis in this way. Here, the predetermined time may be greater than the time interval between frames extracted by the frame extractor 130 that are temporally most adjacent, and this time is such that the video reproducing apparatus 10 of FIG. 2 is within any one frame. may have a time interval equal to the time indicating that it can provide meta data for an object that exists.

For example, if the predetermined time is 10 seconds, even if at least one frame is extracted by the frame extractor 130 for 10 seconds from the time when it is analyzed that all N objects exist in the first frame, the feature vector extractor (140) does not extract feature vectors from any frames thus extracted. Since the feature vector is not extracted, the object recognition unit 150 also does not perform the object recognition process.

That is, according to an embodiment, if it is analyzed that all predefined objects exist in a frame, a process of recognizing objects in the frame (a process of extracting a feature vector and a process of comparing a reference vector and a feature vector) is performed for a predetermined time. It doesn't work. Accordingly, time and resources consumed by the meta data providing apparatus 100 to provide meta data to a video can be reduced.

(Second case) When some of the N objects are all present in the first frame:

In this case, the feature vector extractor 140 extracts the second feature vector from the second frame temporally behind the first frame. At this time, according to an embodiment, a plurality of second feature vectors may be extracted.

Next, the object recognizing unit 150 selects at least one reference vector to be used for comparison among the N reference vectors. The selection criteria are as follows. Among the N reference vectors, reference vectors other than the reference vector corresponding to the object recognized to exist in the first frame are selected as at least one reference vector to be used for the above comparison. The number of selected reference vectors is less than N.

The object recognizing unit 150 compares each of the at least one reference vector selected in this way with the second feature vector. If there are a plurality of second feature vectors as described above, each of the plurality of feature vectors is compared with each of at least one reference vector. Here, the object recognition unit 150 compares each of the at least one reference vector selected as described above with the second feature vector or another feature vector without comparing each of the N reference vectors with the second feature vector. It is valid only for a time of , and when a predetermined time elapses, the object recognizing unit 150 compares each of all N reference vectors with the target feature vector.

For example, assuming that one of the N reference vectors is a reference vector for 'cat', and 'cat' is recognized in the first frame, the object recognition unit 150 corresponds to 'cat' among the N reference vectors. The remaining reference vectors, that is, N-1 reference vectors, except for the reference vectors to be used for comparison are selected as reference vectors to be used for comparison. In addition, the object recognizing unit 150 compares each of these N-1 reference vectors with the second feature vector.

That is, according to an embodiment, if it is analyzed that some of the predefined objects exist in the frame, the object recognition process for the above-mentioned some objects is not performed for a predetermined time. Accordingly, time and resources consumed by the meta data providing device 100 to provide meta data to a video can be reduced.

(Third case) When none of the N objects exist in the first frame:

In this case, the feature vector extractor 140 sequentially extracts feature vectors from each frame starting from the second frame. In addition, the object recognizing unit 150 sequentially compares the feature vectors for each frame with the N reference vectors, starting from the feature vector for the second frame, which corresponds to the first case or the second case. This continues until at least one object is recognized as existing in the video.

Next, the meta data providing unit 160 provides meta data to the video. Meta data may be previously stored in the storage unit 120 or obtained from an external server (not shown) through the communication unit 110 . Meta data provided in this way may be added to a video along with time information.

4 is a diagram illustrating the above-described first to third cases as examples. In FIG. 4, there are a total of 5 predefined reference vectors, and it is assumed that they are vectors used to recognize the following objects (a, b, c, d, e), respectively.

In FIG. 4, the horizontal axis is time, and the upper horizontal axis conceptually shows the time required for object recognition. In addition, on the horizontal axis below, an arrow indicates the moment when the frame extraction unit 130 extracts a frame from the video.

Referring to FIG. 4 , an analysis process, that is, a process of extracting a feature vector is performed from a first frame extracted from a video, and then a process of analyzing whether 5 objects exist in the corresponding frame is performed. To this end, a process of selecting a reference vector is preceded. In this case, since there is no previously recognized object, all five reference vectors are selected as reference vectors to be used for comparison.

As a result of the analysis, among the objects a to e, a is recognized as existing in the corresponding frame. This corresponds to the second case described above. Therefore, the feature vector extractor 140 extracts the feature vector from the next frame, and the object recognizer 150 analyzes whether the remaining objects b to e excluding object a among the five objects exist in the corresponding frame (for this, A process of selecting reference vectors corresponding to objects b to e among the five objects is performed). Here, since the process of analyzing whether the object a exists in the corresponding frame is omitted, time and resources required for the object recognition process can be reduced accordingly.

Next, as a result of the analysis, it is recognized that all objects b to e are present in the corresponding frame. Since object a already exists in the previous frame, this case corresponds to the first case described above. Accordingly, the feature vector extraction unit 140 does not extract the feature vector from the next frame, and the object recognition unit 150 does not perform an object recognition process either. Accordingly, the time and resources required for the object recognition process can be reduced accordingly.

On the other hand, the process of selecting a reference vector by the object recognizing unit 150 excluding the recognized object from among the five objects is performed only for a predetermined time, and after this predetermined time elapses, the reference vectors for all objects are used for comparison. It is selected as a reference vector. In FIG. 4, there is an overlapping time between the time when object a is not recognized and the time when objects b to e are not recognized. Since it is the first case), the feature vector is also not extracted. However, since the time to exclude object a has passed first, as shown in the figure, after a predetermined time for object a has elapsed, the feature vector extractor 140 extracts a feature vector from a predetermined frame, and the object recognizer In step 150, a reference vector for object a is selected and an object recognition process is performed based thereon. After that, the object recognition process is performed by selecting reference vectors for the five objects again.

5 conceptually illustrates a procedure of a meta data providing method according to an embodiment. This meta data providing method may be performed by the meta data providing apparatus 100 described above.

On the other hand, since the procedure shown in FIG. 5 is only exemplary, the spirit of the present invention is not limited to that shown in FIG. 5, and the procedure may be performed in a different order from that shown in At least one of the procedures shown in FIG. 5 may not be performed, and a procedure not shown in FIG. 5 may be additionally performed.

First, the frame extraction unit 130 extracts a frame from a video (S10). In this step S10, the frame extraction unit 130 may extract frames at predetermined time intervals, that is, periodically. At this time, this time interval may be changed according to circumstances or conditions. Alternatively, the frame extractor 130 may analyze the video and extract a frame whenever a large screen transition occurs.

If it is recognized that all previously defined objects exist in the previous frame (S20), the feature vector extraction unit 140 does not extract any feature vectors from the frame extracted in step S10, and thus the object recognition unit 150 ) Also, the object recognition process is not performed (S30). In addition, the process in which no operation is performed in step S30 continues for a predetermined time (S31), and returns to step S10 when the predetermined time elapses.

However, if it is recognized that at least one object exists in the previous frame (S10), the feature vector extractor 140 extracts a feature vector from the frame extracted by the frame extractor 130 (S40). If a plurality of objects exist in a frame, a plurality of feature vectors may also be extracted from the corresponding frame.

Next, the object recognizing unit 150 selects at least one reference vector to be used for comparison among a plurality of reference vectors (S41). The selection criteria are as follows. Among the plurality of reference vectors, a reference vector other than a reference vector corresponding to an object recognized to exist in a previous frame in time is selected as at least one reference vector to be used for the above comparison.

The object recognition unit 150 performs object recognition by comparing each of the at least one reference vector selected as described above with a feature vector (S42). If there are a plurality of feature vectors as described above, each of the plurality of feature vectors is compared with each of at least one reference vector. Here, the object recognizing unit 150 does not compare all of the plurality of reference vectors with feature vectors, but compares each of the at least one reference vector selected as described above with feature vectors or other feature vectors only for a predetermined time. If it is valid and a predetermined time has elapsed, the object recognizing unit 150 again compares each of the plurality of reference vectors with the target feature vector.

That is, according to an embodiment, if it is analyzed that some of the predefined objects exist in the frame, the object recognition process for the above-mentioned some objects is not performed for a predetermined time. Accordingly, time and resources consumed by the meta data providing device 100 to provide meta data to a video can be reduced.

Meanwhile, each step included in the meta data providing method according to the above-described embodiment may be implemented in a computer readable recording medium recording a computer program programmed to perform these steps.

Meanwhile, each step included in the meta data providing method according to the above-described embodiment may be implemented in a computer readable recording medium recording a computer program programmed to perform these steps.

The above description is merely an example of the technical idea of the present invention, and various modifications and variations can be made to those skilled in the art without departing from the essential qualities of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

According to an embodiment, time and resources consumed by a meta data providing device to provide meta data to a video can be reduced.

100: metadata providing device

Claims (8)

A frame extraction unit for extracting a plurality of frames spaced apart in time from a video;
a feature vector extractor extracting a feature vector from each of the plurality of frames;
Object recognition process by selecting at least one reference vector from among a plurality of reference vectors serving as a basis for object recognition, and comparing each of the selected at least one reference vector with a feature vector extracted from any one of the plurality of frames. An object recognition unit that performs;
A metadata providing unit that obtains metadata about an object in any one of the frames and provides it to the video;
The object recognition unit,
If it is recognized that an object exists in a frame temporally preceding any one of the frames, the remaining reference vectors excluding the reference vector corresponding to the object recognized to exist among the plurality of reference vectors are used as the at least one reference vector. selected
Metadata Provider. According to claim 1,
The feature vector extractor and the object recognizer,
It is implemented by adopting a convolution neural network model among deep learning methods.
Metadata Provider. According to claim 1,
The object recognition unit,
During a predetermined time, an object recognition process is performed by comparing at least one reference vector selected from among the plurality of reference vectors with a feature vector extracted from any one frame;
After the predetermined time has elapsed, each of the plurality of reference vectors is compared with a feature vector extracted from a frame later than any one frame in time to perform an object recognition process.
Metadata Provider. According to claim 3,
The predetermined time is
having the same time interval as the time indicating that the video reproducing device that reproduces the video can provide meta data for an object within one of the frames
Metadata Provider. According to claim 1,
The feature vector extraction unit,
If it is recognized that all objects corresponding to each of the plurality of reference vectors exist in a frame that chronologically precedes any one frame, the feature vector is not extracted from any one frame,
The object recognition unit,
The object recognition process for any one of the above frames is not performed.
Metadata Provider. As a meta data providing method performed by a meta data providing device,
extracting a first frame from a video;
extracting a first feature vector from the first frame;
performing an object recognition process by comparing the first feature vector extracted from the first frame with each of a plurality of reference vectors serving as object recognition standards;
obtaining meta data for a first object recognized in the first frame and providing it to the video;
extracting a second frame temporally later than the first frame from the video;
extracting a second feature vector from the second frame;
selecting at least one reference vector excluding the reference vector corresponding to the first object from among the plurality of reference vectors;
performing an object recognition process by comparing the selected at least one reference vector with the second feature vector;
Acquiring meta data for a second object recognized in the second frame and providing it to the video
How to provide metadata. extracting a first frame from a video;
extracting a first feature vector from the first frame;
performing an object recognition process by comparing the first feature vector extracted from the first frame with each of a plurality of reference vectors serving as object recognition standards;
obtaining meta data for a first object recognized in the first frame and providing it to the video;
extracting a second frame temporally later than the first frame from the video;
extracting a second feature vector from the second frame;
selecting at least one reference vector excluding the reference vector corresponding to the first object from among the plurality of reference vectors;
performing an object recognition process by comparing the selected at least one reference vector with the second feature vector;
Programmed to perform the step of obtaining meta data for a second object recognized in the second frame and providing it to the moving image.
A computer program stored on a computer-readable recording medium. extracting a first frame from a video;
extracting a first feature vector from the first frame;
performing an object recognition process by comparing the first feature vector extracted from the first frame with each of a plurality of reference vectors serving as object recognition standards;
obtaining meta data for a first object recognized in the first frame and providing it to the video;
extracting a second frame temporally later than the first frame from the video;
extracting a second feature vector from the second frame;
selecting at least one reference vector excluding the reference vector corresponding to the first object from among the plurality of reference vectors;
performing an object recognition process by comparing the selected at least one reference vector with the second feature vector;
Programmed to perform the step of obtaining meta data for a second object recognized in the second frame and providing it to the moving image.
A computer-readable recording medium that stores a computer program.

Images