KR101485820B1 - Intelligent System for Generating Metadata for Video - Google Patents

Abstract

An intelligent system for generating video meta information comprises: an object information storage unit which stores a standard (object information, for example, an image of the object) capable of identifying and extracting each of objects by comparing objects (person, thing, background and the like) with each other appearing in a video; a cut generation unit which sequentially compares each of frame images of the video and divides the video into cuts in which an image of the frame image is continuous depending a similarity between a previous frame image and and the next frame image; a cut information generation unit which extracts information on whether the object appears in each cut by comparing each of cuts generated by the cut generation unit with the object information stored in the object information storage unit and stores the extracted information as cut information; a scene generation unit which extracts a scene by combining similar cuts by using at least one reference among the similarity between objects, similarity of colors, and a maximum of a predetermined section length by comparing cuts generated from the cut generation unit with the previous and next cuts and divides the video into a set of scenes; and a meta information storage unit which stores information on the existence of the object appearing in each of the scenes divided by the scene generation unit as meta information by using the cut information.

Description

{Intelligent System for Generating Metadata for Video}

The present invention relates to an intelligent video meta information generation system, and more particularly, to an intelligent video meta information generation system that automatically divides a moving picture into a plurality of scenes as a meaning unit of a scene and automatically generates and provides meta information for each scene .

It is possible to provide various services using moving pictures by extracting the objects appearing in the moving pictures and generating information about the appearance times of the objects in order to provide various services for the moving pictures.

In this way, a method of extracting an object appearing in a moving image and generating appearance information of the object is conventionally performed by a person manually checking an image of the moving image. This is a very inefficient, time-consuming and costly method.

Korean Patent Laid-Open Publication No. 10-2011-0010083 discloses a method of generating moving picture markup data. In the method of generating the moving image markup data, the moving image is divided into sections without any specific criteria, and the time of appearance and the time of disappearance of the object are generated as object information within each section. That is, although the time of appearance and the time of disappearance of an object are generated as object information, a method of automatically dividing a moving picture according to a section is not provided.

In a method of providing object information about a moving picture and meta information about the object, it is important to generate the object information and the meta information by dividing the moving picture into appropriate intervals. It is possible to improve the quality of the object information providing service for the moving picture by providing the object information by automatically and effectively dividing the segmentation of the moving picture.

An object of the present invention is to provide an intelligent video meta information generation system that automatically divides a moving picture into meaningful sections and provides object information or meta information thereon.

According to an aspect of the present invention, there is provided an intelligent video meta information generation system for generating a meta information of a video object, For example, an image of the object) is stored; A cut generation unit for sequentially comparing each frame image of the moving image to divide the moving image into continuous sections (cuts) of the frame image according to similarity between the previous and next frame images; A cut information generation unit for comparing the cuts generated by the cut generation unit with the object information stored in the object information storage unit to extract whether the objects appear in each cut and store the extracted cut information; Extracting scenes by combining similar cuts using at least one of the similarities of objects appearing by comparing the cuts generated by the cut generation unit with the front and rear cuts, whether there is similarity of colors, the similarity of colors and the upper limit of a predetermined section length, A scene generation unit that divides a moving picture into a set of scenes; And a meta information storage unit for storing, as meta information, whether or not the objects appearing in the respective scenes are distinguished by the scene creation unit using the cut information.

The intelligent moving image meta information generation system of the present invention divides a moving image into a plurality of segments in a unit of a scene interval instead of dividing the moving image at a predetermined time interval or a scene change unit to provide information about an object appearing in each segment, There is an effect that the level of the service providing the information can be improved.

1 is a block diagram of an intelligent motion meta information generation system according to the present invention.

Hereinafter, an intelligent moving image meta information generating system according to the present invention will be described in detail with reference to the drawings.

1, the intelligent moving image meta information generation system of the present embodiment includes an object information storage unit 110, a cut generation unit 130, a cut information generation unit 210, a scene generation unit 230, (250).

The object information storage unit 110 stores object information corresponding to the objects appearing in the moving image to be generated in the meta information. An image of a person appearing in a moving picture, a voice, and the like can be object information. In addition to characters, objects such as objects appearing in videos, objects such as small objects, buildings, and plazas can be objects. Such object information is a criterion for extracting information on an object appearing later in a moving image. That is, the object information stored in the object information storage unit 110 is compared with an image or voice appearing in the moving image to determine whether or not the corresponding object appears in the moving image. In addition, the object information may include text information describing the object to the viewer watching the moving image, in addition to the image and voice of the object. In other words, human information about the actor who plays the character can be additionally included as the object information.

The cut generation unit 130 sequentially compares frame images of a moving image and divides the moving image into a plurality of cuts based on a point at which a moving image is switched. In the present invention, the cut of a moving image is generally the same as a cut in a movie or broadcast. In the moving picture, the section of the frame image continuously continuous without interruption becomes a cut. That is, the section of the image shot at one time by the camera is cut. In the case of the present invention, the cut generation unit 130 sequentially compares the frame images constituting the moving image and classifies the frame images into the same cut or the other cut according to the similarity of the adjacent front and rear frame images.

In the present embodiment, the cut generation unit 130 separates cuts based on similarity of hue values of respective pixels of the front and rear frame images. Various methods can be used as a method of comparing the degree of similarity of color values between frame images. For example, it is possible to represent hue values for each pixel in each frame image as histogram distributions, compare the histogram distributions of the adjacent front and rear frame images, and separate the cuts according to the similarity. The histogram distribution is generated by dividing the color value into a plurality of intervals according to the numerical range and calculating the number of pixels having the color value of the corresponding interval as a frequency.

There are also various ways to compare the color histogram distributions. For example, if a threshold value for determining similarity of a color histogram distribution is set in advance, a difference in histogram distribution (frequency) of each color (per color value interval) is calculated, and if the sum of the differences is greater than a threshold value, Are not similar to each other and can separate the cut at that point.

By dividing the cuts of a movie in this way, the videos can be separated by cuts at approximately the same level, although not exactly the same as the cuts in the dictionary meaning. In particular, there is an advantage that the cuts can be separated by extracting the cuts of the video automatically without going through the work of the person.

If the moving image is separated in units of cuts in the cut generation unit 130, the cut information generation unit 210 extracts whether the objects stored in the object information storage unit 110 appear for each cut, and stores the extracted cut information do. Depending on the case, the cut raw ward 130 may store the time at which the object appears and the time at which the object appears at each cut as cut information.

The cut information generation unit 210 compares a frame image of each cut with an image of each object stored in the object information storage unit 110 to extract whether or not an object appears in each cut. The cut information generation unit 210 stores the identification number of the cut and the identification number of the object extracted from the cut as cut information. For example, it stores the fact that the object 5 (movie actor breadfeet) appears in the cut 3, in the form of cut information. As described above, the cut information generating unit 210 may store the appearance time and the extinction time of each object as cut information.

If the object is not the image information but the voice information, the voice of the object information is compared with the voice of the corresponding cut, and whether or not the voice appears is stored as the cut information. For example, we store the fact that the number 6 object (movie actress Guinness Felto's voice) appears in the third cut in the form of cut information. In this case, the image of the corresponding object is not displayed on the screen, but the appearance of the object in the form of voice is recognized as a form of cut information and stored. Not only the human voice but also the background music are recognized as objects and are stored as cut information by the cut information generation unit 210. [

The scene generation unit 230 receives the cut information from the cut information generation unit 210 and adds the cuts to generate a scene. In the present invention, a scene generally matches a scene in a dictionary meaning. Similar cuts among adjacent front and rear cuts are connected to form a scene. Various methods can be used as criteria for combining different cuts into one scene. For example, if the same object appears consecutively in adjacent front and rear cuts, the same scene can be judged. Also, if it is determined that the colors of adjacent cuts are similar in the same manner as the above-described similarity judging method of colors, the same scene can be judged. It is also possible to divide the scene at a time when the upper limit of the section length of the scene is preset and the cuts are accumulated and the accumulated cuts are smaller than the upper limit of the section length to combine them into one scene and exceed the upper limit of the section length . In some cases, if the two objects appear consecutively in succession, the cuts can be combined to belong to the same scene. In this case, the characters are alternately appearing on each cut, and the scenes where the characters are alternately appearing and conversing are bundled into one scene. In some cases, even if the appearance of an object is repeated, these cuts can be combined into one scene.

It is also possible to create scenes by using two or more criteria or methods of composing the scenes by combining the cuts described above. If the sum of the points exceeds the threshold point by assigning points to each of the above criteria, the cuts belonging to the same scene It is also possible to judge. In some cases, it is also possible to apply two or more criteria or methods for constructing a scene as described above, input the result into a predetermined weight function, and compare the resultant value with a threshold value to identify the scene.

As described above, the moving image is divided into a set of scenes by the scene generating unit 230. [ The moving image is divided into scenes corresponding to individual meaningful intervals by the scene generating unit 230 as described above. That is, it is divided into a set of cuts having a common meaning such as a duel scene, a recall scene, a kiss scene, etc., in which a plurality of cuts are combined and completed. Even if the scene is switched in the same scene and the cut is divided, the cuts can be divided into one, two, or three, using the common background, the common characters, the background music, Scene.

With this method, it is possible to automatically divide a moving picture into a plurality of scenes.

 The meta information storage unit 250 stores information on the scene classified by the scene generation unit 230 and the cut information generated by the cut information generation unit 210 to determine whether each object appears in each scene, And stores it as information. The appearance time and the disappearance time of each object may be stored as meta information. The meta information storage unit 250 may additionally store as a meta information a total screen contrast ratio of an object appearing in each scene in each scene, and may further store the degree of time during which a plurality of objects are simultaneously exposed on the screen as meta information . ≪ / RTI > The meta information storage unit 250 may store descriptive information about each object or object information itself as meta information. The meta information storage unit may further store the main colors exposed on the screen of each scene as meta information.

It is possible to divide a movie into a plurality of scenes in the same manner as above, and to automatically generate meta information for each scene and utilize it for various purposes. It is also possible to provide the user with meta information for each section of the moving picture and to sell the audience for each section. Also, it is possible to input advertisement information for each section, and to provide a preview of the moving picture for each section. The intelligent moving image meta information generating system of the present invention can provide a better quality of service to the user by separating the moving image into meaning units instead of dividing the moving image at a predetermined time interval. In particular, for each scene separated by semantic units, the description information of the object, the degree of appearance of the object, the main color, the ratio of the simultaneous exposure time of the objects, the co- The background information of the background may be provided to the user so that the user can selectively view the scene or the video related provider can select a specific scene and use it effectively for the video related business There are advantages.

Hereinafter, the case will be described in which the scene is divided into scene units through the above-described process, meta information is provided for each scene, and the advertisement information is provided to the advertiser and used for receiving announcement bids.

The meta information providing unit 310 provides the meta information stored in the meta information storage unit 250 so that the advertiser can view meta information corresponding to each scene.

The advertiser can view the meta information of the videos classified by the scene and decide whether to advertise only the scenes that are necessary or desired by the advertiser.

The bid information receiving unit 330 receives the identification number of the scene selected by the advertiser to display the advertisement and the bid amount of the advertisement cost for the selected scene.

The related information receiving unit 330 receives the related information, which is the content of the advertisement to be displayed in the scene selected by the advertiser.

The association information setting unit 410 determines the winning bid amount by selecting the highest bid amount corresponding to the identification number of each scene received by the bid information receiving unit 330 and sets association information of the successful bidder in the corresponding scene. There are various methods for setting the association information. Typically, it is possible to input association information in the form of subtitles on the screen of the corresponding scene. It is also possible to input the association information by a method of image input rather than a caption, or to set association information by a method of inputting a voice.

Although the preferred embodiment of the present invention has been described as an example, the scope of the present invention is not limited to the above described and illustrated forms.

For example, the case where the video classified into the scenes and generated the meta information is used for the advertisement bidding has been described as an example, but the meta information providing unit 310, the bid information receiving unit 330, the related information receiving unit 350, It is also possible to construct an intelligent video meta information generation system that does not include the association information setting unit 410 and the like. In this case, the meta information and the scene information are extracted, and the user uses the meta information and the scene information to use the intelligent video meta information generation system of the present invention for various purposes other than the purpose of advertisement bidding .

110: object information storage unit 130:
210: Cut information generating unit 230:
250: meta information storage unit 310: meta information providing unit
330: auction information receiving unit 350: related information receiving unit
410: association information setting unit

Claims (11)

An object information storage unit for storing a reference (object information, for example, the image of the object) that can identify and extract each object compared with each object (person, object, background, etc.) appearing in the moving image;
A cut generation unit for sequentially comparing each frame image of the moving image to divide the moving image into continuous sections (cuts) of the frame image according to similarity between the previous and next frame images;
A cut information generation unit for comparing the cuts generated by the cut generation unit with the object information stored in the object information storage unit to extract whether the objects appear in each cut and store the extracted cut information;
Extracting scenes by combining similar cuts using at least one of the similarities of objects appearing by comparing the cuts generated by the cut generation unit with the front and rear cuts, whether there is similarity of colors, the similarity of colors and the upper limit of a predetermined section length, A scene generation unit that divides a moving picture into a set of scenes; And
And a meta information storage unit for storing the presence or absence of the object appearing in each scene as meta information using the cut information for each scene classified by the scene creation unit, system. The method according to claim 1,
Wherein the object information stored in the object information storage unit further includes a voice of a person and a background music. 3. The method according to claim 1 or 2,
Wherein the cut generation unit separates cuts based on a degree of similarity of hue values of respective pixels of the front and rear frame images. The method of claim 3,
Wherein the cut generation unit generates a histogram of pixel-by-pixel hue values for each frame image, compares the histogram with a color histogram distribution of the frame images before and after, and separates the cuts according to the similarity. . 5. The method of claim 4,
Wherein the cut generation unit calculates a difference of a color histogram distribution of adjacent front and rear frame images and separates the color histogram distribution into cuts if the sum is greater than or equal to a preset threshold value. 3. The method according to claim 1 or 2,
Wherein the scene generation unit further generates a scene by combining neighboring cuts by further considering a criterion of whether or not the appearance of the object is repeated. The method according to claim 1,
Wherein the meta information storage unit further stores at least one of the exposure ratio of the object in the respective scenes as a whole screen and the degree of time of simultaneous exposure on the screen among the plurality of objects as meta information for each scene, Intelligent video meta information generation system. The method according to claim 1,
Wherein the meta information storage unit further stores main colors exposed on the screen of each scene as meta information. The method according to any one of claims 1, 7, and 8,
Wherein the object information further includes text information describing the object to a viewer of the moving image,
Wherein the meta information further includes the object information. 10. The method of claim 9,
A meta information providing unit for providing the meta information corresponding to each scene of the moving picture to be viewed by the advertiser;
A bidding information receiving unit for receiving an identification number of a scene selected by an advertiser to display an advertisement and an advertisement bidding amount for the selected scene; And
And an association information receiver for receiving association information, which is the contents of an advertisement to be displayed in a scene selected by the advertiser for bid. 11. The method of claim 10,
And an association information setting unit for determining whether to win the bid by selecting the highest bid amount corresponding to the identification number of each scene received by the bid information receiving unit and setting association information of the successful bidder in the corresponding scene of the moving image Wherein the intelligent video meta information generation system comprises:

Images