KR101313285B1 - Method and Device for Authoring Information File of Hyper Video and Computer-readable Recording Medium for the same - Google Patents

Abstract

The present invention relates to a device and method for producing a hyper video information file, and a recording medium thereof,
In a hyper video in which additional information of a person or an object included in a video file is produced as an information file and linked by a hyperlink, a method of producing a hyper video information file executed in an information file producing apparatus, the method comprising: (a) from an input video file Extracting frames; (b) detecting a shot boundary from the extracted frames, extracting a representative frame for each detected shot, and storing information about the shot and the representative frame in a video layer DB; (c) grouping similar shots of each shot into one scene, generating video hierarchy information in which the frames, shots, and scenes are structured, and storing the same in a video hierarchy DB; (d) selecting and inputting an object region of interest in an arbitrary frame from a user, detecting the object region in the representative frames, and storing the object region in an object position DB; (e) tracking the position in the adjacent frame from the representative positions of the objects detected in the respective representative frames and storing the additional position in the object position DB; (f) storing the object information input from the user in the object information DB; And (g) generating a hyper video information file by mapping the object location DB and object information DB contents.

Description

Apparatus and method for producing hyper video information file, recording medium {Method and Device for Authoring Information File of Hyper Video and Computer-readable Recording Medium for the same}

The present invention relates to an apparatus and method for producing a hyper video information file, and a recording medium thereof, wherein an information file of a hyper video in which additional information of a person or an object included in video data is linked by a hyperlink, is divided into a scene and a face. The present invention relates to a hyper video information file production apparatus and a method for producing a hyper video information file efficiently by automatically detecting and grouping, detecting an object, and tracking an object, and a recording medium thereof.

Hyper video is a video including a hyper link, that is, a hyperlink is connected to a face area or a desired object area of a person appearing in the video, and through this, additional information of the person or objects is provided. Speak the edited video to view it.

Such hyper video is an extension of the concept of HyperText, which is widely used on the Internet home page, to the video field. Unlike normal video, hyper video allows the user to easily select and view additional information on the people and objects that are hyperlinked during playback.

Recently, the application range of hyper video has been expanded considerably, for example, an interactive service such as an interactive advertisement in an IPTV using a product in PLacement (PPL) concept.

Examples of the additional information linked to the video through a hyperlink include an additional image, a text, a voice, a video, a homepage address (URL), and the like related to the person included in the video.

This hyper video is a hyper video that stores the location of each person's (eg, face) or objects (eg, glasses, earrings, clothing, bags, etc.) contained in the video data on each video frame and additional information associated with it. Produced through an information file production device, and reproduced so that the video data and the information file to be hyperlinked in the hyper video playback device.

An important issue in the creation of hyper-video information files is the definition and verification of the desired person or object area to which the hyperlinks should be placed.

In this regard, conventional hyper video information file authoring apparatus, such as VideoClix for MacOS, takes the method of manually specifying and editing the definition of the desired object area and the change in every frame of video, so that the hyper video information can be edited. There was a limit to the amount of time and effort required to produce a file.

In consideration of this point, for example, Korean Patent Laid-Open Publication No. 10-2009-0044221 (published 2009.05.07) has proposed a method for providing an interactive advertisement information file authoring service.

However, the method for providing the interactive advertisement information file authoring service proceeds by manually selecting the advertisement object region included in the extracted frame and extracting the next frame sequentially according to a preset frame extraction frequency. However, there was a problem that the basic work was not limited to the manual work and the working time for the entire frame was not small.

In addition, the method for providing an interactive advertisement information file authoring service includes a scene change determination routine, and measures the degree of change of the extracted frame and the previous frame, and determines that the scene is changed when it is larger than a threshold value. After the video is taken in a multi-layered video structure such as frame-shot-scene according to the structure of the video shot through the camera, Without this division, the frame conversion degree is simply determined as to whether or not to change scenes according to the measurement, and thus the practical effect of distinguishing scene transitions is insignificant.

In addition, since the preset fixed threshold value is used to classify the scene change as described above, there is a problem that the precision of the scene change determination is lowered without being able to cope with various screen changes.

In addition, for the purpose of a technology for creating an interactive advertisement information file, it is configured to only detect the position of the object object to be advertised, there was a limit that does not provide an effective means for determining the position of the face of the person.

The present invention for solving the problems according to the prior art, in particular, while producing an information file of the hyper video in which the additional information of the person or objects included in the video data is connected by a hyperlink, scene division, face detection and grouping, An object of the present invention is to provide an apparatus and method for producing a hyper video information file, which is configured to automatically perform object detection, object tracking, and the like to efficiently produce a hyper video information file, and a recording medium thereof.

One embodiment of the present invention for achieving the above object is, in the hyper-video is created in the information file and additional information of the person or object included in the video file is linked to the hyperlink, which is executed in the information file production apparatus A method of producing a hyper video information file, comprising: (a) extracting frames from an input video file; (b) detecting a shot boundary from the extracted frames, extracting a representative frame for each detected shot, and storing information about the shot and the representative frame in a video layer DB; (c) grouping similar shots of each shot into one scene, generating video hierarchy information in which the frames, shots, and scenes are structured, and storing the same in a video hierarchy DB; (d) selecting and inputting an object region of interest in an arbitrary frame from a user, detecting the object region in the representative frames, and storing the object region in an object position DB; (e) tracking the position in the adjacent frame from the representative positions of the objects detected in the respective representative frames and storing the additional position in the object position DB; (f) storing the object information input from the user in the object information DB; And (g) generating a hyper video information file by mapping the object location DB and object information DB contents.

In one embodiment according to another aspect of the present invention, hyper video information that is executed in an information file production apparatus in a hyper video in which additional information of a person or an object included in a video file is produced as an information file and linked by a hyperlink A file production method, comprising: detecting a shot boundary from frames extracted from an input video file, extracting a representative frame for each detected shot, and storing information about the shot and the representative frame in a video layer DB; Grouping similar shots among the shots into one scene, generating video hierarchy information in which the frames, shots, and scenes are structured, and storing the same in a video hierarchy DB; Detects a face area included in each representative frame, performs a face grouping with similar faces among the detected face areas, stores the object in the object position DB, detects an object area selected by a user in the representative frame, and Tracking the position of the face grouped with the object in a frame and storing the position in the object position DB; Storing the object information input from the user in the object information DB, and mapping the object location DB and the object information DB contents to generate a hyper video information file.

According to another aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a program for executing each step of the method for producing a hyper video information file.

An embodiment according to another aspect of the present invention is a hyper video information file production apparatus for configuring a hyper video in which additional information of a person or an object included in a video file is linked by a hyperlink, and includes a frame from an input video file. Extracting frames, detecting a shot boundary from the extracted frames, extracting a representative frame for each detected shot, storing information about the shot and the representative frame in a video layer DB, and similar shots among the shots. A scene grouping module configured to group the video data into a single scene, and to generate the video hierarchy information in which the frames, shots, and scenes are structured, and store the video hierarchy information in a video layer DB; An object detection module configured to detect the object area in the representative frames and store the object area in an object location DB; An object tracking module for tracking a location in an adjacent frame from the representative locations of the objects detected in the representative frames and storing the location in an object location DB; A user UI module for selecting and receiving an object region of interest in an arbitrary frame from a user and storing the object information input from the user in an object information DB; And a control module configured to at least perform a function of generating a hyper video information file by mapping the object location DB and object information DB contents.

Preferably, a face region detection module for detecting a face region included in each of the detected representative frames; and a face grouping module for performing face grouping with similar faces among the detected face regions and storing them in an object position DB. It may be configured to include more.

An embodiment according to another aspect of the present invention is a hyper video information file production apparatus for constructing a hyper video in which additional information of a person or an object included in a video file is linked by a hyperlink, and extracted from an input video file. Detecting a shot boundary from the captured frames, extracting a representative frame for each detected shot, and storing information about the shot and the representative frame in a video layer DB; Grouping similar shots among the shots into one scene, and generating and storing the video hierarchy information in which the frames, shots, and scenes are structured and stored in a video layer DB; Detects a face area included in each representative frame, performs a face grouping with similar faces among the detected face areas, stores the object in the object position DB, detects an object area selected by a user in the representative frame, and Tracking the position of the face grouped with the object in a frame and storing the position in the object position DB; And storing the object information input from the user in the object information DB, and mapping the object location DB and the object information DB contents to generate a hyper video information file.

As such, the present invention has an advantage in that a hyper video information file can be easily produced by automatically performing scene division, face detection and grouping, object detection, and object tracking.

In particular, the present invention generates a hierarchical structure of a video by detecting shot boundaries in a frame and grouping similar shots into scenes, thereby providing access to individual shots when editing a long video having a large number of frames or shots. It is easy to do, and consequently offers the advantage of increasing the efficiency of hyper video production.

In addition, the present invention automatically detects an object position in representative frames using extraction or object detection, and then, within the same shot, subsequently adjacent from the position in all representative frames for the entire detected object or a particular object. Since the position in the frames is tracked by a background processing method, an edit operation of a user can be performed in parallel with the tracking of an object that takes a long time with respect to a frame in which scene grouping has already been performed.

In addition, the present invention enables to actively set the threshold value used when detecting the shot boundary, so that the shot boundary can be detected stably and with high precision regardless of the contents of the video.

In addition, the present invention, as well as the object object to be advertised, automatically detects the position of the person face to provide a variety of person-related information through the face detection and grouping, the advantage of further expanding the utilization of the hyper video have.

In addition, since the present invention generates the hyper video information file as meta data in an XML format, it also provides an advantage that different hyper video editing results can be played in one player when the hyper video is played.

1 is a block diagram of a hyper video production apparatus according to an embodiment of the present invention;
2 is a conceptual diagram of a video hierarchy structure of a hyper video making apparatus according to an embodiment of the present invention;
3 is an overall flowchart of a method for producing a hyper video according to an embodiment of the present invention;
4 is a flowchart of a scene grouping step of the hyper video production method according to an embodiment of the present invention;
5 is a flowchart of a face detection step of a hyper video production method according to an embodiment of the present invention;
6 is a flowchart of a face grouping step of the hyper video production method according to an embodiment of the present invention;
7 is a flowchart of an object detection step of a hyper video production method according to an embodiment of the present invention;
8 is a flowchart of an object tracking step of a hyper video production method according to an embodiment of the present invention;
9 is a reference diagram for explaining shot boundary detection in a method of manufacturing a hyper video according to an embodiment of the present invention;
10 is a reference diagram for explaining a attraction relationship in a continuous shot row of a hyper video production method according to an embodiment of the present invention;
11 is a conceptual illustration photo of a quadrilateral feature point for detecting a face region of a hyper video production method according to an embodiment of the present invention;
12A and 12B illustrate examples of frame screens of before and after changes (# 91 and # 93) when the shot boundary changes in the hyper video production method according to an embodiment of the present invention.
13A and 13B illustrate examples of frame screens of before and after changes (# 5350 and # 5352) when the scene changes in the hyper video production method according to an embodiment of the present invention.
14 is an example of faces grouped into similar faces in the method of producing a hyper video according to an embodiment of the present invention;
15 is a screen example in which a user selects an object area in a hyper video production method according to an embodiment of the present invention;
16 is an example of a screen in which an object is detected in each representative frame of an object selected by a user of a method of producing a hyper video according to an embodiment of the present invention;
17A to 17C illustrate examples of screens for which an object is tracked in adjacent frames # 404, # 420, and # 479 with respect to an object detected in a representative frame of the hyper video production method according to an embodiment of the present invention;
18A and 18B illustrate basic screen examples of a user UI of a hyper video production method according to an embodiment of the present invention;
19 is an example of object information editing screen of a user UI of the method of producing a hyper video according to an embodiment of the present invention;
20 is an example of an object DB export window screen of a user UI of a method of producing a hyper video according to an embodiment of the present invention;
21 is an example of a media output window screen of a user UI of a method of producing a hyper video according to an embodiment of the present invention;
22A to 22C illustrate examples of an object information display and a hyperlink screen of a user UI of the hyper video production method according to an embodiment of the present invention.

The present invention can be embodied in many other forms without departing from the spirit or main features thereof. Accordingly, the embodiments of the present invention are to be considered in all respects as merely illustrative and not restrictive.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, the terms "comprises", "having", "having", and the like are intended to specify the presence of stated features, integers, steps, operations, components, Steps, operations, elements, components, or combinations of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like or corresponding elements are denoted by the same reference numerals, and a duplicate description thereof will be omitted. In the following description of the present invention, if it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

1 is a block diagram of a hyper video production apparatus according to an embodiment of the present invention, Figure 2 is a conceptual diagram of a video hierarchy structure of a hyper video storage device according to an embodiment of the present invention.

A hyper video information file producing apparatus 1000 is provided for composing a hyper video in which additional information of a person or an object included in a video file is linked by a hyperlink.

The hyper video information file producing apparatus 1000 may be a conventional computer system having a computing element such as a central processing unit, a system DB, a system memory, an interface, and the like. It can be seen that it functions as the hyper video information file producing apparatus 1000 by installation and driving. A description of the general configuration of such a computer system is omitted, and the following description will focus on the configuration of functional aspects required for the description of the embodiments of the present invention.

The hyper video information file producing apparatus 1000 includes a scene grouping module 110.

The scene grouping module 110 may extract a frame from an input video file, detect a shot boundary from the extracted frames, extract a representative frame for each detected shot, and provide information about the shot and the representative frame. It performs a function of storing in the video layer DB (170). This function can be implemented by shot detection means (not shown).

In addition, the scene grouping module 110 groups similar shots among each shot into one scene, and generates video hierarchy information in which the frames, shots, and scenes are structured as shown in FIG. 170) to save. This function can be implemented by scene dividing means (not shown).

The video layer DB 170 is a DB that manages a video hierarchy such as shot and scene information of a video and a representative frame of shots.

The apparatus 1000 for producing a hyper video information file also includes an object detection module 140 that detects the object area from the representative frames and stores the object area in the object location DB 180.

The apparatus 1000 for producing a hyper video information file also tracks positions in adjacent frames from the representative positions of the objects detected in the respective representative frames, and further stores the positions in the object position DB 180. It is provided.

The object position DB 180 is a DB for managing the position of the object region detected in each frame including the representative frame and the position (or grouped face position) of the face region described later.

The apparatus 1000 for producing a hyper video information file also performs a function of receiving a selection of an object region of interest in an arbitrary frame from a user and storing the object information input from the user in the object information DB 190. A UI module 160 is provided. The user UI module 160 provides various user UI means such as a video reproducing means, a scene viewing means, an object viewing means, an object information viewing means, an automatic processing information viewing means, and the like according to the provision function thereof. The object information DB 190 is a DB that manages various additional information provided to objects (or faces).

The user UI module 160 may have a function of providing information stored in the object location DB 180 or the object information DB 190 to a user and receiving correction and edit information on the provided information.

In particular, the user UI module 160 provides a function of the user UI module 160 with respect to a frame in a scene in which scene grouping, object detection, and object tracking are performed, even if object detection or tracking of the entire frame is not completed. This can be configured to take place.

In addition, in the user UI module 160, user input and correction edit information input of information stored in the object location DB 180 or object information DB 190 may be stored in the video layer DB 170. Depending on the video layer, this can be done with a hierarchical approach of scene-shot-frames.

The hyper video information file producing apparatus 1000 also includes a control module 100 that performs at least a function of generating a hyper video information file by mapping the object location DB and the object information DB contents. The control module 100 performs the overall operation control of the hyper video information file producing apparatus 1000 of the present embodiment together with the above functions.

Meanwhile, the apparatus 1000 for producing a hyper video information file according to the present embodiment includes a face region detection module 120 for detecting a face region included in each of the detected representative frames.

The apparatus 1000 for producing a hyper video information file also includes a face grouping module 130 for proceeding face grouping with similar faces among the detected face regions and storing the same in the object position DB 180.

In the hyper-video information file producing apparatus 1000 of the present embodiment, the representative frame preferably uses a frame within a setting range corresponding to the beginning of the shot based on the start frame of each shot, but does not use the first frame.

In the hyper-video information file producing apparatus 1000 of the present embodiment, the object information DB manages the object list in a tree structure.

3 is an overall flowchart of a hyper video production method according to an embodiment of the present invention.

As shown, the hyper video production method according to the present embodiment starts from the start step (S10) of the production process, the video file input step (S20), the scene grouping step (S30), the face region detection step (S40), Face grouping step (S50), object detection step (S60), object tracking step (S70), object information input step (S80), result generation (S90), FLV (Flash Video) and XML (Extensible Mark-up Language) transformation The output stage S92 is followed by the termination stage S94.

4 is a flowchart of a scene grouping step of the hyper video production method according to an embodiment of the present invention, and the scene grouping step S30 will be described in detail with reference to the drawings.

Frames are extracted from the input video file (S310). This can be done, for example, by creating a project for the target video file and then extracting frames from the video file using a sample grabber of DirectX. In this way, if the frame is extracted from the video file using the DirectX method, if the DirectX filter is supported for a codec of any format, it can be used in the hyper video production apparatus.

Shot boundaries are detected from the extracted frames (S320 to S340), a representative frame is extracted for each detected shot, and information about the shot and the representative frame is stored in the video layer DB (S350).

In this case, the representative frame may use a frame within a setting range corresponding to the beginning of the shot based on the start frame of each shot, but does not use the first frame. This is because the first frame of the shot may not be clean in the case of a fade shot. Preferably, a frame five frames after the start frame of the shot is used, but a frame corresponding to the initial portion (eg, six frames) is preferably used.

In this process, the shot boundary detection (S320 to S340) measures the degree of change between each extracted frame and the previous frame (S320), and if the degree of change exceeds the first threshold (S330). Is detected by the shot boundary (S340).

The degree of change between frames can be viewed as the distance between frames. Representative methods for quantitatively calculating the distance between frames known in the art include a pixel point based method, a block based method, and a method based on color information.

Preferably, the present embodiment uses a method based on color information. A method based on general color information first constructs a color histogram in various color spaces RGB, HSV, YIQ, L * a * b *, L * u * v * or Gray space, and then the distance between the color histograms in adjacent frames. Is calculated and compared with a certain threshold T.

For a method of measuring the degree of change between frames, a paper by TY Liu, KT Lo, XD Zhang, and J. Feng, "A new cut detection algorithm with constant false-alarm ratio for video segmentation" (J. Vis. Commun. Image R., 15 (2): 132-144, 2004.), and in the papers of RAJoyce and B.Liu, "Temporal Segmentation of Video Using Frame and Histogram Space" (IEEE Trans. Multimedia, vol. 8, no. 1, pp. 130-140 (2006)).

For example, the color histogram H in the frame can be obtained as follows.

H (m) = (number of pixel points with m color) / (number of total pixel points), (m = 1 ... M, where M is the color value that can be represented in the RGB color space)

At this time, the distance between two frames a and b is calculated as in Equation 1 below.

[Equation 1]

: 프레임 a 에서의 색상히스토그램, (only,

: Color histogram at frame a,

: 프레임 b 에서의 색상히스토그램,

: Color histogram at frame b,

D (a, b): distance between frames a and b)

As a preferred example, the gradient is used as a distance value between color histograms calculated for each frame, and the first threshold value T (i) is actively defined by Equation 2 for the i-th frame.

&Quot; (2) "

(Μ (i): average color histogram distance value at i-th frame,

σ (i): Standard deviation of the color histogram distance value at the i th frame,

α: weight (constant)

In order to calculate the distance mean and standard deviation, a certain window area may be set in the i th frame, and the mean and standard deviation may be calculated within this window area. In addition, if the weight α = 3, the shot boundary may be extracted as a singular value according to the '3σ law' in probability and mathematical statistics.

In this regard, FIG. 9 is a reference diagram for explaining shot boundary detection in the hyper video production method according to an embodiment of the present invention, and illustrates a state in which the shot boundary is detected.

Thereafter, similar shots among the shots are grouped into one scene, and the video layer structure information in which the frames, shots, and scenes are structured is generated and stored in the video hierarchy DB (S360 to S380). This scene grouping is used to facilitate the user's access. In other words, if the content of the video file is long and hundreds of shots are detected, a large amount of time is spent investigating many shots without using the scene. This can be done more conveniently hierarchically with> shot-> frame.

In the above process, the scene grouping (S360 to S380) may obtain a color histogram for each representative frame of each shot, and with the representative frames of a predetermined number of adjacent shots adjacent to each other in both directions, with respect to the representative frame of each shot. When the similarity is calculated (S360), and the workforce ratio received from the adjacent shots in proportion to the mutual similarity exceeds a second threshold value (S370), the shot including the representative frame is added as a start of a new scene. This is done by storing in the video layer DB (S380).

In more detail, in order to quantitatively evaluate the similarity of the shots, mutual similarity calculated based on the color histogram in the representative frames of the shots is used as in Equation 3 below.

&Quot; (3) "

: 샷 a 의 대표 프레임에서의 색상히스토그램, (only,

: Color histogram of representative frame of shot a,

: 샷 b 의 대표 프레임에서의 색상히스토그램,

: Color histogram of representative frame of shot b,

),w: weights (

),

d: Minimum distance between shots a and b (the distance from the end frame of the previous shot to the start frame of the next shot,

C: constant defining the length of the shot,

Cor (a, b): mutual similarity between shots a and b)

FIG. 10 is a reference diagram for explaining a attraction relationship in a continuous shot column of a hyper video production method according to an embodiment of the present invention. It can be assumed that the shot receives manpower proportional to mutual similarity. At this time, the attraction ratio in the shot i to pay attention is calculated as in Equation 4 below.

&Quot; (4) "

R (i) = (right (i) + right (i + 1)) / ((left (i) + left (i + 1))

(Except left (i) = max {Cor (i, i-1), Cor (i, i-2), Cor (i, i-3)},

left (i + 1) = max {Cor (i + 1, i-1), Cor (i + 1, i-2)},

right (i) = max {Cor (i, i + 1), Cor (i, i + 2), Cor (i, i + 3)}

right (i + 1) = max {Cor (i + 1, i + 2), Cor (i + 1, i + 3), Cor (i + 1, i + 4)})

When calculating the manpower ratio R (i) in the shot i to be noted in this manner, the determination formula 'R (i)> T AND R (i)> R (i-1) AND R (i' is determined according to a predefined threshold value T. If i satisfies)> R (i + 1) ', the shot i is set as the start of a new scene. If the above decision is not satisfied, the shot i is added to the previous scene.

5 is a flowchart of a face detection step of the hyper video production method according to an embodiment of the present invention, and FIG. 6 is a flowchart of a face grouping step of the hyper video production method according to an embodiment of the present invention. The area detection step S40 and the face grouping step S50 will be described in detail.

In this step, the face regions included in the detected representative frames are detected (S40), the face grouping is performed with similar faces among the detected face regions, and stored in the object position DB (S50). The object tracking step (S70) described below is performed by using each face area as a tracking target object. While general object detection takes a method of specifying a target object to be detected by a user, face detection is automatically performed in a representative frame when grouping scenes, which provides an advantage of increasing the degree of automation and efficiency of operation. do.

First, referring to the face region detection step S40, face regions included in the detected representative frames are detected (S410 ˜ S430).

If there is a detected face area (S440), the face area is additionally stored in the unknown face list (S450).

If it is determined that a face similar to the detected face region exists in the object position DB (S460), the face region is additionally registered to the corresponding face group of the object position DB (S470).

Known face detection methods include a knowledge-based method, a feature-based method, a template-matching matching method, and an appearance-based method.

Preferably, in this embodiment, an appearance-based method is used. The appearance-based method obtains face and non-face areas from different images, learns the acquired areas to make a learning model, and compares input images and learning model data to detect faces. It is known as a relatively high performance method.

Regarding such face detection, the paper by Jianxin Wu, S. Charles Brubaker, Matthew D. Mullin, and James M. Rehg, "Fast Asymmetric Learning for Cascade Face Detection," by IEEE Transcription on Pattern Analysis and Machine Intelligence, 30, No. 3, MARCH 2008.), and Paul Viola, Michael Jones, "Rapid Object Detection using a Boosted Cascade of Simple Features" (Accepted Conference on Computer Vision and Pattern Recognition 2001.) have.

Particularly, as a preferable example, the face area detection may generate a YCbCr color model from RGB color information of the representative frame, separate color information and brightness information from the created color model, and detect a face candidate area by the brightness information. And defining a quadrilateral feature point model for the detected face candidate region, and detecting a face region based on learning data trained by the AdaBoost learning algorithm. The AdaBoost learning algorithm is known as an algorithm that generates a strong classifier with high detection performance through linear combination of weak classifiers.

In this regard, FIG. 11 is a conceptual illustration photograph of a quadrilateral feature point for detecting a face region of the hyper video production method according to an embodiment of the present invention, and illustrates a state where the quadrilateral feature point is used for face region detection. .

Next, in the face grouping step (S50), when the face region detection for all the representative frames is completed through the above process, the face grouping is performed with similar faces among unknown faces stored in the unknown face list (S510). Store in the object location DB (S520 ~ S530).

Unlike general objects, faces are grouped with a face grouping step. If the grouping is performed and each group is automatically set as a face object, there is an advantage in that an object editing efficiency according to a person can be improved. In addition, when the features of the person who has been edited before are outputted to the object position DB and referred to later editing, the editing of the same person does not have to be repeated, thereby improving the editing efficiency.

Conventionally known face grouping methods, such as geometric matching method that recognizes a face using the position or size of the eyes, nose, mouth, etc., which are geometric features of the face, or distance between them, template image stored in a database Compared with the data analysis, there is a template pattern matching method that recognizes a face by analyzing the correlation between each other, a method using an Arfiti-cial Neural Network (ANN), a Support Vector Machine (SVM) method, and a Hidden Markov Model (HMM) method. .

In particular, as a template pattern matching method, a principal component analysis (PCA) method, an independent component analysis (ICA) method, a linear dis-criminant analysis (LDA) method, and a 2-Dimensional PCA), PCA / LDA, Component-based (DCT / LDA), Local Feature Analysis (LFA).

Particularly, as a preferable example, the face grouping extracts a Gabor representation of a face image by Gabor wavelet transformation, performs nonlinear mapping, and performs linear discriminant analysis in kernel space. And face grouping by a sequential grouping algorithm.

Gabor wavelets can effectively express local and discernible feature quantities, which is known to be useful in pattern detection and face recognition. Linear discriminant analysis (KLDA, GDA) in kernel space is a method of improving recognition ability by realizing linear discriminant analysis (LDA) in kernel space.

In the sequential grouping algorithm, face grouping is performed by feature vectors reduced in L dimension by linear discriminant analysis in kernel space, which is performed by calculating a similarity between a new face and every face already grouped.

Regarding this face grouping, a paper by G. Baudat and F. Anouar, "Generalized discriminant analysis using a kernel approach" (Neu-ral Comput., Vol. 12, no. 10, pp. 2385-2404, 2000.) It may be understood through such.

7 is a flowchart of the object detection step of the hyper video production method according to an embodiment of the present invention, the object detection step (S60) proceeds as follows.

The user selects and inputs an object region of interest in an arbitrary frame, detects the object region in the representative frames, and stores the object region in an object position DB (S610 to S650).

In this object input process, the user selects an object in an arbitrary frame. For example, you can move to any frame in the 'media player and task pane' as well as the representative frame, and select the object region with the quadrilateral selection tool in the work region. The selected object is added to the object list as a new object. You can also integrate this object by dragging and dropping into another object that has already been added.

If an object is selected in any frame like this, the object position in subsequent frames within the shot including the frame can be tracked by clicking 'Track object' in the work tool window.

On the other hand, the object selection in the process may be used a known method generally known. For example, a method of moving an arrow to an object of interest by moving the arrow and then drawing by selecting an area selection figure such as a polygon or a circle may be used.

In a preferred embodiment, the object region detection is performed by detecting a scale invariant feature transform (SIFT) feature, performing an initial matching using a spreading distance, and performing a final matching using a SIFT descriptor.

SIFT feature detection consists of, for example, basic calculation steps such as scale-space extrema detection and keypoint localization, orientation assignment, and keypoint descriptors. .

Regarding such object area detection, D. Lowe's paper, "Distinctive Image Features from Scale-Invariant Keypoints" (IJCV, 60 (2), pp. 91-110, 2004. 1, 2, 6, 7, 8) And, by Haibin Ling, Kazunori Okada, "Diffusion Distance for Histogram Com-parison" (2006), and V. Ferrari, T. Tuytelaars, L. van Gool, "Simultaneous Object Recognition and Segmentation by Image Exploration" ( ECCV, 2004.).

8 is a flowchart of an object tracking step of a hyper video production method according to an embodiment of the present invention.

In the object tracking step (S70), the location in the adjacent frame is tracked from the representative positions of the objects detected in the respective representative frames and further stored in the object location DB (S70).

In more detail, the next frame is input based on one representative frame (S710 to S730).

If it is determined that the next frame does not deviate from the shot range including the representative frame (S740), the object position is tracked in the corresponding frame (S750).

When it is determined that the next frame is out of the shot range including the representative frame (S740), the next representative frame is input and the process proceeds to step S710 or less.

As a result of tracking the object location (S750), if there is a tracked object (S760), the location of the tracked object is additionally stored in the object location DB (S770).

Known object tracking methods include low-level analysis, feature analysis, active shape models, linear subspace methods, neural networks, and statistical methods.

Particularly, as an example, the object tracking may include selecting a representative location area of an object in the representative frame, performing color modeling of the object based on the location area of the selected object and the representative frame, and tracking. To this end, when the next frame is input, the object movement is tracked by means of the Mean Shift method, the position of the search window is determined as the position of the object, and the size change of the object is estimated. The Mean Shift method is a method of modeling an object based on color and raising its gradient to find the mode of stochastic variance of that color.

Next, after the object tracking through the above process, in the object information input step (S80) stores the object information input from the user in the object information DB. The object information may include, for example, an image, a text, a voice, a video, a web address (URL), and the like related to the object.

The object information DB manages a list of objects in a tree structure. For example, in a file system, create a tree structure of virtual object groups (corresponding to directories), such as the relationship between directories and files, and store objects (corresponding to files) in them. do. Managing many objects in a tree structure like this can improve object management efficiency. Such a tree structure can be illustrated in the form of Table 1 below.

Whole object Face Actor Hong Gil Dong Fei Hong Singer Hwang Jin-yi dress Pants Blue jeans ... ... ...

On the other hand, in the object information input step (S80), preferably, when the user provided the information stored in the object location DB or the object information DB, and receives the correction edit information for the provided information, the object to the corrected edit information Manual edits can be made to modify and store the location DB or object information DB.

For example, in automatic object tracking or automatic face region detection, the detection results are indicated by the object position in the frame. If the object position is not correct, the user can, for example, navigate to the frame in the Media Player and Task Pane and change or delete the object position in the frame with the mouse.

The object position is represented, for example, by a square rotated in the frame, with nine handlers in total: four corners, four sides, and one rotation. You can modify the location by moving the handler with the mouse or by clicking inside the object location. You can also delete the object position in the frame by pressing the delete button.

On the other hand, in the case of face grouping, if different faces are grouped into the same face, the face is selected from the object list and the face is dragged and dropped from the list of representative frames displayed on one side of the screen. You can move to or delete it with the Delete button.

In the above process, preferably, even if the object detection or tracking for the entire frame is not completed, the modified edit information may be input to the frames in the scene where the scene grouping, object detection, and object tracking are performed.

In more detail, automatic processes such as scene segmentation, object detection, and object tracking take a long time. In view of this point, in the manufacturing method of the present embodiment, such automatic processes are performed in a background manner, and the user can perform other editing work even during this process.

On the other hand, preferably, the user-provided and corrected edit information input of the information stored in the object location DB or the object information DB is to be a hierarchical approach of scene-shot-frame according to the structured video layer stored in the video layer DB. Can be.

Next, in the result generation step (S90), the hyper video information file is generated by mapping the object location DB and the object information DB contents (S90). The hyper video information file is generated as meta data in XML format.

Then, when the user outputs the result, the video file is converted into the FLV format, and the meta data of the XML format including the object location DB and the object information DB contents is output.

The conversion of the video file to the FLV format is intended to allow this video file to be displayed in a flash player (extension * .swf). In addition, the terminal player to the swf file is to be able to play on the web browser without additional activeX installation.

12A to 22C are screen examples according to the progress of each step.

12A and 12B are examples of frame screens of before and after changes (# 91, # 93) when the shot boundary changes in the hyper video production method according to an embodiment of the present invention, and FIGS. 13A and 13B illustrate one embodiment of the present invention. Example of each frame screen of before and after changes (# 5350, # 5352) when the scene changes in the hyper video production method according to an embodiment, Figure 14 is grouped into similar faces of the hyper video production method according to an embodiment of the present invention Examples of faces, FIG. 15 is a screen example in which a user selects an object area in a hyper video production method according to an embodiment of the present invention, and FIG. 16 is an object selected by a user in a hyper video production method according to an embodiment of the present invention. Examples of a screen in which an object is detected in each representative frame with respect to each other, FIGS. 17A to 17C are adjacent frames (# 404 and # 4) with respect to an object detected in the representative frame of the hyper video production method according to an embodiment of the present invention. 20, # 479) is an example of a screen tracked by the object, Figures 18a and 18b is a basic screen example of the user UI of the hyper-video production method according to an embodiment of the present invention, Figure 19 is an embodiment of the present invention Example of the object information editing screen of the user UI of the hyper video production method, Figure 20 is an example of the object DB export window screen of the user UI of the hyper video production method according to an embodiment of the present invention, Figure 21 is an embodiment of the present invention Examples of the media output window screen of the user UI of the hyper video production method according to the present invention, FIGS. 22A to 22C are examples of the object information display and the hyperlink screen of the user UI of the hyper video production method according to an embodiment of the present invention.

Embodiments of the present invention include a computer readable recording medium including program instructions for performing various computer-implemented operations. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The recording medium may be those specially designed and configured for the present invention or may be those known and used by those skilled in the computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical recording media such as CD-ROMs, DVDs, magnetic-optical media such as floppy disks, and ROM, RAM, flash memory, and the like. Hardware devices specifically configured to store and execute the same program instructions are included. The recording medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, or the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

1000: Hyper video information file producer
110: scene grouping module 120: face region detection module
130: face grouping module 140: object detection module
150: object tracking module 160: user UI module
170: video layer DB 180: object location DB
190: object information DB

Claims (27)

In a hyper video in which additional information of a person or an object included in a video file is produced as an information file and linked by a hyperlink, a method of producing a hyper video information file executed in an information file producing apparatus,
(a) extracting frames from the input video file;
(b) detecting a shot boundary from the extracted frames, extracting a representative frame for each detected shot, and storing information about the shot and the representative frame in a video layer DB, wherein the representative frame is a start frame of each shot; Use a frame within the setting range corresponding to the beginning of the reference;
(c) grouping similar shots among the shots into one scene, and generating and storing the frame, shot and scene-structured video hierarchy information in a video hierarchy DB, wherein the similarity of each shot is determined by For the representative frame, determined based on the mutual similarity obtained for the representative frames of the predetermined number of adjacent shots adjacent in both directions;
(d) selecting and inputting an object region of interest in an arbitrary frame from a user, detecting the object region in the representative frames, and storing the object region in an object position DB;
(e) tracking the position in the adjacent frame from the positions of the object region detected in each of the representative frames and additionally storing the position in the object position DB;
(f) storing the object information input from the user in the object information DB; And
(g) generating a hyper video information file by mapping the content of the object location DB and the object information DB to generate a hyper video information file.
The method of claim 1,
Shot boundary detection of the step (b),
And measuring the degree of change between each extracted frame and the previous frame, and detecting the corresponding frame as a shot boundary when the degree of change exceeds the first threshold value.
3. The method of claim 2,
The gradient is a distance value between color histograms calculated for each frame,
And wherein the first threshold value T (i) is defined by Equation 2 for the i-th frame.
&Quot; (2) "

(Μ (i): average color histogram distance value at i-th frame,
σ (i): Standard deviation of the color histogram distance value at the i th frame,
α: weight (constant)
The method of claim 1,
Scene grouping of step (c),
Find the color histogram for each frame of each shot,
Regarding the representative frames of each shot, the mutual similarity with the representative frames of the predetermined number of adjacent shots adjacent to each other in both directions,
When the workforce ratio received in proportion to the mutual similarity from the adjacent shots exceeds the second threshold value, the shot including the representative frame is added to the beginning of a new scene and stored in the video layer DB. How to make a hyper video information file.
The method of claim 1,
The object region detection of the step (d),
Scale Invariant Feature Transform (SIFT) A method for producing a hyper video information file comprising detecting a feature, performing an initial match using a spreading distance, and performing a final match using a SIFT descriptor.
The method of claim 1,
The step (e)
(e1) receiving a next frame based on one representative frame;
(e2) if it is determined that the next frame does not deviate from the shot range including the representative frame in step (e1), tracking an object position within the frame;
(e3) if it is determined that the next frame is out of the shot range including the representative frame in step (e1), receiving the next representative frame and proceeding to step (e1);
and (e4) if there is the tracked object location in step (e2), additionally storing the tracked location of the tracked object in the object location DB.
The method of claim 1,
Object location tracking of step (e),
Receiving a selection of a location area of an object in the representative frame;
Performing color modeling of the object based on the location area of the selected object and the representative frame;
Tracking an object movement by a mean shift method when a next frame is input for tracking; And
And determining the position of the search window as the position of the object and estimating a change in the size of the object.
The method of claim 1,
After the step (c)
(h) detecting the face regions included in the representative frames extracted in the step (b), proceeding to face grouping by the face grouping method with respect to the detected face regions, and storing them in the object position DB; And proceeding to step (e) by using each of the grouped face areas as a tracking target object.
delete 9. The method of claim 8,
The face area detection is,
Creating a YCbCr color model from the RGB color information of the representative frame, separating color information and brightness information from the created color model, and detecting a face candidate area based on the brightness information; And
Defining a quadrilateral feature point model for the detected face candidate region, and detecting a face region based on the training material trained by the AdaBoost learning algorithm on the quadrilateral feature point model; hyper video information How to create a file.
9. The method of claim 8,
The face grouping is,
Extract Gabor expression of face image by Gabor wavelet transform, perform nonlinear mapping, linear discriminant analysis in kernel space, and face grouping by sequential grouping algorithm Method for producing a hyper video information file, characterized in that is made.
The method of claim 1,
(i) providing the user with the information stored in the object location DB or the object information DB, and correcting and storing the corrected edit information in the object location DB or the object information DB when receiving correction edit information for the provided information Hyper video information file production method characterized in that it comprises a; further comprising.
The method of claim 12,
Hyper video, characterized in that the step (i) can be performed on the frames in the scene in which the scene grouping, object region detection, and object position tracking are performed even if the object region detection or object position tracking for the entire frame is not completed. How to create an information file.
The method of claim 12,
The user input and correction edit information input of the information stored in the object location DB or the object information DB is configured to be a hierarchical approach of scene-shot-frame according to the structured video layer stored in the video layer DB. How to make a hyper video information file.
The method of claim 1,
And the representative frame uses a frame within a setting range corresponding to the beginning of the shot based on the start frame of each shot, but does not use the first frame.
The method of claim 1,
And the object information DB manages a list of objects in a tree structure.
The method of claim 1,
And wherein the hyper video information file is generated as meta data in an XML format.
In a hyper video in which additional information of a person or an object included in a video file is produced as an information file and linked by a hyperlink, a method of producing a hyper video information file executed in an information file producing apparatus,
Detecting a shot boundary from the frames extracted from the input video file, extracting a representative frame for each detected shot, and storing information about the shot and the representative frame in the video layer DB, wherein the representative frame is used for each shot. Use a frame within a setting range corresponding to the beginning relative to the start frame;
Grouping similar shots of each shot into one scene, and generating and storing the frame, shot and scene-structured video hierarchical information in a video hierarchy DB-similarity of each shot to a representative frame of each shot Is determined based on mutual similarity obtained for representative frames of a predetermined number of adjacent shots adjacent to each other in forward and backward directions;
Detects the face area included in each representative frame, performs face grouping by the face grouping method with respect to the detected face areas, stores them in the object position DB, and detects the object area selected by the user in the representative frame. And tracking the position of the face grouped with the object in an adjacent frame and storing the position in the object position DB;
And storing the object information input from the user in the object information DB and mapping the object location DB and the object information DB contents to generate a hyper video information file.
A computer-readable recording medium having recorded thereon a program for executing each step of the method according to any one of claims 1 to 8 and 10 to 18.
An apparatus for producing a hyper video information file for composing a hyper video in which additional information of a person or an object included in a video file is linked by a hyperlink,
Extracting frames from the input video file, detecting shot boundaries from the extracted frames, extracting representative frames for each detected shot, and storing information on shots and representative frames in the video layer DB; The representative frame uses a frame within a setting range corresponding to the beginning based on the start frame of each shot. Groups similar shots of each shot into one scene, and the frame, shot and scene are structured in a video hierarchy. A function of generating information and storing the information in the video layer DB-The similarity of each shot is determined based on the mutual similarity obtained for the representative frames of a predetermined number of adjacent shots adjacent to each other in both directions. A scene grouping module for performing the task;
An object detection module configured to detect an object area in the representative frames and store the object area in an object location DB;
An object tracking module for tracking a position in an adjacent frame from positions of the object region detected in each representative frame and additionally storing the position in an object position DB;
A user UI module for selecting and receiving an object region of interest in an arbitrary frame from a user and storing the object information input from the user in an object information DB; And
And a control module for mapping the object location DB and the object information DB contents to generate a hyper video information file.
21. The method of claim 20,
A face region detection module detecting a face region included in each representative frame extracted by the scene grouping module; and
And a face grouping module configured to perform face grouping by the face grouping method on the detected face regions and to store them in an object position DB.
21. The method of claim 20,
The user UI module,
And providing information stored in the object location DB or the object information DB to a user, and receiving correction edit information for the provided information.
The method of claim 22,
Even if the object region detection or object position tracking for the entire frame is not completed, the user UI module can be provided with respect to frames in the scene where scene grouping, object region detection, and object position tracking are performed. Hyper video information file producer.
The method of claim 22,
The user input and correction edit information input of the information stored in the object location DB or the object information DB is configured to be a hierarchical approach of scene-shot-frame according to the structured video layer stored in the video layer DB. Hyper video information file producer.
21. The method of claim 20,
And the representative frame uses a frame within a setting range corresponding to the beginning at the beginning of each shot, but does not use the first frame.
21. The method of claim 20,
And the object information DB manages the object list in a tree structure.
An apparatus for producing a hyper video information file for composing a hyper video in which additional information of a person or an object included in a video file is linked by a hyperlink,
Detecting shot boundaries from frames extracted from the input video file, extracting representative frames for each detected shot, and storing information about shots and representative frames in the video layer DB. Use a frame within a setting range corresponding to the beginning relative to the start frame;
Grouping similar shots of each shot into one scene, and generating and storing the frame, shot and scene-structured video hierarchy information in a video hierarchy DB-the similarity of the shots is represented in a representative frame of each shot. Is determined based on mutual similarity obtained for representative frames of a predetermined number of adjacent shots adjacent to each other in forward and backward directions;
Detects the face area included in each representative frame, performs face grouping by the face grouping method with respect to the detected face areas, stores them in the object position DB, and detects the object area selected by the user in the representative frame. And tracking a position of the face grouped with the object in an adjacent frame and storing the position in the object position DB; And
And storing the object information input by the user in the object information DB, and mapping the object location DB and the object information DB contents to generate a hyper video information file.

Images