KR101283759B1 - Method for semantic annotation and augmentation of moving object for personalized interactive video in smart tv environment - Google Patents

Abstract

PURPOSE: The representation of semantic annotations of a moving object in individualized interactive video in a smart TV environment and an augmentation method thereof are provided to sample video data using a dynamic sampling method according to the movement speed of the moving object, thereby accurately tracking uncertain position information of the object and effectively reducing the amount of data. CONSTITUTION: Relative coordinate variations are calculated which indicate the movement speed of an object moving according to a frame change of video data. A dynamic sampling cycle regarding the coordinates of an object position is determined according to the relative coordinate variations. The video data is sampled according to the dynamic sampling cycle. The coordinates of the object position are saved. [Reference numerals] (AA) Standard deviation; (BB) Sampling cycle; (CC) Real position data

Description

FIELD OF SEMANTIC ANNOTATION AND AUGMENTATION OF MOVING OBJECT FOR PERSONALIZED INTERACTIVE VIDEO IN SMART TV ENVIRONMENT}

Embodiments of the present invention relate to a method of expressing meta information for merchandise and personalization in TV content in a smart TV environment, and more specifically, to perform video semantic annotation on TV content more effectively by dynamically calculating a sampling period. The present invention relates to a method for personalization based on metadata of a product in TV content.

Recently, the production and consumption of various multimedia contents by users in the Web 2.0 environment has exploded, and due to the convergence of broadcasting and communication, the form of unidirectional services such as TV can be interactively interactive (Interactive TV, Smart TV). In this environment, smart TV not only accurately understands the user's intentions and provides a search for the corresponding content, but also provides real-time information on the semantic information such as information or events of various objects present in the content while the user watches the content. Interactive video is attracting attention in the environment, and there are many attempts to efficiently store and retrieve semantic information such as information or objects of objects present in content for interactive video.

In particular, since the video data includes moving objects representing spatial position changes with the change of time, in order to represent moving objects included in the video data, it is effective to store the position change of the object with the change of time in a database. I need a way. In general, when managing continuous data using a database, it is very inefficient to store all changed position data every frame, so that the position data of the object is stored by determining a constant sampling time. At this time, there is a need for a method capable of properly expressing uncertain location data since location data does not exist at any point in time not stored in the database.

Traditionally, as a method for estimating the position of a moving object and augmenting a virtual object, methods for estimating the position of the moving object at an unstored viewpoint using two or more position data have been studied. However, these studies store the movements of objects by a fixed sampling time and do not consider the moving speed of objects in the content. The moving speed of the object is an important factor in determining the sampling time, and the sampling time determines the amount of position information of the moving object stored.

Therefore, when the movement of the object is fast, the sampling period is shortened to accurately represent the uncertain position data. On the contrary, when the movement of the object is slow, the sampling period is increased to reduce the amount of data. .

In addition, information providers Seller want to provide their information (advertising products, travel information, weather, etc.) to information buyers, and information buyers are interested in the information they are interested in. Since they want to be provided only, there is a need for a method of expressing information for personalization.

In order to augment virtual objects by following the movement of moving objects (including uncertain position data) shown in the video, Cubic Spline Interpolation is used to provide a dynamic sampling method according to the moving speed of moving objects.

The dynamic sampling method according to the moving speed of the moving object accurately tracks the uncertain position information of the object and at the same time effectively reduces the amount of data and provides a method of expressing meta information to represent the augmented object. Provides a way to personalize

According to an aspect of the present invention, a method of representing a moving object moving in a video comprises: obtaining a relative coordinate change amount representing a moving speed of an object moving according to a frame change in the video data; Determining a dynamic sampling period for the position coordinate of the object according to the relative coordinate change amount; And storing the position coordinates of the object by sampling the video data at the dynamic sampling period.

The determining of the dynamic sampling period may include determining a dynamic sampling period for the X coordinate of the object according to the relative coordinate change amount of the X coordinate according to the frame change, and changing the relative coordinate change amount of the Y coordinate according to the frame change. The method may include determining a dynamic sampling period for the Y coordinate of the object.

The determining of the dynamic sampling period may include defining a plurality of sampling periods having different frame periods and threshold conditions for each of the sampling periods so that the sampling period may be dynamically determined according to a threshold condition corresponding to the relative coordinate change amount. Can be.

The determining of the dynamic sampling period may be determined as a short sampling period among the plurality of sampling periods as the amount of change with respect to the position coordinates of the object increases.

According to another aspect of the present invention, it occurs during the process of superimposing the object through the augmentation information viewer according to the user's personalization by separating the interactive server to augment the moving moving object in the video content played in the video content server (Service) In order to solve the tracking problem of a moving object, a random interpolation is performed using an algorithm for interpolating uncertain position data between points in two arbitrary position data with respect to the position coordinates of the object stored in the dynamic sampling period. The method may further include estimating the position coordinates of the object with respect to the viewpoint.

According to another aspect of the present invention, a method of representing a moving object moving in a video may further include annotating meta information associated with the object in a position coordinate of the object. In this case, the meta information may be displayed by being synthesized with the video data on a screen on which the video data is reproduced.

The meta information has an ontology class hierarchy structure based on LOD (Linking Open Data), a target class for setting a target user to provide the meta information, and the meta information is displayed on a screen on which the video data is played. PlayTime class for setting the time point to be displayed, Shape class for setting the form of the meta information, Event class for defining the interaction environment for the meta information, The meta information may include at least one class of a motion class for setting an interpolation condition for the position coordinate of the object.

The meta information may be provided as personalized information for a target user set in the target class among users who receive the video data.

According to another aspect of the present invention, a system for personalization of video content includes a video content server for maintaining the video content; And an interactive content server that maintains annotated meta information about the video content. In this case, the video content server obtains a relative coordinate change amount representing a moving speed of an object moving according to a frame change in the video content, and then determines a dynamic sampling period for the position coordinate of the object according to the relative coordinate change amount, The video content is sampled at the dynamic sampling period to store position coordinates of the object. Thus, the meta information may be annotated with the position coordinates of the object and displayed on the screen on which the video content is played back.

It is an effective way to augment virtual objects on the screen of an individual by selectively expressing the augmented objects in the N-Screen era. Give a way.

By introducing the concept of a video content server and an interactive content server, a new dimension of video is synthesized by overlapping augmented objects using meta information of augmented objects in the interactive content server without changing the original data of the video content. Can present techniques for betting. As a technique for this, a method of accurately expressing meta information for representing an augmented object while effectively tracking an uncertain position information of an object and effectively reducing the amount of data through a dynamic sampling method according to a moving speed of a moving object And effectively suggest ways to personalize augmented objects.

1 is an exemplary diagram for describing an abstract data model and a discrete data model representing a position of a moving object in a video.
2 is an exemplary diagram for explaining the result of a polynomial according to the order.
3 is an exemplary diagram for describing a cubic spline interpolation method according to an embodiment of the present invention.
FIG. 4 is a diagram for describing a method of dynamically determining a sampling period using a standard deviation with respect to a moving speed of an object according to one embodiment of the present invention.
5 illustrates hierarchical structure and object attribute information of ontology class for merchandise and personalization according to an embodiment of the present invention.
FIG. 6 illustrates data attributes of a target class according to an embodiment of the present invention.
FIG. 7 illustrates data attributes of a PlayTime class according to an embodiment of the present invention.
FIG. 8 illustrates data attributes of a Shape class according to an embodiment of the present invention.
FIG. 9 illustrates data attributes of an Event class according to an embodiment of the present invention.
FIG. 10 illustrates data attributes of a motion class according to an embodiment of the present invention.
FIG. 11 is a diagram illustrating a system structure for merchandise and personalization of TV content according to one embodiment of the present invention.
12 illustrates a conceptual diagram for synthesizing video and meta data of TV content according to an embodiment of the present invention.
FIG. 13 is an exemplary screen illustrating a video of TV content before an information provider is annotated, according to an embodiment of the present invention.
FIG. 14 is an exemplary screen illustrating a video of personalized TV content based on profile information of a user and target information of an information provider according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In the embodiment of the present invention, an object recognition and tracking method based on Augmented Reality technology is used.

Augmented reality is a technology derived from one field of virtual reality and refers to a technology that combines the real world and the virtual world. While virtual reality realizes human-computer interaction by building a real-world environment into a computer environment and immersing in this virtual world, augmented reality can be viewed in the real world and the visual information that can be seen using a computer camera. It is a technology that makes it possible to experience various digital information more realistically about the real world by interacting by projecting non-visual information into real world information. For the augmented reality of the completed form, it is important to minimize the error that occurs in the process of registering the virtual object in the real space. In order to reduce this error caused by the mismatch between the real world coordinate system and the virtual world coordinate system, the virtual object must be accurately positioned in the real world that the user watches, and a tracking technique for accurately measuring the user's position and gaze direction is very important.

For this augmented reality: 1. Register virtual images in the real world 2. Combine real and virtual images, and 3. Technical elements of the real-time interaction process.

In the embodiment of the present invention, the physical position of the real world is considered as a video image. Based on this, 1. technology to register a virtual image in the video image, 2. combine the video image and virtual image, and 3. technical elements of the process of real-time interaction.

The approach of augmented reality used in the present invention has been studied to be divided into marker based and markerless based. The approach used in this patent uses a non-marker based augmented reality approach.

In the early stages of augmented reality, an artificial marker was introduced into the real-world environment, and the marker-based method that tracks the marker specified a distinguishable marker for the object to be registered. Examples are colored markers, markers with features, and 2D barcodes. The use of markers makes it easy to detect and has the advantage of being successfully applied to augmented reality, but it is widely used in indoor environments because of its narrow application range and unrealistic application.

Non-marker based can be classified into model based method and non-model based method. In the model-based approach, knowledge about the real world is acquired and stored as a 3D model of the object before object tracking is performed, and used to estimate the camera direction. In contrast, in the non-model-based method, the camera direction is estimated without prior information through a frame input from the camera. The model-based approach is simpler than the non-model approach, but the disadvantage is that only the objects stored in the model are tracked when they are in the visible region. Another advantage of the model-based approach is that virtual objects such as obscuration or collisions, Interaction between objects is possible. For this reason, the model-based method is currently the most researched field and can be divided into edge-based, optical flow-based, and texture-based. Edge-based can be divided into point sampling and edge extraction, and texture-based can be divided into template matching and interest point-based.

The technique used in the embodiment of the present invention will use augmented reality technology based on the point of interest.

There are three main techniques for augmented reality based on the interests: object recognition, object tracking, and object composition.

In order to extract these points of interest, feature descriptor information is extracted from image information using an algorithm such as Scale-Invariant Feature Transform (SIFT) or Speeded Up Robust Features (SURF). The information is compared with the feature descriptor information of the learned images, and a similar image is selected from the learned images based on the feature descriptor information extracted from the current image. This process is called 'object recognition'.

Subsequently, if the current still image is n-1, n, n + 1, ..., n + k images go through the same process. This process is called 'object tracking'. Coordinate values obtained during the object tracking process can be used to predict the movement of the object. The coordinates of the object can be extracted using SIFT and SURF, or the user can manually extract the actual object in the image. have.

Then, the information about the virtual object stored together with the selected image (from the learned image) is calculated with a pose (direction, tilt, etc.) with the current image, and the selected image is added to the video image based on the calculated pose value. Synthesis is performed and this is called object synthesis.

Representation of moving objects in video requires consideration of spatial properties that continuously change over time. To this end, abstract data, which is a set of continuous infinite points in three-dimensional space of (x, y, z) as shown in FIG. It can be expressed as an abstract data model.

However, storing an infinite number of sets in a database is a very inefficient method, and even when collecting data of a moving object, the position value of the object is obtained discretely at a predetermined time unit.

Therefore, the moving object can be efficiently stored in the database based on a discrete data model (FIG. 1B) that can represent the moving object in a finite set.

When the position of the moving object represented by the abstract model as shown in FIG. 1 (a) corresponds to the discrete model, it may be represented as a polygon formed by a set of three-dimensional finite points as shown in FIG. .

This discrete model is represented with a method of interpolating the values between each discrete interval in correspondence with the abstract model. In the case of FIG. 1B, the discrete data model can be simply expressed using linear interpolation.

Simple linear based interpolation and curve-based Lagrange Polynomial Interpolation to reduce the amount of data by sampling the position data of the abstract data model at regular intervals and to estimate the position data at uncertain points between sampling intervals. , Newton Interpolation, and Spline Interpolation are used.

When an interactive video needs to be annotated with the movement of a moving object that continuously changes its position in the video, it may be necessary to express the whole or partial movement of the moving object.

The simplest way to handle all of these moving object movements is to store the position of the moving object that changes continuously over all time. However, this method has a problem that even in the case of discrete data, the amount of data stored in the database is greatly increased, which is inefficient.

In order to solve the problem of increasing the data storage space, there is a study applying linear interpolation by acquiring the positions of objects in a sorted order at regular time intervals. However, the moving object in the video is more effective than the linear form obtained by the linear interpolation method to represent the irregular movement is more effective. Video consists of many frames, and digital video usually contains 15 to 30 frames per second.

Therefore, it is not suitable to solve the motion of the object that changes every frame by Lagrangian polynomial interpolation or Newton's interpolation. Because interpolating the given data using Lagrangian polynomial interpolation or Newton's method will increase the computational complexity and introduce more errors as the order increases.

In the case of (b) having a higher degree than in (a) of FIG. 2, the vibration occurs between 0 and 2 around the first data on the x-axis and 8 and 10 near the last data on the x-axis to interpolate between the sections. In this case, it can be seen that a large error occurs. Due to the uncertain location information of the moving object, data modeling, query processing, indexing, and inaccuracy of query results occur. The accuracy of the interpolation method is not proportional to the order of the polynomial used, and when interpolating between points in a straight line, a large error may occur and a more effective position estimation method is needed.

In order to solve this problem, in the present invention, when the degree of the polynomial is large, the given section is divided into several subsections, and the points between the points can be represented by a flexible curve using a lower order polynomial for each subsection. Using cubic spline interpolation, we also propose a method that can calculate video semantic annotation in a more effective way by dynamically calculating the sampling period.

Hereinafter, a method of expressing meta information for merchandise and personalization in TV content in a smart TV environment will be described in detail. This can be done for each step by the system for merchandise and personalization on TV content.

The following cubic spline interpolation is a known mathematical concept, which is used to correct (movement object tracking) coordinate values of moving objects.

1. How to estimate the position of a moving object

Cubic spline interpolation represents a set of data points as a polynomial curve that allows you to draw a flexible curve that is not subject to error through a given point.

A position corresponding to a specific viewpoint of the moving object in the video may be expressed as two-dimensional data having a viewpoint t and coordinate values x and y in the form of {time, position data}.

A moving object is referred to as O and two-dimensional positional data of O is represented by (x, y). Since the position data of the object O changes with time, the position data (x _k , y _k ) of the moving object O corresponding to a specific time point t _k may be obtained by adding a time value t. When the position data of the x and y axes of the moving object O are expressed with respect to the time axis, respectively, it may be represented as (t, x) for the x axis and (t, y) for the y axis. Thus, two separate third-order polynomials are needed to find the values of x and y to find the actual position coordinates.

이 있을 때, n개의 부분구간 3차 다항식을 구함으로써, 연속적으로 존재하는 모든 구간에서의 추정된 위치 좌표 값을 구할 수 있다. 이때, m은 (x , y) 좌표 값을 의미하며, 이동 객체가 나타나는 전체 유효 시간 간격이

이면, 여기에서 t₀는 저장된 유효 시간의 시작 시점을 의미하고 t_n은 마지막 시점을 나타낸다.

는

과 같이 n개의 부분 시간 구간

로 분할할 수 있으며, 이때 {k=0, 1, …, n-1}이 된다. 임의의 시점에 대한 객체 위치는 질의 시점 t_j에 대해,

의 조건을 만족할 때 t_j에서의 위치 좌표 값 m_j를 구하는 부분구간 3차 스플라인 보간 다항식은

의 네 점을 이용하여 구한다. 실제 객체의 위치 좌표는 x와 y의 값을 구하기 위한 개별적인 두 개의 3차 다항식으로, x좌표

, y좌표

를 이용하여 계산한다.3, the set consisting of n + 1 points

In this case, by estimating the third order polynomial of the n subdivisions, the estimated position coordinate values in all consecutive sections can be obtained. In this case, m means a (x, y) coordinate value, and the overall effective time interval in which the moving object appears

Where t ₀ represents the start time of the stored valid time and t _n represents the last time.

The

N partial time intervals, such as

Divided by {k = 0, 1,... , n-1}. The object position for any point in time is relative to the query point in time t _j ,

The quadratic spline interpolation polynomial of the partial interval to find the position coordinate value m _j at t _j when

Find four points using. The position coordinates of the actual object are two separate cubic polynomials for calculating the values of x and y.

, y coordinate

Calculate using

As the third-order polynomial of each partial section for the estimation position calculation, the interpolation equation S (t) may be expressed by Equation 1 below.

Equation 1 is a function for obtaining the x-coordinate value, and X _i (t _p ) and a function Y _i (t _p ) for obtaining the y-coordinate value are represented by equations (2) and (3).

Since 4 undetermined coefficients a _i , b _i , c _i , and d _i must be determined for each partial interval, 4n conditional expressions are required for n partial intervals. The conditions that can be known to determine these unknown coefficients are as follows. The cubic polynomial must pass both ends of the partial interval (2n conditions).

Condition 1:

Condition 2:

The third-order polynomials of each subinterval must have the same first derivative (tilt) and second derivative (curvature) at both ends they meet (2n-2 conditions).

Condition 3:

Condition 4:

Since Equation 1 passes two points, the result of substituting two points is expressed as Equation 4 and Equation 5.

The first and second derivatives of Equations 4 and 5 are expressed as Equations 6 and 7.

From the above equations, the undetermined coefficients a _i , b _i , c _i , d _i can be obtained using Equations 2 and 3, respectively, and can be summarized as in Equation 8.

의 관계에서 공통으로 갖는 한 점(t_i,m_i)의 미분한 값이 동일하다는 조건을 이용하여 수학식 6에 (t_i,m_i)의 값을 대입해 보면 수학식 9와 같이 표현된다.now,

Substituting the value of (t _i , m _i ) into Equation 6 using the condition that the derivatives of one point (t _i , m _i ) that are common in the relationship of is equal to Equation 9 .

의 관계에서 주어진 1차 미분값에 (t_i,m_i)값을 대입하면 수학식 10과 같이 표현된다.

Substituting the value of (t _i , m _i ) into the first derivative given in relation to, is expressed as

To simplify this, Equation 11 is obtained.

In order to determine 4n undecided coefficients, 4n conditions are required, but the number of conditions obtained through conditions 1, 2, 3, and 4 is 4n-2 and Equation 11 is i = 1 to i = n-1. It can be used for the points of. Therefore, the values of S ₀ and S _n are not known and two conditions are insufficient. The condition for solving this is called the endpoint condition. The shape is slightly different depending on the method for obtaining the end point condition. In the present invention, an end point condition ending with a parabola is defined as in Equation (12).

Now S ₁ , S ₂ ,.. , Since the values of S _n can be known, and after obtaining four coefficients by substituting Equation 8 and substituting each coefficient into Equation 2 and Equation 3, the position coordinates x and y at the time point t can be obtained. .

을 만족하는 추정 함수는 수학식 2, 수학식 3을 이용하여 구한다. 먼저 수학식 2를 이용하여 X_i(t_p)를 구할 때, 각 구간에 대한 곡선을 구하기 위한 미정 계수들은 수학식 8을 이용하여 표 2와 같이 구할 수 있다. 임의의 시점 t_p=1.2에 대한 X(1.2)는 [t₁, t₂]구간에 해당하므로 값 1.69를 수학식 2로부터 얻을 수 있다.For example, suppose that in a video with a playback time of 10 seconds, annotated position data (Table 1) of a moving object stored at intervals of 1 second. At this time, the position data x, y for the unstored time point t _p is

An estimation function that satisfies is calculated using Equation 2 and Equation 3. First, when X _i (t _p ) is obtained by using Equation 2, undetermined coefficients for obtaining a curve for each section may be obtained as shown in Table 2 by using Equation 8. Since X (1.2) for an arbitrary time point t _p = 1.2 corresponds to the section [t ₁ , t ₂ ], a value 1.69 can be obtained from Equation 2.

을 만족할 때, 4개의 미정 계수를 구하는 방법은 X_i(t_p)를 구할 때와 동일한 수학식 8을 이용하여 구한다. 임의의 시점 t_p=1.2에 대한 Y(1.2)는 [t₁, t₂]구간에 해당하므로 값 3.84를 수학식 2로부터 얻을 수 있다.at t _p to obtain the y coordinate value of the function Y _i (t _p) is seeking to use the equation (3)

When 4 is satisfied, the method of obtaining four undecided coefficients is obtained by using the same Equation 8 as when X _i (t _p ) is obtained. Since Y (1.2) for an arbitrary time point t _p = 1.2 corresponds to the section [t ₁ , t ₂ ], the value 3.84 can be obtained from Equation 2.

In the present invention, as well as the above-described cubic spline interpolation method, any algorithm capable of interpolating and expressing uncertain position data between points in any two position data can be applied.

2. How to determine the dynamic sampling period

In interpolating the movement of a moving object using the cubic spline method, the moving speed of the objects is a very sensitive problem. That is, the period of sampling should be short to interpolate the coordinates of the moving object moving at high speed, and the sampling period need not be fast to interpolate the coordinates of the moving object moving at slow speed. This is because a shorter period of sampling means that the amount of data increases by that amount, and a longer period of sampling means that the amount of data decreases by that amount. The present invention proposes a method for performing dynamic sampling based on moving speeds of objects.

FIG. 4 shows that the sampling period is determined according to the standard deviation of the object position coordinates collected for 30 frames and the value exceeding the threshold.

In the system for merchandise and personalization of TV content, the relative coordinate change amount representing the moving speed of the object moving according to the frame change in the video data can be obtained, and then the dynamic sampling cycle for the position of the object can be determined according to the relative coordinate change amount. . In this case, as the relative coordinate change amount, a standard deviation, relative standard deviation, Euclidean distance, and the like may be used.

In this case, in a system for merchandise and personalization in TV content, a threshold condition for each of a plurality of sampling periods and sampling periods having different frame periods is defined, and corresponds to a standard deviation of the position coordinates of an object among the plurality of sampling periods. The frame period of the threshold condition can be applied. In this case, a shorter sampling period may be determined as the standard deviation with respect to the position coordinates of the object increases.

For example, if the threshold x∈ [0,5] is satisfied, sampling is performed in 15 frame periods, and if the threshold x∈ [5,10] is satisfied, sampling is performed in 7 frame periods. Can be. That is, a difference of more than 10 pixels in the coordinate position between frames means that the movement of the moving object is fast, and therefore, the sampling period should be set shorter. 4 is an example of an X coordinate value according to a change of a frame, and this process may be similarly applied to a Y coordinate to determine a sampling period.

In this embodiment, by using the method of determining the dynamic sampling period using the standard deviation, the coordinates of the objects can be stored more effectively and the coordinates of the accurate moving object can be interpolated.

3. Ontology Structure for Product Information and Personalization

는 클래스를 의미하며,

는 클래스와 클래스간의 관계 정보를 의미한다.FIG. 5 is a diagram illustrating a hierarchical structure and object attribute information of an ontology (LOD) class for merchandise and personalization. 5,

Means class,

Means class and relationship information between classes.

For example, the relationship between the object class and the motion class is defined as 'hasMotion'. This relationship means that the object class has a class called motion. More specifically, an object has a motion value, that is, a description of how the object moves is defined through a data property of a class called motion.

In another example, an object class has a Shape class. That is, the definition of the object type is expressed using the shape class. In this case, when the data attribute value of the shape class is used, various types of objects may be represented.

As another example, an object class can be set up to represent shapes only for users suitable for the target class. That is, the part displaying information on the shape or motion of the currently expressed object is determined by the data attribute value of the target class.

In other words, the information provider (Seller) expresses the information about the product using the classes of Motion, Shape, Event, and Playtime, and the viewers who will see the product information That is, the information about the target user is set, and only the user who matches the information can display the product.

3-1. Ontological target ( Target ) Class data attribute ( Data property ) Information

The target class is a class that can be expressed so that the target user who receives the metadata provided by the information provider can be set. The data attribute value of the target class is shown in FIG. 6. Here, 'ID' is an attribute for identifying an instance of the target class, 'Age' is an attribute for the age of the target user, 'Gender' is an attribute for the gender of the target user, and 'Location' is a target located in a specific region. Attributes for the user, 'Interests' are attributes for expressing the target user's disposition,' Groups' are attributes for the group to which the information provider belongs or all of the information providers' friends, and 'Relationships' is the relationship between the target user and the information provider. Alternatively, an attribute corresponding to the current state of the target user, 'Education' may describe information on the current attribute value for the target user. Table 2 shows an example of data attributes of the target class.

3-2. Ontology playtime ( PlayTime ) Class data attribute ( Data property ) Information

The playtime class is a class for displaying information on when information on an object annotated by the information provider is displayed on the screen. The data attribute value of the playtime class is shown in FIG. Here, 'ID' is an attribute that can distinguish playtime, and 'Type' is a value for determining values of 'StartValue' and 'EndValue', and may be selected from Frame or Time. 'StartValue' refers to the start point of the current object. If 'Type' is a frame, it is the number of the corresponding frame. If 'Type' is a time, it is expressed as 00:00:00. 'EndValue' refers to the end point of the current object. If 'Type' is a frame, it is the number of the corresponding frame. If 'Type' is a time, it is expressed as 00:00:00. Table 3 shows an example of data attributes of the playtime class.

3-3. Ontological Shape ( Shape ) Class data attribute ( Data property ) Information

The shape class is a class for displaying information about an object annotated by the information provider on the screen. The data attribute value of the shape class is shown in FIG. 8. Here, 'ID' is a property that can distinguish shapes, 'Type' is a shape such as image, text, video, label, circle, and ' 'Width' is the width of the object, 'Height' is the height of the object, 'Text' is the character to display on the screen, 'Alpha' is the transparency value of 0-100, and 'URL' Is an attribute for linking information on the web like zero image video, and 'color' is an attribute for the color of an object. Table 4 shows an example of the data attributes of the shape class.

3-4. Ontology event ( Event ) Class data attribute ( Data property ) Information

An event class is a class for defining the objects and interactions annotated by the information provider. The data attribute value of the event class is shown in FIG. Here, 'ID' is a property that can distinguish the event, 'Type' is the type of interaction, for example, events such as onClick, onMouseOver, onKeyPress, 'Action' is an example of the information when the interaction occurs For example, a property of displaying a web page such as a shopping mall site on the entire screen or specifying a split of a screen, etc., and a 'URL' refers to an address of a shopping mall provided at the time of interaction. Table 5 shows an example of data attributes of an event class.

3-5. Ontological motion ( Motion ) Class data attribute ( Data property ) Information

The motion class is a class for representing the movement of an object annotated by the information provider. The data attribute value of the motion class is shown in FIG. Here, 'ID' is a property that can distinguish motion, 'InterpolationType' is a type property of an algorithm for interpolating a moving object, 'SampleRate' is a property of sampling period, and 'TrajectoryType' is an object information and location information. As an attribute, it may be the same as position or coefficient, and 'Trajectories' has a value of (x, y) if the coordinate is according to 'TrajectoryType' and a predetermined coefficient value if the coefficient. Table 6 shows an example of data attributes of a motion class.

The following is an example of the use for merchandise and personalization of TV content in a smart TV environment.

This embodiment may operate in the environment of FIG. Information providers want to provide information such as Annotator or Seller to annotate the information on moving moving objects in the video content. Annotated information is created based on product and personalization ontology, which is stored separately from the content for interaction with video content. After the annotated information is stored, video and interactive contents are synthesized together by a user who wants to receive information such as a vierwer or buyer, and then synthesized and played by a player such as a semantic video player.

Synthesis is the same as FIG. In order to express and augment the semantic annotation of the moving object in the personalized interactive video, as shown in FIG. 12, the video content server and the interactive content server may be operated separately. That is, a system for merchandise and personalization of TV content may be composed of a video content server for maintaining video content and an interactive content server for maintaining annotated meta information about the video content.

In this case, the video content server stores video content generated by a user, and may also include an address of a video existing on another web site. In the present embodiment, the video content server may apply the above-described dynamic sampling period determination method in storing the position coordinates of the object in the video data. In addition, the interactive content server includes personalization information generated by collaboration of various users, and the personalization information (ie, metadata information) may be augmented and provided on the video content at the request of users later.

In other words, in the user device of the target user, video content provided from the video content server and metadata information provided from the interactive content server are synthesized and provided, and the metadata information is provided to the user who received the corresponding content. In particular, it is possible to provide a personalized service by providing information to a target user set by the information provider providing the interest or information. For example, suppose that an information provider selling gloves and an information provider selling shoes input their product information and personalization information into the video content of FIG. 13 as shown in Table 7. Such video content and meta information are provided to viewers (ie, users), and personalized information provided by the information provider by analyzing the user's personal profile (gender, age, school information, etc.) information and the user's friend profile information as shown in FIG. By comparing the personal profile of the user and the user can provide the product information to the users that the information provider has determined as the target. Accordingly, the user may be provided with information annotated by the information provider through various video views (eg, smart TVs) and user views (eg, smart phones, tablets, etc.). Tables 8 and 9 show examples of profiles for two users.

<Object rdf: ID = "Object_001">
<hasID rdf: datatype = "&xsd;literal"> 1 </ hasID>
<hasName rdf: datatype = "&xsd;literal"> Glove </ hasName>
<OnlyRepresented rdf: resource = "# Target_001"/>
<hasPlayTime rdf: resource = "# PlayTime_001"/>
<hasShape rdf: resource = "# Shape_001"/>
<hasMotion rdf: resource = "# Motion_001"/>
<appearsIn rdf: resource = "# Shot_001"/>
<hasEvent rdf: resource = "# Event_001"/>
</ Object>

<Object rdf: ID = "Object_002">
<hasID rdf: datatype = "&xsd;literal"> 1 </ hasID>
<hasName rdf: datatype = "&xsd;literal"> Shoes </ hasName>
<OnlyRepresented rdf: resource = "# Target_002"/>
<hasPlayTime rdf: resource = "# PlayTime_001"/>
<hasShape rdf: resource = "# Shape_002"/>
<hasMotion rdf: resource = "# Motion_002"/>
<appearsIn rdf: resource = "# Shot_001"/>
<hasEvent rdf: resource = "# Event_002"/>
</ Object>

<Target rdf: ID = "Target_001">
<hasID rdf: datatype = "&xsd;literal"> 1 </ hasID>
<hasAge rdf: datatype = "&xsd;literal"> 10-20 </ hasAge>
<hasGender rdf: datatype = "&xsd;literal"> All </ hasGender>
<hasEducation rdf: datatype = "&xsd;literal"> at secondary school </ hasEducation>
</ Target>

<Target rdf: ID = "Target_002">
<hasID rdf: datatype = "&xsd;literal"> 2 </ hasID>
<hasAge rdf: datatype = "&xsd;literal"> 20-30 </ hasAge>
<hasGender rdf: datatype = "&xsd;literal"> man </ hasGender>
<hasEducation rdf: datatype = "&xsd;literal"> university </ hasEducation>
</ Target>

<PlayTime rdf: ID = "PlayTime_001">
<hasID rdf: datatype = "&xsd;literal"> 1 </ hasID>
<hasType rdf: datatype = "&xsd;literal"> frame </ hasType>
<hasStartValue rdf: datatype = "&xsd;literal"> 500 </ hasStart>
<hasEndValue rdf: datatype = "&xsd;literal"> 549 </ hasEnd>
</ PlatTime>

<Shape rdf: ID = "Shape_001">
<hasID rdf: datatype = "&xsd;literal"> 1 </ hasID>
<hasType rdf: datatype = "&xsd;literal"> image </ hasType>
<hasWidth rdf: datatype = "&xsd;literal"> 100 </ hasWidth>
<hasHeight rdf: datatype = "&xsd;literal"> 70 </ hasHeight>
<hasUrl rdf: datatype = "&xsd;literal"> http://www.ebay.com/Glove_Image.jpg </ hasUrl>
</ Shape>

<Shape rdf: ID = "Shape_002">
<hasID rdf: datatype = "&xsd;literal"> 2 </ hasID>
<hasType rdf: datatype = "&xsd;literal"> image </ hasType>
<hasWidth rdf: datatype = "&xsd;literal"> 100 </ hasWidth>
<hasHeight rdf: datatype = "&xsd;literal"> 70 </ hasHeight>
<hasUrl rdf: datatype = "&xsd;literal"> http://www.ebay.com/Shoes_Image.jpg </ hasUrl>
</ Shape>

<Event rdf: ID = "Event_001">
<hasID rdf: datatype = "&xsd;literal"> 1 </ hasID>
<hasType rdf: datatype = "&xsd;literal"> onClick </ hasType>
<action rdf: datatype = "&xsd;literal"> popup </ hasType>
<hasUrl rdf: datatype = "&xsd;literal"> http://www.ebay.com/Glove_wesite.html </ hasUrl>
</ Event>

<Event rdf: ID = "Event_002">
<hasID rdf: datatype = "&xsd;literal"> 2 </ hasID>
<hasType rdf: datatype = "&xsd;literal"> onClick </ hasType>
<action rdf: datatype = "&xsd;literal"> popup </ hasType>
<hasUrl rdf: datatype = "&xsd;literal"> http://www.ebay.com/Shoes_wesite.html </ hasUrl>
</ Event>


<Motion rdf: ID = "Motion_001">
<hasID rdf: datatype = "&xsd;literal"> 1 </ hasID>
<hasType rdf: datatype = "&xsd;literal"> moving </ hasType>
<interpolationType rdf: datatype = "&xsd;literal"> cubic spline </ interpolationType>
<samplingRage rdf: datatype = "&xsd;literal"> 7 </ samplingRage>
<trajectoryType rdf: datatype = "&xsd;literal"> position </ trajectoryType>
<trajectories rdf: datatype = "&xsd;literal"> 134 247 138 249 140 253 135 260 139 270 138 250 135 247 </ samplingRage>
</ Motion>

<Motion rdf: ID = "Motion_002">
<hasID rdf: datatype = "&xsd;literal"> 2 </ hasID>
<hasType rdf: datatype = "&xsd;literal"> moving </ hasType>
<interpolationType rdf: datatype = "&xsd;literal"> cubic spline </ interpolationType>
<samplingRage rdf: datatype = "&xsd;literal"> 7 </ samplingRage>
<trajectoryType rdf: datatype = "&xsd;literal"> position </ trajectoryType>
<trajectories rdf: datatype = "&xsd;literal"> 234 347 238 349 240 353 235 360 239 370 238 350 235 347 </ samplingRage>
</ Motion>

As described above, according to the present exemplary embodiment, by using cubic spline interpolation to express uncertain position data of a moving object of TV content and applying a dynamic sampling period according to the moving speed of the moving object, the position information of the moving object can be accurately corrected. At the same time, the amount of data can be effectively reduced to effectively express meta information for personalization.

The methods according to embodiments of the present invention may be implemented in the form of a program instruction that can be executed through various computer systems and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. In addition, the above-described file system can be recorded in a computer-readable recording medium.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above embodiments, and those skilled in the art to which the present invention pertains various modifications and variations from such descriptions. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

Claims (9)

In how to represent moving objects in a video,
Obtaining a relative coordinate change amount representing a moving speed of an object moving according to a frame change in the video data;
Determining a dynamic sampling period for the position coordinate of the object according to the relative coordinate change amount; And
Sampling the video data at the dynamic sampling period and storing position coordinates of the object
Method of representing a moving object in a video comprising a. The method of claim 1,
Determining the dynamic sampling period,
Determining a dynamic sampling period for the X coordinate of the object according to the relative coordinate change amount of the X coordinate according to the frame change;
Determining a dynamic sampling period for the Y coordinate of the object according to the relative coordinate variation of the Y coordinate according to the frame change
Method of representing a moving object in a video comprising a. The method of claim 1,
Determining the dynamic sampling period,
A plurality of sampling periods having different frame periods and threshold conditions for each of the sampling periods are defined so that the sampling period is dynamically determined according to the threshold condition to which the relative coordinate change amount corresponds.
How to represent the moving object in the video, characterized in that. The method of claim 3,
Determining the dynamic sampling period,
The larger the amount of change with respect to the position coordinates of the object, the shorter sampling period is determined among the plurality of sampling periods.
How to represent the moving object in the video, characterized in that. The method of claim 1,
Estimating the position coordinates of the object with respect to an arbitrary time point using an algorithm for interpolating uncertain position data between points in any two position data with respect to the position coordinates of the object stored in the dynamic sampling period.
The method of representing a moving object in the video further comprising. The method of claim 1,
Annotating meta information associated with the object in the position coordinates of the object
Further comprising:
The meta information is displayed by being synthesized with the video data on a screen on which the video data is reproduced.
How to represent the moving object in the video, characterized in that. The method according to claim 6,
The meta information has an ontology class hierarchy structure based on LOD (Linking Open Data),
A target class for setting a target user to provide the meta information, a PlayTime class for setting a time point at which the meta information is displayed on a screen on which the video data is played, and a form of the meta information. Shape class for setting the, Event class for defining the interaction environment for the meta information, Motion for setting the interpolation condition for the position coordinates of the object where the meta information is annotated Containing at least one of the Motion classes
How to represent the moving object in the video, characterized in that. The method of claim 7, wherein
The meta information is provided as personalized information for a target user set in the target class among users who receive the video data.
How to represent the moving object in the video, characterized in that. In the system for personalization of video content,
A video content server for holding the video content; And
Interactive content server that maintains annotated meta information about the video content
Lt; / RTI &gt;
The video content server,
After obtaining a relative coordinate change amount representing a moving speed of an object moving according to a frame change in the video content, a dynamic sampling period for the position coordinate of the object is determined according to the relative coordinate change amount, and the video content is used as the dynamic sampling period. Sampling to store the position coordinates of the object,
The meta information is,
Being synthesized with the video content and displayed on a screen on which the video content is played by annotating the position coordinates of the object.
Video content personalization system.

Images