KR101772066B1 - Real time consolidated data mapping apparatus and method for product coordinate tracking data in image of multiple users - Google Patents

Abstract

An apparatus and method for real-time integrated data mapping of commodity coordinate tracking data in multi-user image contents is disclosed. A web interface providing module for providing a web interface for tracking products in video contents to a user terminal; A commodity coordinate tracking request receiving module that receives a commodity coordinate tracking request from a user terminal for a commodity designated by a user among commodities appearing in image content reproduced and played back on the web interface provided in the web interface providing module; A commodity coordinate automatic tracking module for automatically tracking the commodity coordinate tracking requested by the commodity coordinate tracking request receiving module to provide automatic tracking coordinates to the user terminal in real time; Receiving the correction coordinates corrected by the user's touch input with respect to the automatic tracking coordinates on the web interface of the user terminal from the user terminal and applying the received correction coordinates to the automatic tracking coordinates to generate final product coordinates And forms a final product coordinate creation module.

Description

Field of the Invention < RTI ID = 0.0 > [0001] < / RTI > A real-time integrated data mapping apparatus and method for multi-

More particularly, the present invention relates to an apparatus and method for real-time integrated data mapping of product coordinate tracking data in image contents of multiple users.

In recent years, a product placement (PPL) advertisement method, which advertises a product through a drama or a movie, has been used as a major advertising method.

PPL advertising is an advertising method that exposes products to video contents such as movies and dramas, and in many cases leads to an increase in merchandise sales.

However, since the PPL advertisement product is often exposed naturally in the video content, there are cases where it can not be discriminated whether or not the product is actually a PPL advertisement product. In view of the advertisers paying a large amount of advertisements, it is necessary to increase the advertising effect by directly or indirectly recognizing the PPL advertisement product as an advertisement product.

However, even if the PPL advertisement product is displayed in the moving picture contents, it is difficult to mark the PPL advertisement product whose position changes every frame. There is a problem that it requires a great deal of manual work to mark the PPL advertisement product moving every frame every frame.

Meanwhile, since the PPL advertisement product information is conventionally collected and uploaded from the provider side, there is a problem that a considerable difficulty is required in the workload in the collection of the PPL advertisement product information and the data processing.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a real-time integrated data mapping device for multi-user product coordinate tracking data in image contents.

Another object of the present invention is to provide a real-time integrated data mapping method for product coordinate tracking data in multi-user image contents.

The present invention provides a real-time integrated data mapping device for multi-user image coordinate tracking data in a multi-user, comprising: a web interface providing module for providing a web interface for tracking products in image contents to a user terminal; ; A commodity coordinate tracking request receiving module that receives a commodity coordinate tracking request from a user terminal for a commodity designated by a user among commodities appearing in image content reproduced and played back on the web interface provided in the web interface providing module; A commodity coordinate automatic tracking module for automatically tracking the commodity coordinate tracking requested by the commodity coordinate tracking request receiving module to provide automatic tracking coordinates to the user terminal in real time; Receiving the correction coordinates corrected by the user's touch input with respect to the automatic tracking coordinates on the web interface of the user terminal from the user terminal and applying the received correction coordinates to the automatic tracking coordinates to generate final product coordinates And a final product coordinate creation module.

The product coordinate storage module may further include a product coordinate storage module for storing the final product coordinate in association with the image content and the product.

And a merchandise information storage module for receiving and storing merchandise information on the merchandise from the user terminal.

On the other hand, when the image content is reproduced on the web interface of another user terminal, the commodity coordinate storing module searches for the commodity coordinates of the commodity of the commodity appearing in the video content to be reproduced, And may further comprise a provisioning module.

When the product coordinate information providing module is provided with the final product coordinates and the product information corresponding to the final product coordinates is requested by the other user terminal, the requested product information is retrieved from the product information storage module, And a product information providing module for providing the product information to another user terminal in real time.

On the other hand, the product coordinate automatic tracking module detects and specifies an object model through an object detection algorithm for a product touch-input by a user on the user terminal 10, extracts designated characteristic points, , And can be configured to track the tracing feature points using a pyramid LK algorithm, which is a sparst optical flow.

And the commodity coordinate automatic tracking module may be configured to combine the CamShift algorithm based on the histogram when the shape of the object model changes to correct the tracking.

The commodity coordinate automatic tracking module estimates a motion of the object model by applying a ransan algorithm based on the tracking characteristic points and updates the object model through a similarity transformation algorithm through estimation .

The commodity coordinate automatic tracking module may be configured to perform object detection by estimating a motion of the object model by applying a Kalman filter when the object model is completely displayed again when the updated object model is partially obscured .

In another aspect of the present invention, there is provided a method for real-time integrated data mapping of product coordinate tracking data in a multi-user image content, comprising: providing a web interface for tracking a product in an image content to a user terminal ; When the image content is reproduced on the web interface provided in the web interface providing module, the commodity coordinate tracking request receiving module requests the commodity coordinate tracking from the user terminal to the commodity designated by the user among the commodities appearing in the reproduced video content Receiving; The product coordinate automatic tracking module performs an automatic tracking on the product coordinate tracking requested by the product coordinate tracking request receiving module to provide automatic tracking coordinates to the user terminal in real time; The final product coordinate creation module receiving from the user terminal correction coordinates corrected by the user's touch input with respect to the automatic tracking coordinate on the web interface of the user terminal; And the final product coordinate creation module may apply the received correction coordinates to the automatic tracking coordinates to generate final product coordinates.

In this case, the product coordinate storage module may be configured to store the final product coordinates in association with the image content and the product.

And a product information storage module receiving and storing product information on the goods from the user terminal.

On the other hand, when the image content is reproduced on the web interface of another user terminal, the commodity coordinate providing module searches the commodity coordinate storing module for the final commodity coordinates for the commodities appearing in the reproduced video contents, And providing the real time information.

When the commodity coordinate providing module is provided with the final commodity coordinates and the commodity information for the commodity corresponding to the final commodity coordinates is requested by the other user terminal, the commodity information providing module transmits the requested commodity information to the commodity information Retrieving from the storage module and providing the selected user terminal to the other user terminal in real time.

Here, the step of automatically tracking the product coordinate tracking requested by the product coordinate automatic tracking module by the product coordinate tracking request receiving module and providing the automatic tracking coordinate to the user terminal in real time may include: The object model is detected and designated by the object detection algorithm for the product that is touched by the user, the specified feature points are extracted and registered as tracking feature points, and the pyramid LK algorithm, which is a sparst optical flow, To track the tracking feature points.

The step of automatically tracking the product coordinate tracking requested by the product coordinate automatic tracking module by the product coordinate tracking request receiving module and providing the automatic tracking coordinate to the user terminal in real time may include: And can be configured to combine the histogram-based CamShift algorithm to correct tracing.

The step of automatically tracking the product coordinate tracking requested by the product coordinate automatic tracking module by the product coordinate automatic tracking module and providing the automatic tracking coordinate to the user terminal in real time comprises: ) Algorithm to estimate the motion of the object model and to update the object model through a similarity transformation algorithm through estimation.

The step of providing the automatic tracking coordinates to the user terminal by performing the automatic tracking on the tracking of the product coordinates requested by the product coordinate tracking request receiving module by the product coordinate automatic tracking module, And when the object model is completely hidden, the Kalman filter is applied to estimate the motion of the object model to perform object detection.

According to the above-described real-time integrated data mapping device and method for tracking merchandise coordinate tracking data within a multi-user image content, when a user views the image content and his / her own advertising product appears, So that it is possible to advertise through the image contents and sell the product at any time even if the advertiser does not place the advertisement directly.

In the present invention, the advertisement information and the product information are collectively collected and uploaded. However, in the present invention, since the advertiser can directly upload the product information and modify the product coordinate tracking information, the advertisement product can be easily and conveniently provided There is an effect that can be.

In the conventional case, there is a disadvantage in that the time and cost for registering a product are very much consumed, which makes it difficult to secure contents.

In order to solve this problem, various interfaces are provided and multi-user product registration is supported, so that the time for registering the product in the image is greatly reduced by being processed in the cloud simultaneously. This means that it is possible to quickly process a product registration for one or a plurality of images within a short period of time, and it is possible to quickly gather a large amount of contents in a short period of time and quickly form a market.

According to the above-described real-time integrated data mapping device and method for tracking merchandise coordinate tracking data within a multi-user image content, when a user views the image content and his / her own advertising product appears, So that it is possible to advertise through the image contents and sell the product at any time even if the advertiser does not place the advertisement directly.
In the present invention, the advertisement information and the product information are collectively collected and uploaded. However, in the present invention, since the advertiser can directly upload the product information and modify the product coordinate tracking information, the advertisement product can be easily and conveniently provided There is an effect that can be.
In the conventional case, there is a disadvantage in that the time and cost for registering a product are very much consumed, which makes it difficult to secure contents.
In order to solve this problem, various interfaces are provided and multi-user product registration is supported, so that the time for registering the product in the image is greatly reduced by being processed in the cloud simultaneously. This means that it is possible to quickly process a product registration for one or a plurality of images within a short period of time, and it is possible to quickly gather a large amount of contents in a short period of time and quickly form a market.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail to the concrete inventive concept. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

The terms first, second, A, B, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a schematic diagram of a real-time integrated data mapping apparatus for multi-user product coordinate tracking data in image contents according to an embodiment of the present invention.

Referring to FIG. 1, a real-time integrated data mapping device (hereinafter, referred to as a 'real-time integrated data mapping device') 100 for tracking multi-user product coordinate tracking data in multi- The user is allowed to upload product information about his / her sales product or manufactured product while viewing the video content directly, thereby enhancing the convenience of product advertisement.

The user can request automatic coordinate tracking for a specific product while watching a moving picture on the web interface provided in the present invention. The real-time integrated data mapping device 100 tracks the coordinate of the corresponding product, And is configured to provide product coordinates to other users. The user uploads product information for the product in real time, and the real-time integrated data mapping device 100 is configured to provide the product information to other users.

Unlike the conventional method, the indirect product on the image contents can be specified by the users during the viewing, and the product information can be uploaded to the user so that the product coordinates and the product information can be easily and conveniently collected at any time.

FIG. 2 is a block diagram of a real-time integrated data mapping apparatus for tracking product coordinate data in multi-user image contents according to an exemplary embodiment of the present invention.

The detailed configuration will be described with reference to Fig.

The web interface providing module 110 may be configured to provide a web interface for tracking the goods in the image contents to the user terminal 10. [

Figure 3 illustrates a web interface in accordance with the present invention. As shown in FIG. 3, the image content provided by the OTT provider or the like can be reproduced on the web interface, and the product coordinates on the image content can be automatically tracked and marked. If the product is selected by the user, the product information for the product can be retrieved from the right side.

This web interface can be configured to work in conjunction with the real-time integrated data mapping apparatus 100.

The image content is reproduced on the web interface provided by the web interface providing module 110 and the commodity coordinate tracking request receiving module 120 receives the commodity coordinate tracking request from the user terminal 10 May be configured to be requested to track merchandise coordinates. The user can designate a product through touch input on the web interface of the user terminal 10. [

The commodity coordinate automatic tracking module 130 may be configured to automatically track the commodity coordinate tracking requested by the commodity coordinate tracking request receiving module 120 to provide the automatic tracking coordinates to the user terminal 10 in real time.

The user terminal 10 receives the automatic tracking coordinates and displays them on the web interface in real time. If there is an error in the automatic tracking, the user directly inputs correction coordinates for correcting the coordinates through touch input or the like, (140).

The final commodity coordinate generation module 140 may be configured to receive from the user terminal 10 the corrected coordinates corrected by the user's touch input with respect to the automatic tracking coordinates on the web interface of the user terminal 10. [

Product commodity coordinate generation module 140 may be configured to apply the correction coordinates received from the user terminal 10 prior to the automatic tracking coordinates to generate final commodity coordinates.

The commodity coordinate storage module 150 may be configured to store final commodity coordinates in association with image contents and commodities. Such final product coordinates may be reused at any time for the image content.

The product information storage module 160 may be configured to receive and store product information on a product from the user terminal 10. [ Such product information may be information such as a merchandise retailer or a merchandise selling site, a price, a quantity, and a type.

When the image content is reproduced on the web interface of another user terminal 20, the product coordinate providing module 170 searches the product coordinate storing module 150 for the final product coordinates of the product appearing in the reproduced image content To another user terminal 20 in real time. Other user terminals 20 may be configured to receive and display the final merchandise coordinates on the image content.

When the product information providing module 180 is provided with the final product coordinates in the product coordinate providing module 170 and is requested by the other user terminal 20 for product information corresponding to the final product coordinates, Information may be retrieved from the product information storage module 160 and provided to another user terminal 20 in real time. The other user terminals 20 can be configured to inquire the commodity information on the web interface and purchase the commodity information by connecting to the purchasing site.

FIG. 4 is a block diagram of an operation of a real-time integrated data mapping apparatus for multi-user product coordinate tracking data in an image content according to an exemplary embodiment of the present invention.

Fig. 4A shows a configuration for automatically recognizing a scene change scene. First, image analysis is performed to extract points where the screen is switched. And the screen switching scene area is separated according to the screen switching scene point.

FIG. 4B illustrates the automatic tracking function of the product coordinates. When a point is set in the still image, the product coordinates are automatically tracked according to the amount of change of the image in the video section of the corresponding scene.

FIG. 4C shows coordinate correction in a drag-and-drop manner. The product coordinates are automatically tracked and then the coordinates are finely corrected by a drag and drop method in order to precisely correct the coordinates.

5 is a flowchart illustrating a method of real-time integrated data mapping for product coordinate tracking data in multi-user image contents according to an exemplary embodiment of the present invention.

Referring to FIG. 5, the web interface providing module 110 provides a web interface to the user terminal 10 for tracking goods in the image content (S101).

Next, when the image content is reproduced on the web interface provided by the web interface providing module 110, the commodity coordinate tracking request receiving module 120 transmits, to the user designated by the user, (Step S102).

Next, the commodity coordinate automatic tracking module 130 automatically tracks the commodity coordinate tracking requested by the commodity coordinate tracking request receiving module 120, and provides the automatic tracking coordinates to the user terminal 10 in real time (S103) .

Next, the final product coordinate creation module 140 receives from the user terminal 10 the correction coordinates corrected by the user's touch input with respect to the automatic tracking coordinates on the web interface of the user terminal 10 (S104).

Next, the final product coordinate creation module 140 applies the received correction coordinates to the automatic tracking coordinates to generate final product coordinates (S105).

Next, the trademark coordinate storage module 150 stores the final product coordinate in association with the image content and the product (S106).

Next, the product information storage module 160 receives and stores the product information on the goods from the user terminal 10 (S107).

Next, when the image content is reproduced on the web interface of another user terminal 20, the commodity coordinate providing module 170 transmits the final commodity coordinates of the commodities appearing in the video content reproduced by the commodity coordinate providing module 170 to the commodity coordinate storing module 150 And provides it to another user terminal 20 in real time (S108).

Next, when the product coordinate providing module 180 is provided with the final product coordinates and the product information about a product corresponding to the final product coordinate is requested by another user terminal 20, the product information providing module 180 transmits a request The received product information is retrieved from the product information storage module 160 and provided to another user terminal 20 in real time (S109).

6 to 9 are detailed flowcharts of a real-time integrated data mapping method for product coordinate tracking data in multi-user image contents according to an embodiment of the present invention.

FIG. 6 shows a product registration process of a multi-user.

In FIG. 6, a user can register a product at any time during viewing of the video content, selects a scene to be registered, determines whether product information exists in the scene, and performs the automatic coordinate tracking or correction according to necessity. The user specifies a product by specifying a point for the product.

In Fig. 7, automatic coordinate tracking of goods is disclosed.

An object model is detected and specified through an object detection algorithm around a user's designated point for a product. Then, feature points of the object model region are extracted and registered.

Trace feature points are traced using a pyramidal LK algorithm, which is a sparse optical flow. At this time, rare optical flow may not perform well when the shape changes like a human, so a histogram-based camshift algorithm is combined to correct the result.

On the other hand, the motion of the object is estimated using the ransac algorithm based on the feature points. The motion of the object may be rotation, translation, size change, and so on.

Then, based on the estimated data of the motion, the object model is updated through a similarity transformation algorithm. Then, the feature points of the updated object model are extracted and registered as tracking feature points.

Here, if an object to be tracked is covered by a certain portion or more, an error may occur in tracking. And even if the object is completely reappeared later, it can be difficult to detect it. Then, the Kalman filter is applied by using the four vertexes of the extracted window, and the obtained data is used as correction information of the model obtained through similarity transformation every frame.

If tracking is no longer feasible, Kalman filter data can be used to estimate the motion of the object model. At this time, the surrounding area is inspected based on the estimated data, and when the object is reappearing, when the tracking characteristic points are detected, the above procedure is brightened and tracking of the characteristic point is continued.

On the other hand, if the object completely disappears from the screen, it is detected through the learned detection algorithm, and if the object appears again within a certain time, detection is successful. Otherwise, the model is updated by the similarity transformation algorithm to perform tracking do. If the object is not detected for a predetermined time, the tracking is terminated.

FIG. 8 shows a case where another user requests product information for stopping reproduction of an image while viewing the image content. The other user terminal 20 transmits the corresponding image information and current timeline information to the real-time integrated data mapping apparatus 100. The real-time integrated data mapping apparatus 100 transmits the product coordinates and product information to another user terminal 20, and the other user terminal 20 displays the same.

Figure 9 discloses tracking data.

First, each user's data consists of frame, coordinates, commodity code, and other information. These are uploaded to the real-time integrated data mapping device 100, each of which is stored by being combined with coordinates, product information, and other information, and the frame at a specific time point can be easily retrieved by the index. And, in the service state, it is stored in the memory DB to provide data to the other user terminals 20 quickly. Thereafter, such information is provided whenever there is a request from another user terminal 20. [

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention as defined in the following claims. There will be.

110: Web interface providing module
120: commodity coordinate tracking request receiving module
130: Commodity coordinate automatic tracking module
140: final product coordinate creation module
150: product information storage module
160: Commodity coordinate storage module
170: Product coordinate providing module
180: Product information providing module

Claims (1)

When the advertiser accidentally discovers the appearance of the advertiser's goods in the viewing image content while the advertiser is viewing the predetermined image content on the first user terminal, A web interface providing module for providing a web interface to the first user terminal,
Receiving a merchandise coordinate tracking request for requesting merchandise coordinate tracking from the first user terminal for a merchandise designated by a first user from among merchandise appearing in an image content reproduced and played back on the web interface provided by the web interface providing module module;
A commodity coordinate automatic tracking module for automatically tracking the commodity coordinate tracking requested by the commodity coordinate tracking request receiving module to provide automatic tracking coordinates to the first user terminal in real time;
Receiving from the first user terminal correction coordinates corrected by a touch input of a first user with respect to the automatic tracking coordinates on the web interface of the first user terminal and applying the received correction coordinates to the automatic tracking coordinates A final product coordinate creation module for creating final product coordinates;
A commodity coordinate storage module for storing the final commodity coordinates in association with the image content and the commodity;
When the image content is reproduced on the web interface of the second user terminal, the product coordinate storing module searches for the final product coordinate for the product appearing in the reproduced image content and provides the product coordinate Providing module;
A product information storage module for receiving and storing product information on the product from the first user terminal;
When the product coordinate information providing module is provided with the final product coordinates and the product information corresponding to the final product coordinates is requested by the second user terminal, the requested product information is retrieved from the product information storage module, And a product information providing module provided in real time to the second user terminal,
Wherein the first user terminal comprises:
An advertiser terminal of the product,
Wherein the second user terminal comprises:
A terminal of the consumer of the goods,
The product coordinate automatic tracking module,
The object model is detected and designated for the goods which are touched by the user on the first user terminal by the object detection algorithm, and the designated characteristic points are extracted and registered as tracking characteristic points, and a sparst optical flow A method for tracking tracing feature points using a pyramid LK (Lucas-Kanade) algorithm, comprising: determining whether a shape of the object model changes, whether motion of the object model occurs, and whether the model is partially obscured, (RANSAC) algorithm is applied based on the tracking feature points of the object model when motion of the object model occurs, when the motion of the object model is generated by combining a CamShift algorithm based on a histogram To estimate the motion of the object model and to estimate Wherein the object model is configured to update the object model through a similarity transformation algorithm, and when a part of the updated object model is obscured, applying a Kalman filter when the object model appears again completely, Wherein the at least one of the plurality of users is configured to perform object detection.

Images