KR102024168B1 - Contents information service system including contents recognition server, terminal device and the control method thereof - Google Patents

Contents information service system including contents recognition server, terminal device and the control method thereof Download PDF

Info

Publication number
KR102024168B1
KR102024168B1 KR1020120135491A KR20120135491A KR102024168B1 KR 102024168 B1 KR102024168 B1 KR 102024168B1 KR 1020120135491 A KR1020120135491 A KR 1020120135491A KR 20120135491 A KR20120135491 A KR 20120135491A KR 102024168 B1 KR102024168 B1 KR 102024168B1
Authority
KR
South Korea
Prior art keywords
information
recognition server
fingerprint
content
server
Prior art date
Application number
KR1020120135491A
Other languages
Korean (ko)
Other versions
KR20140067792A (en
Inventor
최윤희
윤태현
최현철
Original Assignee
삼성전자주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 삼성전자주식회사 filed Critical 삼성전자주식회사
Priority to KR1020120135491A priority Critical patent/KR102024168B1/en
Publication of KR20140067792A publication Critical patent/KR20140067792A/en
Application granted granted Critical
Publication of KR102024168B1 publication Critical patent/KR102024168B1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/237—Communication with additional data server
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/232—Content retrieval operation locally within server, e.g. reading video streams from disk arrays

Abstract

Disclosed is a content association information providing system including a content recognition server and a terminal device. At least one information recognizing server and a content recognizing server for transmitting the feature data of the objects in the frame constituting the content to the information recognizing server corresponding to the type of each object, wherein the information recognizing server receiving the feature data of the object Search for and transmit object information corresponding to the feature data of the server to the content recognition server, and the content recognition server maps and stores the fingerprint extracted from the frame and the received object information, and when the fingerprint is received from an external device, The object information matching the fingerprint is transmitted to the external device. Accordingly, the network cost can be reduced by reducing the load on the server and the terminal device and reducing the traffic.

Description

A content related information providing system including a content recognizing server and a terminal device and a method of controlling the same {CONTENTS INFORMATION SERVICE SYSTEM INCLUDING CONTENTS RECOGNITION SERVER, TERMINAL DEVICE AND THE CONTROL METHOD THEREOF}

The present invention relates to a content related information providing system including a content recognizing server, a terminal device, and a control method thereof, and more particularly, to a content recognizing server providing a related information included in content of a current screen and a terminal device. The present invention relates to a content association information providing system and a control method thereof.

A terminal device such as a TV plays content. The terminal device extracts a feature of the object and transmits it to a server in order to obtain content information to be played or information about an object included in the content. The server provides various information related to the content by using the characteristics of the received object. To this end, the server extracts the features related to the content based on the data collected in advance and stores them in the database. For example, the information related to the content may be person information, music information, advertisement information, program information, and the like.

One server stores only information related to a feature. For example, the person information recognition server stores only information related to a person included in the content as a database. The advertisement information recognition server stores only information related to the advertisement content as a database. Therefore, since the terminal apparatus must extract all the features of the individual targets in order to obtain content related information, there is a disadvantage in that the computational burden is large. In addition, from the network point of view, since the individual terminal device must transmit the individual target feature to the server, the traffic cost increases. Generally, there are millions to tens of millions of terminal devices such as TVs. Therefore, when a large number of terminal devices send a request to an individual recognition server, the cost of the server for processing it increases considerably. Therefore, in the transmission of content-related information, there is a need for a technology capable of reducing the load on the terminal device, the load on the server, and the network cost.

SUMMARY OF THE INVENTION The present invention has been made in accordance with the above-described needs, and an object of the present invention is to provide a content recognizing server capable of reducing the load on a terminal device and a server when transmitting content related information, and a content related information providing system including the terminal device and control thereof. It is an object to provide a method.

According to an embodiment for achieving the above object of the present invention, the content recognition server is a receiver for receiving the content, and transmits the feature data of the object in the frame constituting the content to the information recognition server corresponding to each object type A communication unit receiving object information corresponding to the feature data of each object from each information recognition server, a storage unit mapping and storing the fingerprint extracted from the frame and the received object information, and a fingerprint received from an external device And a control unit controlling the communication unit to transmit object information matching the received fingerprint to the external device.

Meanwhile, the information recognition server is an advertisement recognition server, the object information is advertisement information, and the controller transmits a fingerprint of a frame constituting broadcast content to the advertisement recognition server using the fingerprint as the feature data of the object and sends the finger. The advertisement information corresponding to the print may be received.

The information recognition server may include at least one of a person recognition server, an audio recognition server, an advertisement recognition server, a broadcast program recognition server, and an object recognition server.

The object information may include at least one of person information, audio information, advertisement information, broadcast program information, and object information.

The fingerprint may be an identifier that may be extracted from at least one of an image and audio in the frame to distinguish the frame from other frames.

According to an embodiment for achieving the above object of the present invention, the terminal device is a control unit for capturing a frame constituting the content to extract the fingerprint, an object that transmits the fingerprint to the content recognition server and match the fingerprint And a communication unit for receiving information and a display unit for displaying the received object information, wherein the object information includes at least one of person information, audio information, advertisement information, broadcast program information, and object information. It may be an identifier that is extracted from at least one of the image and the audio in the frame to distinguish the frame from other frames.

According to an embodiment for achieving the above object of the present invention, the content related information providing system includes at least one information recognizing server and feature data of an object in a frame constituting the content to an information recognizing server corresponding to each object type. And a content recognizing server for transmitting the feature data of the object, wherein the information recognizing server retrieves object information corresponding to the feature data of the object and transmits the object information to the content recognizing server. The extracted fingerprint and the received object information are mapped and stored. When a fingerprint is received from an external device, the object information matching the received fingerprint may be transmitted to the external device.

According to an embodiment for achieving the above object of the present invention, a method of controlling a content recognizing server may include: receiving content, and recognizing feature data of an object in a frame constituting the content corresponding to the type of each object; Receiving object information corresponding to the feature data of each object from each information recognition server, mapping and storing the fingerprint extracted from the frame and the received object information, and a fingerprint from an external device If is received, it may include the step of transmitting the object information matching the received fingerprint to the external device.

Meanwhile, the information recognition server is an advertisement recognition server, the object information is advertisement information, and the receiving of the object information uses the fingerprint of a frame constituting broadcast content as feature data of the object, and the advertisement recognition server. And receive advertisement information corresponding to the fingerprint.

The information recognition server may include at least one of a person recognition server, an audio recognition server, an advertisement recognition server, a broadcast program recognition server, and an object recognition server.

The object information may include at least one of person information, audio information, advertisement information, broadcast program information, and object information.

The fingerprint may be an identifier that may be extracted from at least one of an image and audio in the frame to distinguish the frame from other frames.

According to an embodiment for achieving the above object of the present invention, the control method of the terminal device is to extract a fingerprint by capturing a frame constituting the content, transmitting the fingerprint to a content recognition server and to the fingerprint Receiving matching object information and displaying the received object information, wherein the object information includes at least one of person information, audio information, advertisement information, broadcast program information, and object information; The print may be an identifier that may be extracted from at least one of the image and the audio in the frame to distinguish the frame from other frames.

As described above, according to various embodiments of the present disclosure, content-related information may be transmitted by minimizing the load on the terminal device and the server.

1 is a diagram illustrating a system for providing content association information according to an embodiment of the present invention.
2 is a block diagram of a content recognition server according to an embodiment of the present invention.
3 is a block diagram of a terminal device according to an embodiment of the present invention.
4 is a diagram illustrating a storage unit of a content recognizing server according to an exemplary embodiment.
5 is a diagram illustrating a method of recognizing content association information according to an embodiment of the present invention.
6 is a diagram illustrating a method of recognizing advertisement information according to an embodiment of the present invention.
7 is a diagram illustrating an advertisement information recognition result according to an embodiment of the present invention.
8 is a flowchart illustrating a control method of a content recognition server according to an embodiment of the present invention.
9 is a flowchart illustrating a control method of a terminal device according to an exemplary embodiment.
10 is a timing diagram illustrating a method of providing content association information according to an embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. Terms to be described later are terms defined in consideration of functions in the present invention, and may be changed according to intentions or customs of users or operators. Therefore, the definition should be made based on the contents throughout the specification.

1 is a diagram illustrating a system for providing content association information according to an embodiment of the present invention. According to FIG. 1, a content related information providing system includes a content recognizing server 100, a terminal device 200, and a plurality of information recognizing servers 300a, 300b, 300c, and 300d.

The content recognition server 100 receives the content. For example, the content may be a broadcast program, advertisement content, video content, or the like. The content recognizing server 100 transmits feature data of an object included in a frame constituting the content to the information recognizing servers 300a, 300b, 300c, and 300d corresponding to the type of each object. The plurality of information recognition servers 300a, 300b, 300c, and 300d may include, for example, at least one of a person recognition server, an audio recognition server, an advertisement recognition server, a broadcast program recognition server, and an object recognition server. The plurality of information recognition servers 300a, 300b, 300c, and 300d search for object information by using the characteristic data of the transmitted object, and transmit the search result to the content recognition server 100. For example, the object information may include at least one of person information, audio information, advertisement information, broadcast program information, and object information.

The broadcast program and the video content include a plurality of frames. One frame includes a person, an animal, a background, an object, or audio data such as BGM (BackGround Music). For example, to obtain information about a person included in the frame, the content recognition server 100 extracts a human face image included in the frame. The extracted human face image may be feature data of an object for person recognition. The extracted human face image is transmitted to the person recognition server, and the person recognition server searches for a person using the transmitted feature data of the object and transmits the search result to the content recognition server 100. The content recognizing server 100 extracts a fingerprint from a corresponding frame of the received content and maps the fingerprint to the received person information and stores the fingerprint. The fingerprint will be described later.

For example, audio data of one or two measures may be used as feature data of an object for audio recognition. Each image may be used as feature data of an object to recognize an object, an animal, or a place, and a fingerprint extracted from a frame may be used as feature data of an object to recognize a broadcast program and an advertisement.

Even when a plurality of objects are included in one frame, object information is stored in a similar manner. For example, when a person, an object, or music is included, the content recognition server 100 extracts feature data of an object for a person, feature data of an object for an object, and feature data of an object for music from one frame. do. Also, the fingerprint of the frame is extracted. The feature data of the extracted object is transmitted to the person recognition server, the object recognition server, the audio recognition server, and the broadcast program recognition server, respectively. Each information recognition server 300a, 300b, 300c, or 300d searches for object information using the received feature data of the object and transmits a search result to the content recognition server 100. The content recognizing server 100 maps and stores the fingerprint extracted from the frame, the transmitted person information, object information, music information, and broadcast program information into one.

Examples of the terminal device 200 may be a TV, a smartphone, a PC, a notebook, a tablet, an e-book, a digital photo frame, a PDA, a kiosk, and the like. The terminal device 200 captures a frame of content and extracts a fingerprint. The extracted fingerprint is transmitted to the content recognition server 100. The content recognizing server 100 matches the fingerprint received from the terminal device 200 with the stored fingerprint. The object information mapped to the matched fingerprint is transmitted to the terminal device 200. The terminal device 200 displays the received object information. For example, when a person, object, or music is included in one frame, the object information received by the terminal device 200 may receive at least one of person information, object information, music information, and broadcast program information. have. Alternatively, all object information mapped to the fingerprint may be received in a batch. That is, when the terminal device 200 transmits the fingerprint of the frame and the person information request signal, the content recognizing server 100 may transmit only the person information mapped to the fingerprint to the terminal device 200. Alternatively, when the terminal device 200 transmits only the fingerprint of the frame or transmits the fingerprint and all the information request signals, the content recognizing server 100 may transmit all mapped object information to the terminal device 200.

The content recognizing server 100 transmits feature data of individual objects to the individual information recognizing servers 300a, 300b, 300c, and 300d and receives object information from the individual information recognizing servers 300a, 300b, 300c, and 300d. The content recognizing server 100 and the individual information recognizing servers 300a, 300b, 300c, and 300d may be implemented as one device or may be implemented as separate devices. When implemented as a single device, the feature data and the object information of the object is transmitted internally, so it is faster than data transmission through the external network. In addition, even when implemented as a separate device, it is possible to reduce the cost and speedy processing because information can be transmitted through the internal network or short-range communication without passing through the external network.

Hereinafter, configurations of the content recognizing server 100 and the terminal device 200 included in the content association information providing system will be described.

2 is a block diagram of a content recognition server according to an embodiment of the present invention.

Referring to FIG. 2, the content recognition server 100 includes a receiver 110, a communication unit 120, a storage unit 130, and a controller 140.

The receiver 110 receives the content. For example, a broadcast program can be received. The broadcast program includes an advertisement. In some cases, EPG information related to a broadcast program may be received. The EPG information may be received from a provider of a broadcast program or may be supplied from a separate EPG provider. The communication unit 120 transmits the feature data of the object included in the frame constituting the content to the information recognition server corresponding to the type of the object, and receives the object information corresponding to the feature data of each object.

For example, the information recognition server may be one of a person recognition server, an audio recognition server, an advertisement recognition server, a broadcast program recognition server, and an object recognition server. The object information may be one of person information, audio information, advertisement information, broadcast program information, and object information. In some cases, the broadcast program information may be recognized by the content recognizing server 100 without a separate server. Since the content recognizing server 100 may receive a broadcast program and EPG information, the content recognition server 100 may recognize broadcast program information by using the received broadcast program and EPG information. The storage unit 130 maps and stores the fingerprint of the frame and the received object information.

The controller 140 may extract feature data of the object from a frame constituting the received content. For example, in the case of broadcast content, a number of content corresponding to a received channel may exist. In addition, the feature data of the object may be extracted for all frames constituting the content, and there may be a plurality of feature data of the object extracted in one frame. That is, the controller 140 may extract the feature data of the object from the real time broadcast channel.

The controller 140 extracts the fingerprint with the feature data of the object for every frame. The fingerprint refers to unique data that can distinguish one frame from another frame. In other words, it is also called a visual signature. The fingerprint may be extracted from the image or audio included in the frame. As one example, a feature point is extracted that includes the magnitudes of a particular coordinate in the image and the direction value of that coordinate by a given reference. Several feature points are selected in order of increasing direction. If a certain relationship between the selected feature points is set, a fingerprint for one frame is extracted. When using the same algorithm, the same fingerprint can be extracted. The fingerprint may be extracted by various algorithms in addition to the above examples.

When the fingerprint of the frame constituting the content is received from the terminal device 200, the controller 140 controls the communication unit 120 to transmit object information corresponding to the received fingerprint to the terminal device 200. The controller 140 searches for a matching fingerprint among the stored fingerprints by using the fingerprint received from the terminal device 200. The stored fingerprint is mapped with the object information received from the information recognition server. Therefore, when a matching fingerprint is found, the controller 140 may control the communication unit 120 to transmit object information mapped to the fingerprint to the terminal device 200.

3 is a block diagram of a terminal device according to an exemplary embodiment.

Referring to FIG. 3, the terminal device 200 includes a control unit 210, a communication unit 220, and a display unit 230.

The controller 210 extracts a fingerprint by capturing a frame constituting the content. The fingerprint extracted from the terminal device 200 and the fingerprint extracted from the content recognition server 100 may be extracted using the same algorithm.

The communication unit 220 transmits the fingerprint to the content recognition server 100 and receives object information matching the fingerprint. The terminal device 200 may request only specific object information together with the fingerprint, and may request all object information mapped with the fingerprint.

The display 230 displays object information input from the content recognition server 100. The display unit 230 includes a liquid crystal display panel (LCD panel), a plasma display panel (PDP), an organic light emitting diode (OLED), a vacuum fluorescence display (VFD), and a FED. (Field Emission Display), ELD (Electro Luminescence Display) and the like can be implemented in various display units. In some cases, the display 230 may be implemented as a general display panel without a touch input function, or may be implemented as a touch display panel capable of recognizing a user manipulation using a proximity sensor or a touch sensor. When implemented as a touch display panel, at least one touch gesture may be input through a user's body (eg, a finger including a thumb) or a senseable input means (eg, a stylus pen).

The terminal device 200 transmits only the fingerprint to the content recognizing server 100 in order to receive a plurality of object information for one frame. The fingerprint has a small capacity of several tens of bytes, and if only one fingerprint is transmitted, the plurality of object information included in the corresponding frame may be received, thereby reducing the load and transmission traffic of the terminal device 200.

As described above, the terminal device 200 receives object information from the content recognition server 100. The object information is stored in the storage unit 130 of the content recognition server 100.

4 is a diagram illustrating a storage unit of a content recognizing server according to an exemplary embodiment.

Referring to FIG. 4, one database 131 of the storage 130 of the content recognition server 100 is illustrated. The database 131 stores the fingerprint 10 constituting the content and the object information 20 mapped to the fingerprint 10. The content recognition server 100 receives the content and extracts the fingerprint 10 from a frame constituting the content. Also, a person image, audio data, and an object image are extracted from the frame and transmitted to the person recognition server, the audio recognition server, and the object recognition server, respectively. The content recognition server 100 transmits the extracted fingerprint to the broadcast program recognition server. Person information, audio information, article information, and broadcast program information are received from each information recognition server. In some cases, the broadcast program information may be searched using the broadcast program and EPG information received from the content recognition server 100. The content recognizing server 100 maps the fingerprint and the received object information 20 and stores the fingerprint in the database 131.

When the content recognition server 100 receives the fingerprint from the terminal device 200, the content recognition server 100 transmits the object information 20 mapped to the matching fingerprint 10 to the terminal device 200. Hereinafter, a method of recognizing content association information by matching a fingerprint in the content recognition server 100 will be described.

5 is a diagram illustrating a method of recognizing content association information according to an embodiment of the present invention. The content recognition server 100 stores the fingerprint and the object information for each frame. The terminal device 200 extracts the fingerprint 15 by capturing a frame of the content. The extracted fingerprint 15 is transmitted to the content recognition server 100.

According to FIG. 5, the terminal device 200 extracts a fingerprint 15 for one frame 65 and transmits the fingerprint 15 to the content recognition server 100. The content recognition server 100 compares the received fingerprint 15 with the stored fingerprint and finds a matching fingerprint 11. Since the fingerprints 11 and 15 are unique identifiers of specific frames that are distinguished from other frames, it means that the fingerprints 11 and 15 correspond to the same frames 61 and 65. Accordingly, searching for the fingerprint 11 that matches the received fingerprint 15 may find the same frames 61 and 65.

As described with reference to FIG. 4, the fingerprint 11 of the content recognition server 100 is mapped with the object information in the corresponding frame. Therefore, the content recognizing server 100 may transmit the mapped object information to the terminal device 200.

The above-described process may be similarly applied to the case where the terminal device 200 receives broadcast program information or advertisement information. The following describes a method of recognizing advertisement information.

6 is a diagram illustrating a method of recognizing advertisement information according to an embodiment of the present invention.

Referring to FIG. 6, the advertisement information providing system includes a content recognition server 100, a terminal device 200, an advertisement recognition server 300a, and a receiver 190. The receiver 190 receives a broadcast program including an advertisement. The receiver 190 may further receive EPG information. EPG information may be provided from a broadcast program provider or a separate provider. In addition, the content recognizing server 100 may receive the EPG information before receiving the broadcast program. The receiver 190 may be included in the content recognition server 100 or may be a separate receiver. The broadcast program is received in real time, and the content recognizing server 100 may receive and process the broadcast program in real time.

The content recognition server 100 extracts the fingerprint and transmits the fingerprint to the advertisement recognition server 300a. That is, in order to recognize broadcast program information or advertisement information, the fingerprint of the frame is used as the feature data of the object. This is because the fingerprint capacity is the smallest among feature data of various objects, and broadcast program information or advertisement information can be recognized only by a specific frame. The advertisement recognition server 300a may process only the information request of the content recognition server 100 instead of processing the requests of the plurality of terminal devices 200. Thus, the load on the server is reduced.

The content recognition server 100 stores the advertisement information transmitted from the advertisement recognition server 300a together with the fingerprint. In addition, the broadcast program information received from the broadcast program recognition server or the EPG information received from the content recognition server 100 may be stored together.

The terminal device 200 captures a frame, extracts a fingerprint from an image or audio included in the frame, and requests advertisement information from the content recognition server 100. The content recognizing server 100 transmits advertisement information matching the received fingerprint to the terminal device 200. The content recognizing server 100 may also transmit broadcast program information before and after the advertisement information using the EPG information. The terminal device 200 may display the received advertisement information on the display 230.

7 is a diagram illustrating an advertisement information recognition result according to an embodiment of the present invention.

Referring to FIG. 7, the display 230 of the terminal device 200 displays advertisement content. The advertisement content of FIG. 7 relates to a tablet, and the display unit 230 displays a scene in which a person holds a tablet. When the advertisement content is displayed, the terminal device 200 captures a frame and extracts a fingerprint from the captured frame. The extracted fingerprint is transmitted to the content recognition server 100.

The content recognizing server 100 transmits the matching advertisement information to the terminal device 200 using the received fingerprint. In some cases, the content recognition server 100 may transmit person information, broadcast program information before and after advertisement content, and the like.

The terminal device 200 receives the advertisement information 69 and displays it on the display 230. The display of the advertisement information 69 may open and display a separate window on a part of the terminal device 200. Alternatively, the display 230 may be divided into left and right or up and down and displayed in one area. The displayed advertisement information 69 may include a product name, a manufacturer, a price, a release date, an online shopping mall, and the like. In some cases, additional information such as user ratings and related news may be displayed.

So far, the configuration and operation of the content recognizing server 100 and the terminal device 200 in the content related information providing system have been described. Hereinafter, a method of controlling the content recognizing server 100 and the terminal device 200 performing the above-described process will be described.

8 is a diagram illustrating a control method of a content recognition server according to one embodiment of the present invention.

Referring to FIG. 8, the content recognition server receives content (S810). The content may be a broadcast program including advertisement content, a VOD, or the like. When the content recognition server receives a broadcast program, the content recognition server may also receive EPG information. Therefore, the content received and processed by the content recognition server may be real-time content.

The content recognition server transmits the feature data of the object included in the frame of the content to the information recognition server, and receives the object information corresponding to the feature data of the object from the information recognition server (S820). In addition, the content recognition server extracts the fingerprint from the frame.

The fingerprint of the frame and the received object information are mapped and stored (S830). The received object information includes person information, audio information, advertisement information, broadcast program information, object information, and the like. The content recognizing server maps and stores each object information received from each information recognizing server with the extracted fingerprint.

In operation S840, object information corresponding to a fingerprint received from an external device such as a terminal device is transmitted to the external device. When the fingerprint is received from the external device, the received fingerprint is matched with the stored fingerprint. The matching fingerprint is mapped to the object information. As a result, the object information matching the received fingerprint is transmitted to the external device.

9 is a diagram illustrating a control method of a terminal device according to one embodiment of the present invention.

Referring to FIG. 9, the terminal device extracts a fingerprint by capturing a frame constituting content (S910). The terminal device transmits the fingerprint to the content recognition server, and receives object information matching the fingerprint (S920). Since fingerprints are about tens of bytes in size, they are relatively small compared to other feature data of objects. In addition, the terminal device can obtain a plurality of object information from the content recognition server only by extracting the fingerprint included in the frame, thereby reducing the load on the terminal device and reducing the transmission traffic to the content recognition server.

The terminal device displays the received object information (S930). The display unit of the terminal device may display a received object information by opening a new window in a partial region. Alternatively, the display unit may display object information received in one region by dividing the image into upper, lower, left, and right regions. Since a detailed process has been described above, it will be omitted.

Up to now, the control method of the content recognizing server and the terminal device has been described separately. In the following, a general method of providing content related information will be described.

10 is a timing diagram illustrating a method of providing content association information according to an embodiment of the present invention.

The content recognition server 100 receives the content (S1010). The content may be real time content, such as a broadcast program including advertisement content. The content recognition server 100 transmits the feature data of the object to the information recognition server (S1020). The feature data of the object is information about each object included in the frame. For example, if the object included in the frame is a person, the human face image becomes the feature data of the object. The object image may be object image for object, music data of one or two measures for music, place image for background place, and fingerprint of frame for advertisement and broadcast program.

The content recognizing server 100 receives object information from the information recognizing server (S1030). The information recognition server may include a person recognition server, an audio recognition server, an advertisement recognition server, an object recognition server, a broadcast program recognition server, and the like. Each information recognition server receives only the feature data of the corresponding object from the content recognition server 100 and transmits object information matching the feature data of the received object to the content recognition server 100. The content recognizing server 100 maps and stores the fingerprint of the frame and the received object information in operation S1040.

The terminal device 200 extracts a fingerprint by capturing a frame constituting the content (S1050). The terminal device 200 transmits the fingerprint to the content recognition server 100 (S1060). The terminal device 200 may request only specific object information from the content recognition server 100. Alternatively, all object information mapped with the fingerprint may be requested.

The content recognizing server 100 matches the received fingerprint with the stored fingerprint (S1070). The content recognizing server 100 transmits object information mapped to the matched fingerprint to the terminal device 200 (S1080). The terminal device 200 displays the received object information in operation S1090.

Through the above-described process, it is possible to reduce the load on the content recognition server 100 and the terminal device 200 and to provide content related information to the terminal device while reducing traffic.

The control method of the display apparatus according to the above-described various embodiments may be implemented as a program and provided to the display apparatus.

For example, receiving content, transmitting feature data of an object in a frame constituting the content to an information recognition server corresponding to each type of object, and receiving object information corresponding to feature data of each object from each information recognition server. Receiving, mapping and storing the fingerprint extracted from the frame and the received object information; and if the fingerprint is received from the external device, transmitting the object information matching the received fingerprint to the external device. A non-transitory computer readable medium in which a program is stored may be provided.

Or, extracting a fingerprint by capturing a frame constituting the content, transmitting the fingerprint to a content recognition server, receiving object information matching the fingerprint, and displaying the received object information. This stored non-transitory computer readable medium may be provided.

The non-transitory readable medium refers to a medium that stores data semi-permanently and is readable by a device, not a medium storing data for a short time such as a register, a cache, a memory, and the like. Specifically, the various applications or programs described above may be stored and provided in a non-transitory readable medium such as a CD, a DVD, a hard disk, a Blu-ray disk, a USB, a memory card, a ROM, or the like.

In addition, although the preferred embodiment of the present invention has been shown and described above, the present invention is not limited to the above-described specific embodiment, the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

100: content recognition server
110: receiving unit 120: communication unit
130: storage unit 140: control unit
200: terminal device 210: control unit
220: communication unit 230: display unit
300a, 300b, 300c, 300d: Information Recognition Server

Claims (13)

Receiving unit for receiving the content;
Identify a plurality of objects in a frame constituting the content, identify a plurality of information recognition servers to which feature data of each object is to be transmitted, based on the type of each object,
A communication unit which transmits feature data of each object to an information recognition server corresponding to each object, and receives object information corresponding to feature data of each object from each information recognition server;
A storage unit for mapping and storing the fingerprint extracted from the frame and the received object information; And
And a controller configured to control the communication unit to transmit the object information matching the received fingerprint to the external device when the fingerprint is received from the external device.
The plurality of information recognition server,
And at least one of a person recognition server, an audio recognition server, an advertisement recognition server, a broadcast program recognition server, and an object recognition server. The method of claim 1,
The information recognition server is an advertisement recognition server, the object information is advertising information,
The control unit,
And a fingerprint of a frame constituting broadcast content is transmitted as the feature data of the object to the advertisement recognition server and the advertisement information corresponding to the fingerprint is received. delete The method of claim 1,
The object information includes at least one of person information, audio information, advertisement information, broadcast program information and object information. The method according to claim 1 or 2,
And the fingerprint is an identifier that is extracted from at least one of an image and audio in the frame to distinguish the frame from other frames. delete At least one information recognition server; And
Identify a plurality of objects in a frame constituting content, identify a plurality of information recognition servers to which feature data of each object is to be transmitted based on the type of each object,
And a content recognition server configured to transmit feature data of each object to an information recognition server corresponding to each object.
The information recognition server receiving the feature data of the object,
Search for object information corresponding to the feature data of the object and transmit the object information to the content recognition server;
The content recognition server,
Maps and stores the fingerprint extracted from the frame and the received object information,
When a fingerprint is received from an external device, object information matching the received fingerprint is transmitted to the external device,
The plurality of information recognition server,
And at least one of a person recognition server, an audio recognition server, an advertisement recognition server, a broadcast program recognition server, and an object recognition server. Receiving content;
Identify a plurality of objects in a frame constituting the content, identify a plurality of information recognition servers to which feature data of each object is to be transmitted, based on the type of each object,
Transmitting feature data of each object to an information recognition server corresponding to each object, and receiving object information corresponding to feature data of each object from each information recognition server;
Mapping and storing the fingerprint extracted from the frame and the received object information; And
And when the fingerprint is received from an external device, transmitting object information matching the received fingerprint to the external device.
The plurality of information recognition server,
And at least one of a person recognition server, an audio recognition server, an advertisement recognition server, a broadcast program recognition server, and an object recognition server. The method of claim 8,
The information recognition server is an advertisement recognition server, the object information is advertising information,
Receiving the object information,
And using a fingerprint of a frame constituting broadcast content as feature data of the object, transmitting the fingerprint to the advertisement recognition server and receiving advertisement information corresponding to the fingerprint. delete The method of claim 8,
And the object information includes at least one of person information, audio information, advertisement information, broadcast program information, and object information. The method according to claim 8 or 9,
And wherein the fingerprint is an identifier that can be extracted from at least one of an image and audio in the frame to distinguish the frame from other frames. delete
KR1020120135491A 2012-11-27 2012-11-27 Contents information service system including contents recognition server, terminal device and the control method thereof KR102024168B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020120135491A KR102024168B1 (en) 2012-11-27 2012-11-27 Contents information service system including contents recognition server, terminal device and the control method thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020120135491A KR102024168B1 (en) 2012-11-27 2012-11-27 Contents information service system including contents recognition server, terminal device and the control method thereof

Publications (2)

Publication Number Publication Date
KR20140067792A KR20140067792A (en) 2014-06-05
KR102024168B1 true KR102024168B1 (en) 2019-09-23

Family

ID=51123950

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020120135491A KR102024168B1 (en) 2012-11-27 2012-11-27 Contents information service system including contents recognition server, terminal device and the control method thereof

Country Status (1)

Country Link
KR (1) KR102024168B1 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102424839B1 (en) 2015-10-14 2022-07-25 삼성전자주식회사 Display apparatus and method of controlling thereof
KR101757878B1 (en) * 2015-12-10 2017-07-14 삼성전자주식회사 Contents processing apparatus, contents processing method thereof, server, information providing method of server and information providing system
KR102551550B1 (en) 2018-04-20 2023-07-06 삼성전자주식회사 Electronic device and method for controlling the electronic device thereof
KR102494591B1 (en) * 2018-09-18 2023-02-03 삼성전자주식회사 Electronic apparatus, control method thereof and electronic system

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101211098B1 (en) * 2011-04-25 2012-12-11 (주)엔써즈 System and method for providing advertisement related informatiion included in broadcasting to client terminal through network

Also Published As

Publication number Publication date
KR20140067792A (en) 2014-06-05

Similar Documents

Publication Publication Date Title
US9460204B2 (en) Apparatus and method for scene change detection-based trigger for audio fingerprinting analysis
CN110850593B (en) Mobile device and method for pairing electronic devices by mobile device
KR102217191B1 (en) Terminal device and information providing method thereof
US10333767B2 (en) Methods, systems, and media for media transmission and management
US11314828B2 (en) Dynamic and personalized product placement in electronic files
US20130179436A1 (en) Display apparatus, remote control apparatus, and searching methods thereof
US20120167001A1 (en) Method, system and computer program product for obtaining and displaying supplemental data about a displayed movie, show, event or video game
US20170270506A1 (en) Display terminal-based data processing method
US9538251B2 (en) Systems and methods for automatically enabling subtitles based on user activity
US11755936B2 (en) Systems and methods for determining that a media item is being presented
KR20170116168A (en) System and method for recognition of items in media data and delivery of information related thereto
WO2017008652A1 (en) Information processing method and apparatus
KR102024168B1 (en) Contents information service system including contents recognition server, terminal device and the control method thereof
US10448107B2 (en) Display device
EP3073749A1 (en) Display apparatus, server, and operating method thereof
US20150106531A1 (en) Multicast of stream selection from portable device
US9525918B2 (en) Systems and methods for automatically setting up user preferences for enabling subtitles
US20150206034A1 (en) Apparatus and method for recognizing object using correlation between object and content-related information
JP7332473B2 (en) Server and its control method
US20190141412A1 (en) Display apparatus, control system for the same, and method for controlling the same
CN112154671B (en) Electronic device and content identification information acquisition thereof
US11843829B1 (en) Systems and methods for recommending content items based on an identified posture
KR20140075655A (en) Method for providing logotional advertisement using object recognition based on smart-TV
KR20150103347A (en) Advertising method using smart display
KR20150045000A (en) Advertising method using smart display

Legal Events

Date Code Title Description
A201 Request for examination
E902 Notification of reason for refusal
AMND Amendment
E601 Decision to refuse application
AMND Amendment
X701 Decision to grant (after re-examination)
GRNT Written decision to grant