KR20140044663A - Information retrieval method by using broadcast receiving apparatus with display device and the apparatus thereof - Google Patents

Abstract

A broadcast receiving device is disclosed. The broadcast receiving device for receiving a broadcast signal comprises a receiving unit for receiving broadcast content; a signal processing unit for processing the broadcast content; a display unit for displaying the broadcast content processed by the signal processing unit; an input unit for receiving a searching command; a control unit for controlling the signal processing unit to capture an image frame displayed by the display unit if the searching command is inputted; and a communication unit for transmitting the captured image frame to the other display device. The control unit controls the display unit to display a searching result if a searching result for an object displayed in the image frame captured by the other display device is transmitted, thereby specifying an object image displayed in a broadcast screen and searching information on the specified interest object. [Reference numerals] (110) Receiving unit; (120) Signal processing unit; (130) Display unit; (140) Input unit; (150) Communication unit; (160) Control unit

Description

Broadcast receiving device and display device and searching method using same {INFORMATION RETRIEVAL METHOD BY USING BROADCAST RECEIVING APPARATUS WITH DISPLAY DEVICE AND THE APPARATUS THEREOF}

The present invention relates to a broadcast receiving device and a display device, and a search method using the same. More particularly, the present invention relates to a broadcast receiving device and a display device for specifying and searching an object image on a broadcast screen, and a search method thereof.

The broadcast reception device is a device that outputs broadcast content. Broadcast content often includes places with new objects, delicious food, and beautiful scenery to attract and entertain the user. Therefore, when the user watches the broadcast content, the user may attempt to search with interest in an object, food, place, or the like in the broadcast content.

In the related art, in order to search for an object of interest to a user, a user may search using an internet-connected device (eg, a desktop computer). Therefore, the search is attempted after the broadcast content is finished, or the broadcast content has been abandoned and the search has to be made.

Recently, with the development of wireless Internet technology and portable device technology, a user can search while watching broadcast contents. However, even when the user searches while watching, the user should search using text. Therefore, the search may take a while, so there is a problem in that broadcast content cannot be viewed while the search is performed.

When a user searches while watching broadcast content, the user wants to search for a specific object. For example, if you are looking at aaa mufflers used by AAA performers in a particular drama, you want to search for aaa mufflers. Generally, when a user searches for aaa muffler using a portable device, the word muffler is entered and the search result is checked. However, if you enter only the word muffler and search for it, you will only get a search result for the general muffler. In order to search more accurately, if you search by typing the word 'specific drama AAA performer muffler', it is not only time-consuming, but even if you enter a specific search, the desired search result does not always come out.

In order to solve this problem, various methods have been proposed. For example, a user may draw a muffler and search by picture. However, if the user does not draw well, the search results may be wrong. In addition, since the aaa muffler used by AAA performers is difficult to represent in a picture, it is difficult to obtain accurate search results. Therefore, there is a need for a technique for more effectively obtaining a desired search result.

SUMMARY OF THE INVENTION The present invention has been made in accordance with the above-described needs, and an object of the present invention is to provide a broadcast receiving device and display device for specifying an object image displayed on a broadcast screen and searching for information on a specified object, and a search method thereof. It is done.

According to an embodiment for achieving the above object of the present invention, in a broadcast receiving apparatus for receiving a broadcast signal, a receiver for receiving broadcast content, a signal processor for processing the broadcast content, and the broadcast content processed by the signal processor A display unit for displaying a display unit, an input unit for receiving a search command, a control unit for controlling the signal processor to capture an image frame displayed on the display unit when the search command is input, and the captured image frame. And a communication unit for transmitting a search result for the object displayed on the captured image frame in the other display device, and displaying the search result on the display unit.

The controller may control the signal processor and the communication unit to capture a plurality of image frames displayed within a preset time range based on a time point at which the search command is input and transmit the captured image frames to the other display device.

The communication unit transmits broadcast content information corresponding to the captured image frame to the other display device, wherein the broadcast content information includes EPG information, broadcast content production information, object information for each image frame of the broadcast content, and photographing. It may include at least one of place information, character information, and image frame identification information.

According to an exemplary embodiment of the present invention, a display apparatus includes a communication unit configured to receive a captured image frame from a broadcast receiving apparatus displaying broadcast content, and a UI generator configured to generate a search UI including the image frame. A display unit for displaying the search UI, an input unit for receiving a user selection for an object in the image frame displayed on the search UI, and when the object is specified by the user selection, accesses a web server for the specified object. And a control unit for performing a search.

The controller may specify the object and extract an identifier for searching for the specified object by using the location information and the image area information of the user selection area in the image frame.

The communication unit receives broadcast content information corresponding to the captured image frame from the broadcast receiving device, and the controller performs a search for the specified object by using the extracted identifier and the received broadcast content information. Can be done.

The communication unit may receive a plurality of image frames from the broadcast receiving device, and the search UI may include a first area displaying one of the plurality of image frames, and displaying the rest of the plurality of image frames in thumbnail form. And a second area, a third area for selecting a tool for specifying an object in the first area, and a fourth area for receiving additional information for searching.

The controller may control the communication unit to transmit a search result for the object to the broadcast reception device.

The controller may control the display unit to display a search result for the object.

According to an embodiment for achieving the above object of the present invention, a search method using a broadcast receiving device and a display device includes the steps of receiving broadcast content, processing the broadcast content, and displaying the processed broadcast content. Receiving a search command; if the search command is input, capturing an image frame being displayed; transmitting the captured image frame to another display device; and receiving the captured image frame from the other display device. And transmitting a search result for the displayed object, displaying the search result.

The capturing may capture a plurality of image frames that were displayed within a preset time range based on a time point at which the search command is input.

In the transmitting, the broadcast content information corresponding to the captured image frame is transmitted to the other display device, and the broadcast content information includes EPG information, broadcast content production information, and object information for each image frame of the broadcast content. , Shooting location information, character information, and image frame identification information.

According to an embodiment for achieving the above object of the present invention, a search method using a broadcast receiving device and a display device includes the steps of receiving a captured image frame from a broadcast receiving device displaying a broadcast content, comprising the image frame; Generating a search UI, displaying the search UI, receiving a user selection for an object in the image frame displayed on the search UI, and when the object is specified by the user selection, accesses the web server to perform the search. Performing a search for the specified object.

Meanwhile, in a search method using a broadcast reception device and a display device, the object is specified using the location information and the image area information of a user-selected area within the image frame and an identifier for searching for the specified object is extracted. It may further comprise the step.

The search method using the broadcast reception device and the display device may further include receiving broadcast content information corresponding to the captured image frame from the broadcast reception device. The performing of the search may include the extracted identifier; The search for the specified object may be performed using the received broadcast content information.

The search method using the broadcast reception device and the display device may further include transmitting a search result for the object to the broadcast reception device.

The search method using the broadcast reception device and the display device may further include displaying a search result for the object.

As described above, according to various embodiments of the present disclosure, it is possible to specify an object image displayed on a broadcast screen and search for information on a specified object of interest.

1 is a view showing the configuration of a search system according to an embodiment of the present invention.
2 is a block diagram of a broadcast receiving apparatus according to an embodiment of the present invention.
3 is a block diagram of a display device according to an embodiment of the present invention.
4 is a block diagram of a broadcast receiving apparatus according to another embodiment of the present invention.
5 is a detailed block diagram of a signal processor and a controller according to an embodiment of the present invention.
6 illustrates the capture of an image frame according to an embodiment of the invention.
FIG. 7 illustrates a UI of a display apparatus according to an embodiment of the present disclosure. FIG.
8 is a view for explaining the specification of an object according to an embodiment of the present invention;
9 to 10 illustrate the specification of an object according to another embodiment of the invention.
11 illustrates a search result according to an embodiment of the present invention.
12 is a flowchart of a search method in a broadcast reception device according to an embodiment of the present invention.
13 is a flowchart of a search method in a display apparatus according to an embodiment of the present invention.
14 is a flowchart of a search method in a broadcast reception device according to another embodiment of the present invention.
15 is a timing diagram illustrating an object searching method in a system according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions of the present invention, and may be changed according to the intentions or customs of the user, the operator, and the like. Therefore, the definition should be based on the contents throughout this specification.

1 is a diagram illustrating a configuration of a search system according to an embodiment of the present invention. According to FIG. 1, a system for searching for an object includes a broadcast receiving device 100 and a display device 200.

The broadcast reception device 100 may be generally implemented as a TV, but is not limited thereto and may be various types of devices having a broadcast reception function such as an electronic picture frame, a PC, a notebook PC, and the like. In addition, the display device 200 may also be various types of devices, such as a mobile phone, PDA, MP3, navigation, auxiliary TV, monitor with a communication function.

The user may watch a broadcast program through the screen of the broadcast reception device 100. In this state, the displayed screen can be captured. When the user captures the screen, the broadcast receiving device 100 connects the communication with the display device 200 and transmits the captured screen to the display device 200.

The transmitted capture screen is displayed on the display apparatus 200. The user may select an object on the screen displayed on the display device 200. The object can be selected by drawing a line along the edge of the object displayed on the screen. When the object is selected, the display apparatus 200 extracts an identifier form required for the search from the search server. The extraction of the identifier may be extracted from broadcast content information as well as object image information. The information extracted from the broadcast content information may speed up the matching process and increase the accuracy of the result. The display apparatus 200 receives the search result after transmitting the extracted identifier to the search server.

The search result may be displayed on the display apparatus 200 and may be transmitted to and displayed on the broadcast receiving apparatus 100. Alternatively, the display device 200 may be displayed on both the display device 200 and the broadcast reception device 100. The broadcast reception device 100 outputs a broadcast program even while searching for the selected object. Therefore, when the search result is displayed on the broadcast receiving device 100, the search result may be displayed on a part of the display unit so as not to disturb the user's viewing of the broadcast program. According to the user's selection, the search result may be displayed on the full screen of the display unit, or may be displayed in a transparent or translucent manner.

When the broadcast reception device 100 or the display device 200 displays a search result, the broadcast reception device 100 or the display device 200 may link and display a re-search button or a related site. When the user selects the re-search button, the search results matched next to each other may be displayed. Alternatively, selecting a linked related site may lead to a linked related site. For example, the related site may be a manufacturer's homepage of an article, an online shopping mall, or the like.

Hereinafter, various embodiments of searching for an object using a broadcast reception device will be described.

<Example using broadcast receiving device and display device>

2 is a block diagram of a broadcast receiving apparatus according to an embodiment of the present invention.

According to FIG. 2, the broadcast reception device 100 includes a receiver 110, a signal processor 120, a display 130, an input 140, a communicator 150, and a controller 160.

The receiver 110 refers to a configuration for receiving a broadcast signal such as an antenna and a tuner. The receiver 110 may include various components such as a demodulator, an equalizer, and the like. The receiver 110 may receive not only a broadcast signal but also broadcast content information and object information on objects constituting the screen of the broadcast signal. Such additional information may be included in a stream of broadcast signals and provided from a broadcast station. The object information may be metadata extracted from the product information included in the stream of the broadcast signal. Alternatively, the object information may be transmitted from a source separate from the broadcast signal. For example, it may be transmitted from a web server of a support company that supports the production of a broadcast program, a shopping mall server, or the like.

The signal processor 120 processes and outputs the received broadcast signal and captures an image frame in response to a user's search command. The broadcast signal received from the receiver 110 is separated into an audio signal and a video signal by a demultiplexer. The separated broadcast voice signal is a digital voice signal compressed at the transmitter. Thus, the audio decoder recovers the compressed digital voice signal and converts it into an analog audio signal through a D / A converter. Similarly, the separated broadcast video signal is a digital video signal compressed by the transmitter.

For example, the compressed digital video signal may be composed of IBBP frames. An I frame is an intra frame and is a frame on which motion prediction of another frame is a reference. The P frame is a predicted frame, which has object information in the frame, and is a reference frame of the B frame. A B frame is a between frame, which does not have substantial data, and is a frame decoded using information of an I frame and a P frame. That is, the signal processor 120 obtains object information from the P frame using the I frame as the reference frame. The B frame between the I frame and the P frame is decoded using information such as the direction and distance of movement of the object between the I frame and the P frame. The decoding order is an I frame, a P frame, and a plurality of B frames, and the reproduction order is an I frame, a plurality of B frames, and a P frame.

In order to decode IBBP frames, PTS (Presentation Time Stamp) including reproduction order information and DTS (Decoding Time Stamp) information including decoding order are required. PTS and DTS information is included in the broadcast signal stream and transmitted. The broadcast receiving device 100 temporarily stores each frame for such decoding. The temporary storage place may be a storage unit (not shown) or a buffer in the signal processing unit 120.

The signal processor 120 may include a video capture unit to capture a frame in response to a user's search command. The video capture unit captures an image frame including the object of interest of the user output through the video decoder. The object of interest of the user refers to an object for which a search is included in a broadcast video image. For example, an object can be an object, food, place, or the like. Since the user will be interested in the object displayed on the screen while watching the broadcast program, the image including the object of interest of the user becomes the currently displayed image frame.

Therefore, when the broadcast receiving device 100 receives a search command from the user, the video capture unit captures an image frame currently displayed. However, the broadcast screen displays dozens of frames per second (eg, 30 frames per second), and it takes time to capture an image frame in response to a user's input command. Therefore, since the screen desired by the user may not be captured, the video capture unit may simultaneously capture previous frames when capturing. Specific methods will be described later.

The display 130 processes the broadcast signal received by the receiver 110 and displays the processed broadcast signal on the screen. The search result transmitted from the display apparatus 200 is displayed. Since the function and configuration of the display unit 130 are well known contents, detailed description and illustration are omitted.

The input unit 140 receives a search execution command from a user. The signal received by the input unit 140 may be an IR signal, an RF signal, or the like. In some cases, it may be an electrical signal generated from an input device (not shown) connected by wire.

The communicator 150 transmits the image frame captured by the signal processor 120 to the display apparatus 200. In some cases, the broadcast content information may be included and transmitted. The broadcast content information may include at least one of EPG information, broadcast content production information, object information for each image frame of the broadcast content, shooting location information, person information, and image frame identification information. The EPG information may include broadcaster information, program title information, and broadcast time information. The image frame identification information may include time information related to the captured image frame (for example, a screen broadcast at 29 minutes and 31 seconds during one hour of running time), PTS information, and DTS information.

The controller 160 controls the signal processor 120 to capture an image frame when a search command is input, and controls the communicator 150 to transmit the captured image frame to the display apparatus 200. In some cases, the broadcast content information may be controlled to be transmitted to the display apparatus 200 together with the captured image frame. When the display apparatus 200 transmits the search result, the display apparatus 200 controls to display the search result on the display 130.

3 is a block diagram of a display apparatus according to an embodiment of the present invention.

According to FIG. 3, the display apparatus 200 includes a communication unit 210, a UI generation unit 220, a display unit 230, an input unit 240, and a control unit 250.

The communication unit 210 receives data from the broadcast reception device 100. The received data includes the captured image frame and may additionally include broadcast content information. When the user selects an object of interest, the user connects to the search server using the extracted identifier for image search and performs a search. The search result may be received from the search server, and the search result may be transmitted to the broadcast reception device 100.

The UI generator 220 generates a search UI that can select the captured image frame and the user interest object. The search UI includes a first area for displaying the received image frame. When there are a plurality of received image frames, the display apparatus may include a second area for displaying the plurality of image frames in a thumbnail form. The third region may include a third region for selecting a tool for specifying an object in the first region, and a fourth region for receiving additional information for searching. A detailed UI will be described later.

The display 230 displays the search UI generated by the UI generator. Since the function and configuration of the display unit 230 are well known contents, detailed descriptions and illustrations will be omitted.

The input unit 240 receives a selection of an object of interest from a user. The selection of the object of interest may be performed by specifying a region connecting a line around the object on the screen displayed in the first region of the search UI. Alternatively, the object of interest may be selected by coloring the object of interest.

In detail, the user may select a predetermined area in the image frame by a screen touch or a cursor manipulation method. The input unit 240 provides a coordinate value of the selected point to the controller 250. The controller 250 divides the selected area of the image frame into a plurality of pixel block units and sets the pixel block corresponding to the coordinate value of the selection point as the reference block. The pixel representative values of the neighboring blocks of the reference block are checked. The pixel representative value may be determined by various values such as a pixel average value, a median value, a maximum pixel value, a minimum pixel value, etc. of all pixels included in the block. The controller 250 determines neighboring pixel blocks having pixel values within a predetermined error range based on the pixel representative value of the reference block. The controller 250 combines all the pixel blocks determined to have pixel values within an error range, and specifies them as one object. The shape or pattern of the object may be recognized based on the arrangement of the pixel blocks, and the color of the object may be recognized based on the pixel value of the pixel blocks.

When there are a plurality of received image frames, the user may receive a command for selecting one image frame in the second area of the search UI. The object of interest of the user is specified by the object selection command received from the user.

The control unit 250 receives data from the broadcast receiving device 100, performs a search when an object is specified by user selection, and configures each component of the display apparatus 200 to transmit a search result to the broadcast receiving device 100. To control wealth.

The controller 250 may include an identifier extractor that extracts an identifier format for image search from the region information of the specified object and the broadcast content information received from the broadcast receiving device 100. The identifier extractor extracts at least one identifier corresponding to the image by analyzing the object image selected by the user. Identifier means a letter or symbol for searching. Identifier extraction means extracting at least one character or symbol corresponding to the image by analyzing the image. The identifier may be information such as color, texture, space, model, or a combination thereof.

For example, when a point where a bag is displayed in an image frame including a red rectangle bag is selected, the red rectangle bag is recognized according to the object recognition method described above. The identifier extractor may determine the identifier as "RED" in consideration of the color of the object, or may determine "Rectangular" in consideration of the shape. Alternatively, a plurality of identifiers may be combined. This identifier may be detected from a previously stored database.

The extracted identifier is transmitted to the search server by the communication unit 210, and the communication unit 210 receives a search result from the search server. For a more accurate search, the specified object image may be used as an image query, and the broadcast content information may be used as a tag query. As such, when the image query and the tag query are used at the same time, it is possible to speed up the matching process of the search and increase the accuracy of the result.

The communication unit 210 receiving the search result transmits the search result to the broadcast reception device 100. In some cases, the search result may be displayed on the display apparatus 200. The search result transmission to the broadcast reception device 100 and the search result display on the display device 200 may be selectively performed, or may be simultaneously performed.

<On broadcast receiving device Conducting a search  Example>

2 and 3 illustrate an embodiment in which a search is performed using two apparatuses, the broadcast receiving apparatus 100 and the display apparatus 200. However, in another embodiment, the user may specify an object in the broadcast reception device 100 and the broadcast reception device 100 may search to display a search result.

The basic configuration and role of the broadcast reception device 100 are the same as described with reference to FIG. 2. Therefore, only the components with differences will be described.

The UI generator generates a search UI that can select the captured image frame and the user interest object. The search UI includes a first area for displaying the captured image frame. When a plurality of captured image frames are included, the display apparatus may include a second area for displaying the plurality of image frames in a thumbnail form. The UI generator may generate a search UI similar to that described in the display apparatus 200 of FIG. 3. The display 130 displays a search UI, performs a search, and displays a search result.

The input unit 140 receives a search execution command and a selection of an object of interest from a user. The object of interest of the user is specified by the object selection command received from the user. The specific process of specifying the object of interest is the same as described with reference to FIG. 3.

The controller 160 performs a search when an object is specified by user selection, and controls each component of the broadcast reception device 100 to display a search result on the display 130. The controller 160 may include an identifier extractor that extracts an identifier format for image search from the region information of the specified object and the broadcast content information received from the broadcast receiving device 100. The role of the identifier extracting unit is the same as described with reference to FIG. 3.

The communicator 150 may use the extracted identifier and broadcast content information for image search as an image query and a tag query, respectively. Sends a query to a search server and receives a search result from the search server.

The received search result is displayed on the display 130. In some cases, the search result may be transmitted to the personal portable terminal device in order to inform the specific user of the result.

Searching using two devices, the broadcast receiving device 100 and the display device 200, is useful in the case of a device where a user watches a certain distance such as a general TV. The broadcast receiving apparatus 100 capable of viewing at a short distance may be useful to perform a search by itself and to display a search result. For example, the broadcast reception device 100 that can be viewed at a close distance may be a monitor, an electronic picture frame, a notebook PC, or the like having a broadcast reception function.

<Example of specifying a user interest object>

Up to now, the embodiment in which the user specifies the object of interest in the display apparatus 200 or the broadcast reception apparatus 100 has been described. However, the object of interest may be automatically specified by the broadcast reception device 100.

4 is a block diagram of a broadcast receiving apparatus according to another embodiment of the present invention.

According to FIG. 4, the broadcast reception device 100 may include a receiver 110, a signal processor 120, a display 130, an input 140, a communicator 150, a controller 160, and an imaging unit 170. Include. Since the receiver 110, the signal processor 120, the display 130, the input 140, and the communicator 150 are the same as described above, a description thereof will be omitted.

The controller 160 controls each component of the broadcast reception device 100 to perform a search by specifying an object of interest included in an image frame and to display a search result. In addition, the gaze tracking is performed in real time from the user image captured by the imaging unit 170, and based on this, the gaze tracking result is updated. In the captured image frame, the gaze tracking result is used to specify a user interest object and to extract an identifier for image search. The extracted identifier may include broadcast content information. The process of specifying the object of interest by the controller 160 will be described in detail with reference to FIG. 5.

The imaging unit 170 captures a user image to track the eyes of the user. The user image refers to an image including location information on a screen that the user gazes by grasping the position of the pupil of the user who is watching the broadcast receiving device 100. The captured user image is transmitted to the controller 160. The imaging unit 170 may include various cameras. For example, the imaging unit 170 may include any one or a combination of an IR camera, an RGB camera, and a depth camera.

5 is a detailed block diagram of a signal processor and a controller according to an embodiment of the present invention.

According to FIG. 5, the signal processor 120 of the broadcast reception device 100 may include a video decoder 121, a video capture unit 122, and a display processor 123. The broadcast video signal input to the video decoder 121 is a digital video signal compressed by the transmitter. The video decoder 121 converts the compressed digital video signal into an original video signal. The converted video signal is input to the video capture unit 122. When a user search command is input through the input unit 140, the video capture unit 122 captures the currently displayed image frame output through the video decoder 121. When the user search command is not input, the video capture unit 122 passes the converted image signal to the display processor 123 and transmits the converted image signal. The display processor 123 processes the search result UI generated according to the converted image signal or the search result and outputs the processed result UI to the display 130.

The controller 160 may include a gaze tracking unit 161, an object-of-interest detector 162, and an identifier extractor 163. The gaze tracking unit 161 receives a user image captured by the imaging unit 170. Real-time gaze tracking is performed from the input user image, and based on this, the gaze tracking result (for example, heat map, etc.) accumulated for a predetermined time is updated. An image is one in which each individual frame image is displayed continuously. For example, in the case of a broadcast program, the same scene is maintained for a predetermined time. Although there is movement of the performer or the like, the same background, performer, props, etc. are displayed while the same scene is maintained. After a certain time, the scene changes. This is called a scene change. The controller 160 may set the time for which the same scene is maintained as the unit time of the gaze tracking until the scene is changed and the next scene is changed. That is, the data is stored by accumulating the position on the screen where the user's eyes stay in the unit of time while the same scene is maintained. Alternatively, a predetermined time may be determined and set as the unit time of eye tracking. When a plurality of users simultaneously watch a broadcast program, the gaze tracking data may be extracted by tracking the gaze of each of the plurality of users. Although the eye gaze tracking unit 161 has been described as being included in the control unit 160, the eye tracking unit 161 may exist independently of the control unit 160, and may be included in the imaging unit 170.

The object of interest detector 162 specifies the object of interest in the captured image frame. The video capture unit 122 captures an image frame by a user's search command. The object-of-interest detector 162 detects and categorizes one or more objects in the captured image frame. The accumulated gaze tracking data is received from the gaze tracking unit 161. Using the captured image frame and the cumulative gaze tracking data, the object of which the user's gaze stays the most (or the longest) is identified as the object of interest. Although only one object of interest may be specified, two or three plurality of objects of interest may be specified according to the cumulative amount of accumulated gaze tracking data.

The identifier extractor 613 extracts an identifier format for image search from the region information of the specified object and the broadcast content information received from the broadcast receiving device 100. The extracted identifier is transmitted to the search server by the communication unit 150, and the communication unit 150 receives a search result from the search server.

The broadcast reception device 100 receiving the search result displays the search result on the display 130. In some cases, the search result may be transmitted to the display apparatus 200 of the user who commanded the search.

In one embodiment, the user inputs the identification number of the unique display device 200 to the broadcast receiving device 100 in advance, and the broadcast receiving device 100 inputs a search command through the imaging unit 170. Determine who is When the input user is determined, the search result is transmitted to the display apparatus 200 of the user. As another embodiment, when the display apparatus 200 of each user is used as an input apparatus of the broadcast receiving apparatus 100, an identification number of the corresponding display apparatus 200 may be transmitted to the broadcast receiving apparatus 100 when a search command is input. have. Therefore, the broadcast reception device 100 may transmit a search result to the corresponding display device 200.

So far, the configuration of the broadcast receiving device 100 and the display device 200 has been described. Hereinafter, a method of capturing an image frame in the broadcast reception device 100 and a search UI of the display device 200 will be described.

FIG. 6 is a diagram illustrating capturing an image frame according to an embodiment of the present invention. FIG.

Referring to FIG. 6, successive broadcast video frames 14, 15, 16, 17, and 18 are shown. For example, FIG. 6 shows a frame 14 in which the bus arrives, a frame 15 in which the performer gets off the bus, a frame 16 in which the performer runs with a bag, a frame 17 in which the performer arrives in front of the building, and a performer. A building frame 18 is shown. Of course, there are continuous broadcast video frames before and after the five frames 14, 15, 16, 17, and 18 shown in FIG.

Assume that the user is interested in the bag and wants to search when the performer watches the frame 16 running with the bag while the user is watching the broadcast program. The user immediately inputs a search command to the broadcast reception device 100 using the input device. That is, the image frame that the user wants to capture is an image frame 12 including a bag. When the broadcast reception device 100 receives a search command from a user, the video capture unit captures an image frame currently displayed.

The broadcast screen displays dozens of frames per second (eg, 30 frames per second), and it takes time to capture an image frame in response to a user's input command. Therefore, even if the user inputs a search command in the frame 16 in which the performer runs with the bag, the capturing command of the video capture unit may be performed in the building frame 18 in which the performer enters. Therefore, in order to prevent such a problem, the video capture unit may capture a plurality of previous frames. The plurality of previous frames may be set based on time, such as a frame of 1 second or 2 seconds ago. Or it may be set based on the number of frames, such as 10 frames or 20 frames. As described above, the broadcast reception device 100 temporarily stores the received video frame before displaying it on the display 130 to decode the compressed digital video signal. Therefore, it is possible to capture a plurality of frames before the frame displayed on the display 130.

In FIG. 6, five image frames were captured. Thus, the frame 16 containing the bag desired by the user was captured. The image frame captured by the video capture unit is transmitted to the display apparatus 200 or displayed on the broadcast receiving apparatus 100 to specify an object of interest according to an embodiment.

7 is a diagram illustrating a UI of a display apparatus according to an exemplary embodiment.

According to an embodiment of the present invention, the captured image frame may be transmitted to the display apparatus 200. The display apparatus 200 may generate a search UI and display the search UI on the display 230. According to FIG. 7, the search UI may include a first area 21 displaying one of a plurality of image frames, a second area 22 displaying thumbnails of the plurality of image frames in a thumbnail form, and an object in the first area. It may include a third area 23 for selecting a tool for specifying, a fourth area 24 for receiving additional information for searching, and a fifth area 25 for displaying the input additional information. .

The first area 21 displays one of the captured image frames. In general, the first displayed image frame may be an image frame when a capture command is performed. If the image frame displayed in the first area 21 is not a desired image frame, the user may select a desired image frame in the second area 22. For example, the image frame 16 including the bag desired by the user may be displayed in thumbnail form together with other image frames of the second area. When the image frame 16 including the bag is selected, the image is displayed in an enlarged form on the first area 21.

In the third area 23, a tool used to specify an object may be located. For example, a text input menu, a line drawing menu, an enlarged menu, and an erase menu may be included. The fourth area 24 may receive additional information to be used for searching. It can be used when a user enters additional information and searches. For example, a specific trademark, a specific pattern, or the like may be directly input. When the user inputs a text, the fifth area 25 displays the input text so that the user can confirm whether the input text is correctly recognized by the display apparatus 200. The search UI illustrated in FIG. 7 is an embodiment, and various search UIs may be displayed.

8 is a view for explaining the specification of an object according to an embodiment of the present invention.

The user may specify an object of interest on the image frame displayed in the first area 21. Referring to FIG. 8, a user wants to specify a bag displayed on an image frame as an object. The user may specify the bag as an object by drawing an outline 31 along the boundary line of the bag using a pen 91 or the like. The input device specifying the object may be the user's finger as well as the pen 91. Alternatively, the user may specify an object by using a tool that the display apparatus 200 may recognize an input on the touch screen. In addition to the above-described method, the object specifying method may be specified by coloring the bag area on the image frame.

When the user specifies the object of interest, the identifier extractor extracts an identifier for image search from the region information of the image. The display apparatus 200 transmits the extracted identifier and broadcast content information to the search server to perform a search.

So far, the method of specifying the object of interest on the image frame captured by the user has been described. As described above, according to another exemplary embodiment of the present invention, the broadcast reception device 100 automatically specifies an object of interest using the gaze information of the user. Hereinafter, a method of specifying an object of interest by using gaze information will be described.

9 to 10 are diagrams for specifying the object according to another embodiment of the present invention.

9 shows one image frame. The gaze tracking unit 161 extracts cumulative data by tracking the eyes of the user on the image frame. For example, a cumulative data may be extracted by generating a heat map. The heat map refers to expressing a variety of information that can be expressed in color as a visual graphic in the form of heat distribution on a certain image. In the present invention, a heat map may be generated by using the eye gaze information of the user captured by the eye tracking method in the imaging unit 170 to extract cumulative data of the user.

Referring to FIG. 9, a frame in which a performer runs with a bag is displayed. In addition, the plurality of regions 32 and 34 displayed on the performer and the bag represent the gaze distribution of the user. That is, the user's eyes stayed in the performer's eyes, hands, shoes portion 32 and bag portion 34. In addition, since the area of the bag portion 34 is large and thick, it means that the gaze of the user stays in the bag portion much (or long). The gaze tracking unit 161 extracts cumulative gaze tracking information in this manner, and transmits the cumulative gaze tracking information to the object-of-interest detection unit 162.

FIG. 10 is a diagram illustrating cumulative gaze tracking information applied after a super pixel segment in the object-of-interest detector 162. The super pixel segment refers to dividing an image into super pixels rather than dividing the image into pixel units. The super pixel means a certain area according to the outline of the image. For example, if a video image contains a person, the super pixel segment is divided into regions according to the contours and colors of the eyes, nose, mouth, and head. When the super pixel segment is executed, the video image is divided into mosaics.

The object-of-interest detector 162 categorizes the image image divided into mosaics according to the contour of the object. For example, when in one bag on a video image, one bag is also divided into a plurality of segments by execution of a super pixel segment. When the plurality of segments are recognized as one area based on the outline, color, etc. of the bag, an object called a bag may be recognized.

Referring to FIG. 10, the gaze accumulation data is applied to some segments 32 of the segments of the shoe region and many segments 34 of the segment 34 of the bag region. Since there is a lot of cumulative data applied to the segment 34 of the bag area, it may be displayed in a darker color. The gaze cumulative data is not applied to all segments of the bag area. However, since the object-of-interest detector 162 performs the above-described categorization step, when the gaze accumulation data is applied to a segment of a certain area or more, the object of interest detector 162 may specify the bag 10 as the object of interest.

If the object of interest is specified, the identifier extractor 163 extracts an identifier for searching as described above. The communication unit 150 transmits the broadcast content information together with the extracted identifier or the extracted identifier to the search server to perform a search. The searched result is received through the communication unit 150 and displayed on the display unit 130 or 230 of the broadcast receiving device 100 or the display device 200.

11 is a diagram illustrating a search result according to an embodiment of the present invention.

Referring to FIG. 11, the display units 130 and 230 display the found bags. Information on the bag and related sites may be displayed on the right side of the display units 130 and 230. For example, information such as a product name, a brand, a release date, a price, and the like may be displayed, and a linked button for accessing an online shopping mall or a manufacturer's homepage where a searched bag may be purchased may be displayed. The rescan button may be displayed on the lower ends of the display units 130 and 230. The above-described search result UI is an embodiment and various search result UIs may be displayed.

For example, you can simply display bags and brands. The display units 130 and 230 may display the search results in the full screen, or may be displayed only in certain areas of the top, bottom, left and right. Alternatively, the transparency of the search result UI may be set to translucent and displayed.

The user or the broadcast reception device 100 may select a plurality of objects of interest. In this case, the device displaying the search result may display the search results of all the selected objects of interest on one screen. Alternatively, the user may display only a search result of one object and display a page indicating that there is another screen page.

12 and 13 are flowcharts of a search method using a broadcast reception device and a display device.

12 is a flowchart illustrating a search method in a broadcast reception device according to an embodiment of the present invention. Referring to FIG. 12, the broadcast reception device receives broadcast content (S1210), processes the received broadcast content (S1220), and displays broadcast content (S1230).

The broadcast reception device receives a search command from a user in operation S1240. The search command may be input using a remote controller, a personal portable device, or the like. The broadcast reception device captures an image frame when receiving a search command in operation S1250. The signal processor captures an image frame displayed at the time when the search command is input. Alternatively, the image frame before the time point at which the search command is input may be captured together. The controller extracts broadcast content information included in the received broadcast stream. In some cases, the broadcast content information may be received in a channel separate from the broadcast stream.

The broadcast content information may include at least one of EPG information, broadcast content production information, object information for each image frame of the broadcast content, shooting location information, person information, and image frame identification information. The EPG information may include broadcaster information, program title information, and broadcast time information. The image frame identification information may include time information, PTS information, and DTS information related to the captured image frame.

The broadcast reception device transmits the captured image frame to the display device in operation S1260. The display apparatus searches if the object of interest is specified from the received image frame.

The search result for the object displayed on the captured image frame is received and displayed (S1270). The display processor outputs the received search result according to a predefined format in the broadcast program currently being reproduced.

13 is a flowchart illustrating a search method in a display apparatus according to an embodiment of the present invention.

The display apparatus receives an image frame captured from the broadcast receiving apparatus in operation S1310. Broadcast content information may be received together with the captured image frame. The display device generates a search UI in operation S1320. In one embodiment, the search UI may select a first area displaying one of a plurality of image frames, a second area displaying thumbnails of the plurality of image frames in a thumbnail form, and a tool for specifying an object in the first area. And a third area for receiving a fourth area for receiving additional information to be used for a search.

The display device receives a selection command for an object in the image frame (S1330). The user may specify an object to search by selecting an area of the object of interest in the image frame (eg, drawing an outline, coloring an object, etc.) using various input means (eg, a finger, a pen, etc.). When the search object is specified, the identifier extractor extracts the identifier from the selected object image information and the broadcast content information. For a more accurate search, the specified object image may be used as an image query, and the broadcast content information may be used as a tag query.

The display apparatus searches for the object specified by the selection (S1340). The communication unit transmits the extracted identifier to the search server, performs a search, and receives a search result. The received search result is transmitted to the broadcast receiving device. In some cases, the display device may display a search result.

14 is a flowchart illustrating a search method in a broadcast reception device according to another embodiment of the present invention.

The broadcast reception device receives a search command in operation S1410. Image information and broadcast information are acquired (S1420). Since it is the same as described with reference to FIG. 12, it will be omitted here.

The broadcast reception device specifies an object by using gaze tracking in operation S1430. The imaging unit captures a user image for tracking the user's eyes. The gaze tracking unit performs gaze tracking from a user image in real time and updates the gaze tracking result accumulated for a predetermined time based on this. The object-of-interest detector detects and categorizes at least one object from the captured image frame in response to a user search command and detects the object of interest based on the accumulated gaze tracking information.

The broadcast reception device searches for a specified object in operation S1440. The search result is received and output (S1450). Since it is the same as described with reference to FIGS. 12 and 13, it will be omitted here.

15 is a timing diagram illustrating an object searching method in a system according to an embodiment of the present invention.

The timing diagram of FIG. 15 is a diagram illustrating an entire process of searching using the broadcast reception device 100 and the display device 200.

The broadcast reception device 100 receives broadcast content in operation S1505. Along with the broadcast content, the broadcast content information is also received. The screen is displayed by processing the broadcast content (S1510). The search command is input from the user (S1515).

In response to the user's search command, the broadcast reception device 100 captures an image frame in operation S1520. A predetermined number of previous frames or a previous frame within a preset time may be captured together.

The broadcast receiving device 100 transmits the captured image frame to the display device 200 (S1525). A plurality of image frames may be transmitted, and broadcast content information may be transmitted together with the captured image frames.

The display apparatus 200 generates and displays a search UI in operation S1530. The object selection command in the image frame is input from the user (S1535). The object to be searched is specified by the user's object selection command. The display apparatus transmits the extracted identifier to the search server to perform a search (S1540), and receives a search result (S1545).

The search result may be displayed on the display apparatus 200 (S1550) and transmitted to the broadcast receiving apparatus 100 (S1555). The broadcast reception device 100 displays a search result in operation S1560. The display of the search result may be simultaneously or selectively performed by the broadcast reception device 100 and the display device 200.

The method of controlling a display device according to the above-described various embodiments may be implemented as a program and provided to a display device.

For example, receiving a search command, when the search command is input, capturing the displayed image frame, transmitting the captured image frame to another display device, and the captured image on the other display device. When the search result for the object displayed in the frame is transmitted, a non-transitory computer readable medium may be provided in which a program for performing the step of displaying the search result is stored.

A non-transitory readable medium is a medium that stores data for a short period of time, such as a register, cache, memory, etc., but semi-permanently stores data and is readable by the apparatus. In particular, the various applications or programs described above may be stored on non-volatile readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM,

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present invention.

100: broadcast receiving device
110: receiver 120: signal processor
121: video decoder 122: video capture unit
123: display processing unit
130: display unit 140: input unit
150: communication unit 160:
161: eye tracker 162: object of interest detector
163: identifier extraction unit
170: imaging unit
200: display device
210: communication unit 220: UI generation unit
230: display part 240: input part

Claims (17)

In the broadcast receiving device for receiving a broadcast signal,
Receiving unit for receiving broadcast content;
A signal processor which processes the broadcast content;
A display unit which displays the broadcast content processed by the signal processor;
An input unit for receiving a search command;
A controller configured to control the signal processor to capture an image frame displayed on the display unit when the search command is input; And
And a communication unit which transmits the captured image frame to another display device.
Wherein,
And transmitting the search result for the object displayed on the captured image frame by the other display device, and displaying the search result on the display unit. The method of claim 1,
Wherein,
And controlling the signal processing unit and the communication unit to capture a plurality of image frames displayed within a preset time range based on a time point at which the search command is input and transmit the captured image frames to the other display device. The method of claim 1,
Wherein,
Transmitting broadcast content information corresponding to the captured image frame to the other display device;
The broadcast content information,
And at least one of EPG information, broadcast content production information, object information for each video frame of the broadcast content, shooting location information, person information, and image frame identification information. A communication unit configured to receive a captured image frame from a broadcast receiving device displaying broadcast content;
A UI generator configured to generate a search UI including the image frame;
A display unit to display the search UI;
An input unit configured to receive a user selection for an object in the image frame displayed on the search UI; And
And a controller configured to access a web server and perform a search for the specified object when the object is specified by the user selection. 5. The method of claim 4,
Wherein,
And an identifier for identifying the object and extracting an identifier for searching for the specified object by using location information and image area information of a user-selected area within the image frame. 6. The method of claim 5,
Wherein,
Receiving broadcast content information corresponding to the captured image frame from the broadcast receiving device,
Wherein,
And searching for the specified object by using the extracted identifier and the received broadcast content information. 5. The method of claim 4,
The communication unit receives a plurality of image frames from the broadcast receiving device,
The search UI,
A first area for displaying one of the plurality of image frames, a second area for displaying the remainder of the plurality of image frames in thumbnail form, and a third area for selecting a tool for specifying an object in the first area And a fourth area for receiving additional information to be used for searching. 5. The method of claim 4,
Wherein,
And controlling the communication unit to transmit a search result for the object to the broadcast reception device. 5. The method of claim 4,
Wherein,
And the display unit to display a search result for the object. In a search method using a broadcast receiving device and a display device,
Receiving broadcast content;
Processing the broadcast content;
Displaying the processed broadcast content;
Receiving a search command;
Capturing an image frame being displayed when the search command is input;
Transmitting the captured image frame to another display device; And
And displaying the search result when the other display device transmits a search result for the object displayed on the captured image frame. 11. The method of claim 10,
The capturing step,
And a plurality of image frames that are displayed within a preset time range based on a time point at which the search command is input. 11. The method of claim 10,
Wherein the transmitting comprises:
Transmitting broadcast content information corresponding to the captured image frame to the other display device;
The broadcast content information,
Search using a broadcast receiving device and a display device comprising at least one of EPG information, broadcast content production information, object information for each image frame, shooting location information, character information, image frame identification information of the broadcast content; Way. Receiving a captured image frame from a broadcast receiving device displaying broadcast content;
Generating a search UI including the image frame;
Displaying the search UI;
Receiving a user selection for an object in the image frame displayed in the search UI; And
And searching for the specified object by accessing a web server when the object is specified by the user selection. The search method using the broadcast receiving device and the display device. 14. The method of claim 13,
And extracting an identifier for identifying the object and searching for the specified object by using the location information and the image area information of the user-selected area within the image frame. Search method using. 15. The method of claim 14,
And receiving broadcast content information corresponding to the captured image frame from the broadcast receiving device.
Performing the search,
And searching for the specified object by using the extracted identifier and the received broadcast content information. 14. The method of claim 13,
And transmitting a search result for the object to the broadcast receiving device. 14. The method of claim 13,
Displaying a search result for the object; Search method using a broadcast receiving device and a display device further comprising.

Images